Yeah so this is a common concern we've seen. "How do I know your idea of a senior engineer is the same as my idea of a senior engineer?" And the answer is you don't. We're modifying the report to no longer say things like "Strong Junior dev" but rather to just give you a bunch of raw data, and then you decide how that data is interpreted.

Some examples:

1. 7/10 unit tests passed

2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB

3. Linter returned 10 warnings, 0 errors

4. Cyclomatic complexity counter returned max of 24

5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars

Also, the concern with outsourcing to a cheap contractor is a very legitimate concern, because we ALMOST actually did that. Instead, we decided that it was a bad move and to double down on building product + surfacing clear, unambiguous data. That's also why we're intentionally staying away from ML (at least for now), because it's inherently unpredictable.