Harper
Insurance
SeniorMemberofTechnicalStaff,AIQuality
“Senior Member of Technical Staff, AI Quality at Harper. Skills: AI Quality, LLM Evaluation, Regression Testing. Build capability eval suites. Build regression eval suites”
What You'll Achieve.
Turn agent quality from vibe into number; Know when agent improves; Know when agent regresses before customer
Industry & Context.
Debug hallucination
Long days
What They're Looking For.
Must Have
3-6 years building software, Hands-on production LLM/agent eval experience, Capability + regression suite design, LLM-as-judge graders, Golden datasets, Designed an LLM-as-judge rubric, Debug hallucination by reading transcripts, Familiar with at least one major eval written communication, Write code with AI daily
Nice to Have
Open-source eval-framework red-team/adversarial voice eval, ML eval/observability background
What You'll Do.
Build capability eval suites
Build regression eval suites
Curate golden datasets
Design deterministic graders
Design LLM-as-judge graders
Ship pre-merge eval gates
Wire production trajectory monitoring
Turn ops findings into permanent tests
How You'll Work.
Team & Collaboration
Work alongside engineer
Communication Scope
Written communication
Applying for this Senior Member of Technical Staff, AI Quality role?
Most applicants get filtered before a human reads their resume. See if yours makes the cut.
How to Apply on Ashby
- Ashby is a fast modern ATS — most applications take under 3 minutes.
- The resume parser is strong; verify parsed experience dates and job titles.
- Custom screening questions are often scored algorithmically — answer completely.
- Location field affects geo-based screening; use your actual metro area.
ANONYMOUS · UNFILTERED
What do employees actually say about Harper?
Real rants from real employees. Read before you apply.