SeniorMemberofTechnicalStaff,AIQuality

SENIOR MEMBER OF TECHNICAL STAFF, AI QUALITY Harper is an AI-native commercial insurance company in San Francisco. We're not bolting AI onto insurance — we're rebuilding the entire business as software, on a simple bet: turning expert human judgment into compute is one of the largest transitions left to make, and a trillion-dollar industry still run 90% by hand is the place to prove it. We've grown ~100x in the last year and we move at that speed — on-site, in person, long days, very high standards. Almost no one joins Harper for insurance; they join to build the company that replaces how it works. THE ROLE Turning judgment into compute only compounds if the company can tell whether the compute is getting better. Today that's mostly vibes: an engineer ships a prompt change, a tool change, or a new model and judges it by feel — "seems better," "the demo passed." Vibes don't survive Series B, and they definitely don't survive an agent that's quoting real coverage for real businesses. Your job is to turn agent quality from a vibe into a number. Harper's agents handle intake, sales, service, voice, and submission packaging; every one needs to be evaluated, regression-tested, and monitored in production. You'll work alongside the engineer setting AI-quality direction and own a specific agent surface end-to-end — so that when the agent improves we know, and when it regresses we know before the customer does. That's how we scale judgment without scaling headcount. WHAT YOU'LL DO - Build capability + regression eval suites for your assigned agents — intake, submissions, placements, renewals, CRM, or voice. - Curate golden datasets from real failure modes: real transcripts, real underwriter back-and-forth, real call recordings. 20–50 sharp cases per agent, not thousands of synthetic ones. - Design graders. Deterministic first (string match, state check, tool-call assertions); LLM-as-judge where deterministic fails; human calibration on samples. - Ship pre-merge eval gates. E