Kobie
Loyalty Industry
LeadAIQAEngineer
Neural analysis suggests this role is
optimal for Lead candidates.
“Lead AI QA Engineer at Kobie. Skills: AI QA, LLM testing, Evaluation harnesses, CI/CD integration. Design evaluation harnesses. Build evaluation harnesses”
What You'll Achieve.
Stand up production-grade evaluation harness; Automate trace-based assertions; Define quality scorecard; Define next testing investment
Industry & Context.
Root-cause analysis; Troubleshooting regressions
What They're Looking For.
Must Have
3+ years QA/SDET experience, 1+ years LLM/AI testing experience, Python and PyTest skills, Solid SQL skills, Fluency with Git, Fluency with Docker, Fluency with REST APIs, Solid data security understanding, Solid responsible AI practices understanding, Proven ability to work independently, Proven ability to work within a team, Manage priorities across projects, Manage priorities across time zones, Written communication skills, Verbal communication skills
Nice to Have
Dataiku DSS experience, Dataiku LLM Mesh experience, Dataiku Knowledge Banks experience, Dataiku Prompt Studio experience, Dataiku Visual Agents experience, Dataiku Code Agents experience, Snowflake experience, Snowpark experience, Snowflake Cortex experience, Red-teaming experience, Prompt-injection testing experience, Adversarial test generation experience, Multi-agent patterns familiarity, Performance testing experience, Load testing experience, ISTQB certification, AI Testing certification, QA certification, Experience in loyalty domain, Experience in martech domain, Experience in adtech domain, Experience in data-rich B2B domain
What You'll Do.
Design evaluation harnesses
Build evaluation harnesses
Develop framework to verify AI output
Author automated test suites
Validate guardrails around tool execution
Wire evaluations into CI
Build observability into testing
Triage production drift
Own quality end-to-end
Define release criteria
Partner with engineering to fix regressions
Partner with data engineers on retrieval testing
Help shape internal QA standards
Contribute to design reviews
Share knowledge across teams
Participate in DevOps environment
Work closely with developers
Work closely with AI engineers
Work closely with Data Engineers
Work closely with DBAs
Work closely with product partners
How You'll Work.
Team & Collaboration
Cross-functional teams; DevOps environment; Partner with engineering; Partner with data engineers; Partner with platform teams; Work with developers; Work with AI engineers; Work with Data Engineers; Work with DBAs; Work with product partners; Share knowledge; India and U.S. teams
Communication Scope
Written communication; Verbal communication; Technical communication; Stakeholder communication
Process & Methodology
Manage priorities, Manage concurrent projects
Full Job Description
## Description Join our India Tech Hub – Be among the first hires! Kobie, a 35-year veteran of the loyalty industry, a multi-year Forrester Leader, and USA Top Workplace is expanding its global footprint by establishing a Tech Hub in India. Kobie partners with global brands to build deep connections with their customers through personalized, data-driven loyalty experiences and has a mission of growing enterprise value through loyalty. The Tech Hub will serve as a Global Capabilities Center for a broad range of technology roles, and this is your chance to play a pivotal role in shaping our presence in India. Join us as we continue to lead in loyalty, delivering innovative customer experiences for some of the world’s most recognized brands while working alongside some of the best and brightest in loyalty. ## How you will make an impact Design and build evaluation harnesses for agentic systems in Python — golden datasets, LLM-as-judge graders, multi-turn regression suites and trace-based assertions. In addition, develop framework to verify generated AI output. Author automated test suites for prompts, tools, structured outputs (Pydantic / JSON schema), retrieval pipelines (ETL Experience) and end-to-end agent workflows Validate guardrails around tool execution: auth scoping, input/output validation, PII and prompt-injection protections, and hallucination mitigation Wire evaluations into CI using Dataiku Evaluations, GitHub Actions or Jenkins so every change is graded against quality, safety and cost SLOs before it ships Build observability into testing by instrumenting traces with LangSmith, Langfuse, MLflow or OpenTelemetry and triaging production drift back into the eval harness Own quality end-to-end — define release criteria, run pre-prod and shadow tests, and partner with engineering to root-cause and fix regressions quickly Partner with data engineers on Snowflake-backed retrieval testing patterns (Cortex Analyst and Cortex Search Services) and with platform te
Applying for this Lead AI QA Engineer role?
Most applicants get filtered before a human reads their resume. See if yours makes the cut.
How to Apply on Lever
- Lever uses a streamlined one-page form — apply in under 5 minutes.
- LinkedIn import works well; review parsed data before submitting.
- The cover letter field is optional but visible to reviewers — use it to differentiate.
- Referral codes from employees can significantly boost visibility of your application.
ANONYMOUS · UNFILTERED
What do employees actually say about Kobie?
Real rants from real employees. Read before you apply.