Meltplan
Construction
AISystemsQAEngineer
Neural analysis suggests this role is
optimal for Mid+ candidates.
“AI Systems QA Engineer at Meltplan. Skills: AI QA, LLM evaluation, System testing. Design evaluation frameworks. Develop evaluation frameworks”
What You'll Achieve.
Enhance accuracy; Enhance safety; Improve performance; Improve system reliability; Improve model behavior
Industry & Context.
Analytical skills; Debugging skills; Problem-solving skills; Root cause analysis
What They're Looking For.
Must Have
Bachelor's degree in Computer Science, 5–8 years of experience in QA/testing, Experience in AI/LLM evaluation frameworks, Experience in system testing, Hands-on experience with automated testing, Experience working with Python, Experience with testing frameworks
Nice to Have
Experience in AI/ML or data-driven systems, Exposure to OpenAI Evals, Exposure to LangSmith, Exposure to DeepEval, Exposure to Promptfoo, Worked in construction, Startup experience, Experience with Generative AI, Experience with conversational AI products, Knowledge of CI/CD pipelines, Knowledge of automation workflows, Prior experience in performance testing, Prior experience monitoring distributed systems, Understanding of AI product lifecycle, Understanding of production deployment environments
What You'll Do.
Design evaluation frameworks
Develop evaluation frameworks
Execute evaluation frameworks
Perform end-to-end system testing
Perform regression testing
Perform performance testing
Validate model outputs
Build automated test pipelines
Build quality benchmarks
Collaborate with AI/ML engineers
Collaborate with product teams
Collaborate with platform engineers
Provide actionable feedback
Develop testing scenarios
Measure model performance
Monitor production performance
Improve evaluation metrics
Improve testing standards
Ensure compliance with responsible AI
Ensure compliance with quality assurance
How You'll Work.
Team & Collaboration
AI/ML engineers; Product teams; Platform engineers
Full Job Description
MeltPlan is building the “planning engine” for the $14 Tn construction industry, an AI system designed specifically to optimize decisions before construction begins. While design software optimizes use and aesthetics and construction software optimizes execution and control, MeltPlan is building the missing layer - software that optimizes decisions and tradeoffs upstream, before scope is locked, procurement begins, and change orders become inevitable. MeltPlan’s long-term goal is to help teams make construction “boring” by making planning more intense: surfacing constraints and tradeoffs early, aligning stakeholders before plans are frozen, and reducing the need for late-stage redlines, rework, and change orders. MeltPlan is founded by operators who have built at scale. Kanav previously co-founded Innovaccer, a $3Bn healthtech company focused on making US healthcare more affordable and accessible. He’s now applying that systems-level thinking to construction.He’s joined by Tanmaya Kala, former Project Executive at DPR Construction, who led large commercial, healthcare, and life sciences projects. We combine deep tech scale with real construction execution. What This Role Really is : We are seeking a detail-oriented and technically strong AI QA Engineer to ensure the quality, reliability, and performance of Large Language Model (LLM)-based systems. In this role, you will be responsible for designing and executing test strategies, validating model outputs, and building evaluation frameworks to enhance the accuracy, safety, and overall performance of AI-driven applications.We would particularly value candidates who have hands-on experience in developing evaluation frameworks (evals) for AI systems, along with strong expertise in comprehensive system testing and quality assurance practices.You are responsible for making MeltPlan work in the real world. What You'll Do: Design, develop, and execute evaluation frameworks (Evals) for Large Language Models (LLMs) and AI syst
Applying for this AI Systems QA Engineer role?
Most applicants get filtered before a human reads their resume. See if yours makes the cut.
How to Apply on Greenhouse
- Create a Greenhouse profile before applying — it saves time across multiple applications.
- Upload your resume as a PDF; the parser handles it better than Word.
- Answer all knockout questions carefully — wrong answers auto-reject before a human sees you.
- Enable email notifications to track application status in real time.
ANONYMOUS · UNFILTERED
What do employees actually say about Meltplan?
Real rants from real employees. Read before you apply.