Amazon (China) Holding Company Limited
Editorial, Writing, Content Management, Program Management, selling partner services
ProgramManager,AIModelEvaluation
Neural analysis suggests this role is
optimal for Manager candidates.
“Program Manager, AI Model Evaluation at Amazon (China) Holding Company Limited. Skills: AI model evaluation, Program management, LLM evaluation. Plan benchmarking exercises. Execute benchmarking exercises”
What You'll Achieve.
Improve AI model performance; Enhance seller experience
Industry & Context.
Root-cause analysis; Data analysis
What They're Looking For.
Must Have
3+ years program management, 3+ years cross-functional work, 3+ years process improvement, Advanced Excel, Advanced SQL, Define program requirements, Use data and metrics
Nice to Have
3+ years end to end delivery, Communicate results to senior leadership, 3+ years driving process improvements, Stakeholder management experience, Build processes, Project management experience, Schedule management experience
What You'll Do.
Plan benchmarking exercises
Execute benchmarking exercises
Define acceptance criteria
Escalate regulatory risks
Prepare audit reports
Prepare benchmarking reports
Provide root-cause analysis
Provide recommendations
Drive process efficiencies
Explore automation opportunities
Enhance data generation productivity
Control project scope
How You'll Work.
Team & Collaboration
Cross functional teams; Senior stakeholders
Communication Scope
Stakeholder engagement; Reporting; Presentations
Process & Methodology
Program management, Project management, Schedule management, Risk mitigation
Full Job Description
Join the Seller AI team where you'll lead benchmarking and evaluation of AI models that enhance the seller experience across Amazon's global marketplace. You'll manage a team dedicated to validating, testing, and improving Artificial Intelligence (AI) and Large Language Models (LLMs) that power innovative seller tools. This role combines strategic leadership with hands-on technical oversight, requiring exceptional communication skills, team management, and stakeholder engagement. In this position, you'll drive the development and implementation of comprehensive benchmarking methodologies to evaluate AI model performance across accuracy, robustness, bias, and reliability metrics. Your expertise will be crucial in translating technical findings into actionable insights that improve Seller Assistant's performance and contribute to the growth of Amazon's seller community worldwide. Key job responsibilities 1. Plan and execute benchmarking exercises for AI models, defining test plans, metrics, and acceptance criteria across accuracy, robustness, bias, and reliability dimensions. 2. Lead a team responsible for validating data based on specific annotation guidelines, ensuring accuracy and quality while escalating potential regulatory risks 3. Prepare comprehensive audit and benchmarking reports, including error ratings, root-cause analysis, and recommendations for senior stakeholders 4. Drive process efficiencies and explore automation opportunities to enhance the productivity of data generation initiatives 5. Mentor team members and help develop their skills while managing overall schedules, proactively mitigating risks, and keeping project scope under control. A day in the life Your day begins with prioritizing benchmarking tasks for your team based on current business requirements. You'll review progress on ongoing audits, provide guidance where needed, and ensure deliverables meet quality standards. Throughout the day, you'll analyze performance data from Seller Assist
Applying for this Program Manager, AI Model Evaluation role?
Most applicants get filtered before a human reads their resume. See if yours makes the cut.
ANONYMOUS · UNFILTERED
What do employees actually say about Amazon (China) Holding Company Limited?
Real rants from real employees. Read before you apply.