AgenticEvaluationScientist

Amazon's Customer Service (CS) is undergoing a major transformation leveraging agentic AI techniques to investigate why customers experience defects and how to prevent them in the future. We're looking for scientists to help drive the quality and capabilities of these agentic systems. Amazon's Customer Experience Improvement (CXI Tech) team is looking for a scientists to help drive rigorous development in the agentic era. We seek to identify and eliminate unsatisfactory purchasing experiences that may cause customers to return items or not return to Amazon. This role requires you to re-imagine the way we leverage customer interactions/feedback for defect identification and the actions we take for reliable downstream defect resolution for an improved customer experience throughout the consumer business. Driven by the vision that no defect need ever impact customers twice, we work to develop rigorously tested AI systems to investigate what caused individual defects experienced by customers, and identify the needed changes to ensure they do not occur again. In particular, we are seeking scientists with a proven track record of making efficient use of human annotations to evaluate AI systems. Come help Amazon ensure the best possible experience for our customers! Key job responsibilities - Develop evaluation techniques for agentic solutions that operate at Amazon scale developing techniques to make efficient use of human annotations to establish rigorous performance of agentic systems. - Develop tools and patterns that improve the performance of agentic investigation systems. - Create standalone ML powered models when needed to support automated investigation of customer defects. A day in the life - Develop techniques to measure and improve the performance of agentic systems, even when the performance of those systems exceed the existing human benchmark - Interface closely with Engineering and Product partners to align the direction of scientific development with needs