Amazon. com Services LLC
Technology
SoftwareDevelopmentEngineer,AGIDataServices
Neural analysis suggests this role is
optimal for Senior candidates.
“Software Development Engineer, AGI Data Services at Amazon. com Services LLC. Skills: Generative AI, LLM evaluation, GenAI tooling, AWS development. Own LLM-as-a-Judge evaluation pipeline. Design automated evaluation systems”
What You'll Achieve.
Improve Amazon Nova models; Accelerate GenAI momentum; Scale GenAI momentum; Reduce resolution time; Continuously improve data throughput
Industry & Context.
Root cause analysis; Troubleshooting
What They're Looking For.
Must Have
5+ years software development experience, 5+ years programming experience, 5+ years leading design or architecture experience, 5+ years full SDLC experience
Nice to Have
Bachelor's degree in computer science
What You'll Do.
Own LLM-as-a-Judge evaluation pipeline
Design automated evaluation systems
Scale automated evaluation systems
Leverage large language models
Architect judge pipelines
Develop evaluation rubrics
Develop scoring frameworks
Build calibration mechanisms
Build agreement mechanisms
Design GenAI-powered tools
Build GenAI-powered tools
Build conversational troubleshooting agents
Build automated quality assessment tools
Build guided remediation systems
Build workflow copilots
Leverage agent orchestration frameworks
Design custom orchestration layers
Extend agent orchestration frameworks
Build GenAI-forward practices
Introduce prompt engineering patterns
Introduce RAG patterns
Introduce agent orchestration patterns
Introduce LLM evaluation patterns
Implement production systems
Design backend services
Implement backend services
Design data pipelines
Implement data pipelines
Leverage AWS services
Collaborate with Applied Scientists
Collaborate with Technical Program Managers
Collaborate with domain experts
Collaborate with vendor teams
Review pipeline metrics
Monitor judge accuracy
Monitor calibration drift
Monitor agreement rates
Refine evaluation rubrics
Design new judge architectures
Build conversational troubleshooting agents
Iterate on troubleshooting agents
Fine-tune prompt chains
Expand RAG knowledge bases
Find patterns in data quality
Find root causes in data quality
Propose tooling solutions
Automate manual processes
Share GenAI integration patterns
Communicate impact to partners
Communicate roadmaps to partners
Communicate impact to leadership
Communicate roadmaps to leadership
How You'll Work.
Team & Collaboration
Cross-functional teams; Applied Scientists; Technical Program Managers; Domain experts; Vendor teams
Communication Scope
Communicate impact; Communicate roadmaps
Process & Methodology
Roadmap planning
Full Job Description
AGI Data Services strives to be best in class at acquiring, creating and ground-truth data, with the highest standards of privacy and trust, to power the best AI models on Earth. We are seeking a Senior Software Development Engineer (Sr. SDE) who is passionate about Generative AI and has strong engineering fundamentals to own and accelerate the next generation of GenAI-powered tooling within AGI Data Services. The Sr. SDE will design, build, and maintain LLM-as-a-Judge evaluation pipelines that leverage large language models to assess data quality at scale — including judge architectures, evaluation rubrics, scoring models, and calibration mechanisms that align with the standards set by core scientist teams developing Amazon Nova models. The Sr. SDE will also design and build GenAI-powered workflow tools — such as conversational diagnostic agents, automated quality assessment systems, and guided remediation workflows — that streamline data collection and quality assurance processes, enabling cross-functional teams to rapidly identify issues, reduce resolution time, and continuously improve data throughput. The Sr. SDE's work will directly improve Amazon Nova models. Our team has built a strong foundation of GenAI-powered engineering practices — this senior role will accelerate and scale that momentum. This role offers direct visibility to VP and SVP leadership. Key job responsibilities The Sr. SDE will own the LLM-as-a-Judge evaluation pipeline — designing, building, and scaling automated evaluation systems that leverage large language models to assess data quality. The Sr. SDE will architect judge pipelines, develop evaluation rubrics and scoring frameworks, build calibration and agreement mechanisms, and ensure judge outputs align with quality standards defined by core scientist teams. The Sr. SDE will design and build GenAI-powered diagnostic and workflow tools — including conversational troubleshooting agents, automated quality assessment tools, guided remediati
Applying for this Software Development Engineer, AGI Data Services role?
Most applicants get filtered before a human reads their resume. See if yours makes the cut.
ANONYMOUS · UNFILTERED
What do employees actually say about Amazon. com Services LLC?
Real rants from real employees. Read before you apply.