MemberofTechnicalStaff,Evals

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pre-training, domain-specific RL, ultra-long context, and inference-time compute to achieve this goal. ABOUT THE ROLE Evals builds the internal platform that teams across Magic use to evaluate the performance of internal and external models. The team supports pre-training, post-training, data, inference, and product, and sits on the critical path of many of the company's most important decisions. As a Member of Technical Staff on Evals, you will build both the platform and the evaluations themselves. You'll develop infrastructure for large-scale evaluations, data ablations, and dataset quality analysis, while designing and validating the methodologies used to measure model performance. Sweating the details matters on this team. Many benchmarks, papers, and open-source evaluation frameworks contain subtle bugs or flawed assumptions that lead to misleading conclusions. We care deeply about correctness, reproducibility, and measurement quality. Evals are essential to the success of the company. By building trustworthy evaluation systems, you will help Magic make better research decisions, build better datasets, and ship better products. WHAT YOU'LL WORK ON - Build and maintain the internal evals platform used across Magic - Design, implement, and validate eval tasks for pre-training, post-training, reinforcement learning, inference, and product systems - Develop infrastructure for running large-scale evaluations - Build systems to measure dataset quality and identify opportunities to improve training data - Improve evaluation correctness, reproducibility, and reliability - Audit and improve upon public benchmarks, evaluation methodologies, and open-sourc