AI Evaluation Systems Engineering Lead

Overview

Microsoft AI is looking for a hands-on engineering lead to build the evaluation infrastructure that helps researchers improve frontier models and make reliable release decisions. You will lead the Eval Systems team, building systems that turn model checkpoints into trustworthy results—quickly, reproducibly, and at scale.

The role combines deep technical work with team leadership. You will shape the architecture and roadmap, contribute to implementation, and hire and develop engineers. Working closely with research, training, and inference teams, you will make evaluation a dependable part of the model-development loop, from early experiments through release readiness.

We are looking for someone who enjoys difficult systems problems, makes thoughtful engineering tradeoffs, and builds teams grounded in clear ownership, collaboration, and technical excellence.

Responsibilities

• Set technical direction. Own the team’s architecture, roadmap, and delivery, balancing immediate research needs with long-term reliability and scalability.
• Build end-to-end evaluation systems. Develop infrastructure for benchmark configuration, model inference, grading, aggregation, and results publication across training checkpoints and deployed models.
• Accelerate research iteration. Reduce evaluation turnaround time and improve throughput, scheduling, and GPU efficiency. Make it easier for researchers to integrate benchmarks, investigate results, and compare models.
• Make results trustworthy. Establish reproducible configurations, traceable results, validation, and regression checks. Partner with evaluation owners to preserve correctness and comparability as models and infrastructure evolve.
• Own operational excellence. Define and track reliability and performance objectives. Improve observability, failure recovery, and on-call practices, addressing recurring problems at their source.
• Stay hands-on. Contribute code, lead design reviews, and help debug challenging problems across evaluation, inference, and distributed execution.
• Build and develop the team. Hire and mentor engineers, establish clear responsibilities, and foster an inclusive environment where people can do their best work.
• Lead across teams. Align research and engineering partners on priorities, dependencies, and tradeoffs, and communicate progress and risks clearly.

Required Qualifications

Preferred Qualifications

• Bachelor's Degree in Computer Science or related technical field AND 10+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
• Substantial software engineering experience building and operating production distributed systems, infrastructure platforms, or ML systems.
• Demonstrated technical leadership, including setting direction and delivering complex projects through a team.
• Strong programming skills, including proficiency in Python and experience contributing to large, shared codebases.
• Experience diagnosing distributed-system failures and improving reliability, performance, and resource efficiency.
• Experience with production operations, including monitoring, incident response, and on-call practices.
• Ability to translate evolving requirements into clear engineering priorities and work effectively with research and engineering partners.
• A track record of mentoring engineers and strengthening team engineering practices.
• Experience with LLM evaluation, benchmarking, model-based grading, or reinforcement learning infrastructure.
• Experience with GPU-backed inference, distributed training, or large-scale compute workloads.
• Familiarity with distributed execution frameworks and schedulers such as Ray, Kubernetes, or Slurm.
• Experience improving experimental reproducibility, validating metrics, or investigating differences across evaluation and serving environments.
• Experience hiring engineers, managing people, or growing a technical team.
• A track record of simplifying complex infrastructure and reducing operational burden through automation and systematic reliability improvements.

Software Engineering IC5 - The typical base pay range for this role across the U.S. is USD $142,800 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year. Software Engineering IC6 - The typical base pay range for this role across the U.S. is USD $165,600 - $296,400 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $220,800 - $331,200 per year.