Member of Technical Staff - Infrastructure Reliability

About xAI

xAI’s mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge. Our team is small, highly motivated, and focused on engineering excellence. This organization is for individuals who appreciate challenging themselves and thrive on curiosity. We operate with a flat organizational structure. All employees are expected to be hands-on and to contribute directly to the company’s mission. Leadership is given to those who show initiative and consistently deliver excellence. Work ethic and strong prioritization skills are important. All engineers are expected to have strong communication skills. They should be able to concisely and accurately share knowledge with their teammates.

About the Role

We are training some of the largest models in the world on the latest hardware across multiple environments. To do this reliably at xAI’s pace, we need engineers who have battle-tested experience keeping massive distributed infrastructure up and running 24/7, including on-prem and cloud-based infrastructure. This is a joint xAI/X role: you will own 24×7 reliability for the world’s largest GPU training superclusters and one of the highest-QPS production systems on the planet (X).

You will own the availability, performance, and evolution of xAI’s core compute, storage, and networking infrastructure. This is not an ops-only role — strong coding is a hard requirement. You will design, implement, and ship systems software, automation, and tooling in Python and/or Rust that directly impact training throughput and cluster utilization. You will be expected to participate in a team on-call rotation and to contribute to ushering xAI into the next generation of infrastructure management across multiple data centers and cloud environments.

Responsibilities Will Include

• Define and execute the technical strategy for infrastructure reliability and scalability
• Build and maintain the automation, observability, and control planes that keep multi-datacenter, hybrid cloud/on-prem environments healthy
• Lead incident response, deep-dive root cause analysis, and post-mortems that drive real fixes
• Identify, instrument, and eliminate systemic failure patterns (capacity, network, hardware, storage, software)
• Design and implement high-leverage systems software (daemons, controllers, schedulers, etc.) in Python and Rust.
• Push the state of the art in large-scale GPU cluster operations and AI workload reliability

Location

Work will be in-office based out of either Palo Alto, California or Dublin, Ireland. 

Travel to our Memphis office is both an expectation and requirement for this role.

Required Qualifications

• 5+ years shipping production software and/or operating distributed infrastructure at scale
• Expert-level knowledge of Linux systems, TCP/IP networking, and systems programming
• Strong coding skills with proven production experience in Rust (strongly preferred) and at least one of Python, Go, or C++.
• Deep experience with large-scale distributed systems in on-prem and cloud environments (GCP experience a plus)
• Hands-on expertise with container orchestration (Kubernetes, Borg-class systems, or custom schedulers), container runtimes, and infrastructure-as-code (Puppet/Chef/Ansible/Terraform)
• Intimate understanding of common failure modes in distributed systems and how to mitigate them (blast radius control, failure domains, canaries, chaos engineering, etc.)
• Track record of participating in (or building) effective on-call rotations in high-stakes environments
• Bachelor’s degree in Computer Science, Electrical Engineering, or equivalent real-world experience

Preferred Experiences

• Significant contributions to large-scale GPU clusters or AI/ML infrastructure
• Experience in on-call rotations and incident response in high-stakes environments.
• Strong problem-solving skills and ability to thrive in a fast-paced, ambiguous setting.
• Experience with high-performance networking (RDMA, RoCE, Infiniband) and low level configuration (eBPG, xdp, io_uring)
• Comfortable with deployment, support, monitoring, administration, and troubleshooting across on-prem, cloud and hybrid infrastructures.

Interview Process

After submitting your application, the team reviews your CV and statement of exceptional work. If your application passes this stage, you will be invited to an initial interview (45 minutes - 1 hour) during which a member of our technical team will ask some basic questions. If you clear the initial phone interview, you will enter the main process, which consists of four interviews:

1. Coding assessment in a language of your choice.
2. Deep-dive on distributed systems, networking, and failure modes
3. Manager Interview
4. 30–45 minute presentation on a large-scale reliability or infrastructure project you personally owned from incident/crises to resolution

Our goal is to finish the main process within one week. We don’t rely on recruiters for assessments. Every application is reviewed by a member of our technical team. All interviews will be conducted via Google Meet.

Privacy Policy

xAI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.