Key responsibilities include:

• Maintaining research infrastructure, ensuring health, and optimizing components to extract peak performance from the system (both on application and infrastructure side)
• Scaling infrastructure to meet growing research demands while maintaining reliability and performance
• Collaborating with research teams to deeply understand their infrastructure needs, and design solutions that balance performance with cost efficiency
• Identifying and resolving performance bottlenecks and capacity hotspots through deep analysis of distributed systems at scale
• Building and evolving telemetry and monitoring systems to provide deep visibility into infrastructure performance, utilization, and costs across our cloud and datacenter fleets
• Participating in on-call rotations and incident response to maintain system reliability

Technical focus includes:

• Python, Bash, Go
• Kubernetes
• Nvidia GPU drivers and operators
• OTel, Prometheus

Requirements include:

• Experience building or operating large-scale training platforms
• Worked with large-scale compute clusters (GPUs)
• Proven ability to debug performance and reliability issues across large distributed fleets
• Strong problem-solving skills and ability to work independently
• Strong communication skills and the ability to work effectively with both internal and external partners
• Deep knowledge of modern cloud infrastructure including Kubernetes, Infrastructure as Code, AWS, and GCP
• Experience with SLURM

We offer a competitive base annual salary of $180,000-$300,000 USD and a hybrid work model with a meaningful in-person presence.

XML job scraping automation by YubHub