The ideal candidate will have strong experience working with AWS and/or GCP, cloud-based data streaming and processing services, containerized application deployments, infrastructure automation, and Site Reliability Engineering (SRE) best practices for performance and cost optimization.

Key Responsibilities:

• Drive initiatives to implement and enforce best practices for data streaming, processing, analytics and monitoring infrastructure.
• Deploy and manage services on Kubernetes-based platforms such as Amazon EKS and Google Kubernetes Engine (GKE).
• Provision and manage cloud infrastructure using Terraform, ensuring best practices in security, scalability, and cost-efficiency.
• Maintain and optimize CI/CD pipelines using Jenkins, ArgoCD, and GitHub Enterprise Actions to support automated deployments and testing.
• Work with cloud-native data services such as AWS Kinesis, AWS Glue, Google Dataflow, and Google Pub/Sub, BigQuery, BigTable
• Familiarity with workflow orchestration services such as Apache Airflow and Google Cloud Composer.
• Develop and maintain automation scripts and tooling using Python to support DevOps processes.
• Monitor system performance, troubleshoot issues, and implement proactive solutions to enhance reliability and efficiency.
• Implement SRE practices to improve service reliability, scalability, and cost-effectiveness.
• Analyze and optimize cloud costs, identifying areas for improvement and implementing cost-saving strategies.
• Ensure compliance with security policies and best practices in cloud environments.
• Drive adoption of company standards and influence data teams to align with best DevOps and SRE practices.
• Collaborate with cross-functional teams to improve development workflows and infrastructure.

Requirements:

• 7+ years of experience in a DevOps, Site Reliability Engineering, or Cloud Infrastructure role.
• Strong experience with AWS and GCP data services, including Kinesis, Glue, Pub/Sub, and Dataflow.
• Proficiency in deploying and managing workloads on Kubernetes (EKS/GKE) in production environments.
• Hands-on experience with Infrastructure-as-Code (IaC) using Terraform.
• Expertise in CI/CD pipeline management using Jenkins, ArgoCD, and GitHub Enterprise Actions.
• Programming skills in Python for automation and scripting.
• Experience with observability and monitoring tools (e.g., Prometheus, Grafana, Datadog, or CloudWatch).
• Strong understanding of SRE principles, including performance monitoring, incident response, and reliability engineering.
• Experience with cost optimization strategies for cloud infrastructure.
• Self-motivated and driven, with a strong ability to influence and drive changes across multiple teams.
• Ability to work collaboratively in an agile environment and support multiple teams.

Preferred Qualifications:

• Experience with data lake architectures and big data processing frameworks (e.g., Apache Spark, Flink, Snowflake, BigQuery).
• Familiarity with event-driven architectures and message queues (e.g., Kafka, RabbitMQ).
• Experience with workflow orchestration tools such as Apache Airflow and Google Cloud Composer.
• Knowledge of service mesh technologies like Istio.
• Experience with GitOps workflows and Kubernetes-native tooling.

XML job scraping automation by YubHub

As a software engineer on our data infrastructure team, you'll design, build, and operate scalable, fault-tolerant infrastructure for LLM Research: distributed compute, data orchestration, and storage across modalities. You'll develop high-throughput systems for data ingestion, processing, and transformation , including training data catalogs, deduplication, quality checks, and search. You'll also build systems for traceability, reproducibility, and robust quality control at every stage of the data lifecycle.

You'll collaborate with research teams to unlock new features, improve data quality, and accelerate training cycles. You'll implement and maintain monitoring and alerting to support platform reliability and performance.

If you're excited by distributed systems, large-scale data mining, open-source tools like Spark, Kafka, Beam, Ray, and Delta Lake, and enjoy building from the ground up, we'd love to hear from you.

XML job scraping automation by YubHub