Your work will touch the full GenAI inference stack , from kernels and runtimes to orchestration and memory management. You will contribute to the design and implementation of the inference engine, and collaborate on model-serving stack optimized for large-scale LLMs inference.

Key responsibilities include:

• Collaborating with researchers to bring new model architectures or features (sparsity, activation compression, mixture-of-experts) into the engine
• Optimizing for latency, throughput, memory efficiency, and hardware utilization across GPUs, and accelerators
• Building and maintaining instrumentation, profiling, and tracing tooling to uncover bottlenecks and guide optimizations
• Developing and enhancing scalable routing, batching, scheduling, memory management, and dynamic loading mechanisms for inference workloads
• Supporting reliability, reproducibility, and fault tolerance in the inference pipelines, including A/B launches, rollback, and model versioning
• Integrating with federated, distributed inference infrastructure – orchestrate across nodes, balance load, handle communication overhead
• Collaborating cross-functionally: with platform engineers, cloud infrastructure, and security/compliance teams
• Documenting and sharing learnings, contributing to internal best practices and open-source efforts when possible

Requirements include:

• BS/MS/PhD in Computer Science, or a related field
• Strong software engineering background (3+ years or equivalent) in performance-critical systems
• Solid understanding of ML inference internals: attention, MLPs, recurrent modules, quantization, sparse operations, etc.
• Hands-on experience with CUDA, GPU programming, and key libraries (cuBLAS, cuDNN, NCCL, etc.)
• Comfortable designing and operating distributed systems, including RPC frameworks, queuing, RPC batching, sharding, memory partitioning
• Demonstrated ability to uncover and solve performance bottlenecks across layers (kernel, memory, networking, scheduler)
• Experience building instrumentation, tracing, and profiling tools for ML models
• Ability to work closely with ML researchers, translate novel model ideas into production systems
• Ownership mindset and eagerness to dive deep into complex system challenges
• Bonus: published research or open-source contributions in ML systems, inference optimization, or model serving

XML job scraping automation by YubHub

Key responsibilities include:

• Leading design reviews and driving architecture within the team
• Defining and owning SLIs/SLOs and ensuring post-incident actions land and reliability improves release-over-release
• Implementing advanced optimizations such as micro-batch schedulers, speculative decoding, and KV-cache reuse
• Strengthening incident posture through capacity planning, autoscaling policy, and rollback/traffic-shift strategies
• Mentoring IC1/IC2 engineers and reviewing cross-team designs to elevate coding/testing standards

We're looking for someone with strong coding skills in Python or Go, deep familiarity with networked systems and performance, and hands-on experience with Kubernetes at production scale. If you have experience with inference internals, batching, caching, mixed precision, and streaming token delivery, that's a plus.

In addition to a competitive salary, we offer a range of benefits including medical, dental, and vision insurance, company-paid life insurance, and flexible PTO. We're committed to creating a work environment that's inclusive, diverse, and supportive of our employees' well-being.

XML job scraping automation by YubHub

Key responsibilities include:

• Lead design reviews and drive architecture within the team; decompose multi-service work into clear milestones.
• Define and own SLIs/SLOs; ensure post-incident actions land and reliability improves release-over-release.
• Implement advanced optimizations (e.g., micro-batch schedulers, speculative decoding, KV-cache reuse) and quantify impact.
• Strengthen incident posture: capacity planning, autoscaling policy, graceful degradation, rollback/traffic-shift strategies.
• Mentor IC1/IC2 engineers; review cross-team designs and elevate coding/testing standards.

Requirements include:

• 3-5 years of industry experience building distributed systems or cloud services.
• Strong coding in Python or Go (C++ a plus) and deep familiarity with networked systems and performance.
• Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks (Prometheus, Grafana, OpenTelemetry).
• Practical knowledge of inference internals: batching, caching, mixed precision (BF16/FP8), streaming token delivery.
• Proven track record improving tail latency (P95/P99) and service reliability through metrics-driven work.

Preferred qualifications include contributions to inference frameworks, experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies, and leading multi-team initiatives or partnering with customers on mission-critical launches.

XML job scraping automation by YubHub