Senior Production Engineer - DGX Cloud

About this role

NVIDIA DGX Cloud delivers AI services and endpoints for research and production workloads. We are looking for a Senior Production Engineer to build software and automation that make those services reliable, scalable, and safe to operate. The Production Engineering team works on large-scale distributed systems spanning internal and external model endpoints; regional control plane services that orchestrate workloads and route requests; and the GPU/CPU compute infrastructure where inference and agentic workloads run. Our work spans Kubernetes clusters across AWS, Azure, Google Cloud, other partner cloud environments, and on-premises deployments.

What you’ll be doing:

• Build and operate production software, automation, and tooling for control plane services, model deployments, and inference and agentic workloads across DGX Cloud environments.
• Improve the reliability of inference and agentic platforms and services, including NVIDIA Cloud Functions, SGLang- and vLLM-based endpoints, and inference services built with NVIDIA Dynamo, through health validation, safer rollouts, observability, and recovery.
• Improve endpoint availability, inference routing, capacity management, and service health to maintain predictable performance as workloads and demand change.
• Use infrastructure as code and GitOps to deploy, configure, validate, upgrade, and recover services consistently across environments.
• Build workflows for service enablement, model releases, handoff, deprecation, and ongoing operations; replace repeatable manual work with reliable automation.
• Define and instrument SLIs and SLOs for inference and control plane services, including availability and latency, use error budgets to guide reliability improvements, and make production health visible to partner teams.
• Participate in on-call and incident response, troubleshoot failures across routing, model runtimes, software, and infrastructure, and turn recurring issues into automation and durable fixes.
• ...