Senior AI Infrastructure Engineer, Physical Infrastructure

Seattle · Costa Mesa, CA (OC-00) · USD 166,000-220,000 · 10+ years experience · Full-time · Posted Oct 8, 2026

At a glance

The Senior AI Infrastructure Engineer will lead the vision, execution, and long-term stability of GPU training infrastructure at scale.
This role focuses on cluster robustness, automated resilience, and optimizing high-performance compute, networking, and storage systems for company-wide ML research and platform teams.

Rack, stack, cable, and bring up GPU compute including physical topology, power, cooling, firmware/BIOS, and burn-in validation.
Build and tune the interconnect fabric connecting hundreds of GPUs into low-latency training and inference clusters.
Integrate high-performance parallel storage to sustain throughput for distributed training and multi-modal datasets.
Automate cluster deployment and configuration end-to-end using infrastructure as code.
Operate and extend the Kubernetes/Run:AI environment for GPU scheduling, quota management, and multi-tenant workload isolation.
Own fleet health, including monitoring, alerting, and rapid triage of hardware and network faults.
Onboard engineers and researchers onto the platform and act as an escalation point for debugging and workload optimization.
Partner with product-facing teams to translate emerging compute needs into platform capability.

10+ years in a hands-on infrastructure, HPC, or datacenter engineering role supporting GPU compute at scale.
Hands-on experience with H200/B200/B300 (or comparable) GPU systems including bring-up, cabling, and firmware/driver management.
Experience with high-performance interconnects (NVLink, InfiniBand, RoCE, Spectrum-X) in clusters of hundreds of GPUs.
Experience with high-performance parallel storage (VAST, DDN, Weka, Lustre, or similar).
Proficiency with Kubernetes.
Strong automation background with experience building repeatable, automated deployment pipelines.
Ability to lift/move 50+ lbs and perform physical datacenter work.
Eligible to obtain and maintain an active U.S. Top Secret clearance.

Experience with NVIDIA NVL72 rack scale systems.
Experience supporting LLM token serving/inference infrastructure.
Network fabric tuning experience (congestion control, adaptive routing, QoS) for RoCE/InfiniBand at scale.
Familiarity with GPU/network observability tooling (DCGM, fabric telemetry) and automated fault detection.
Experience supporting infrastructure as a shared platform serving multiple internal customer teams.
Run:ai or similar GPU scheduling/orchestration experience.

Show all 14 skills