Principal Software Engineer, Orchestration

San Mateo, CA · San Mateo · USD 295,250-345,040 / year · 10+ years experience · Full-time · Onsite · Posted Oct 20, 2026

At a glance

The Principal Software Engineer for the Data Platform Orchestration team will lead the architecture and operation of Roblox's data workload scheduling systems.
This role involves managing Apache Airflow and Kubernetes infrastructure to support thousands of daily DAGs and Spark jobs at scale.
The position focuses on improving system reliability, performance, and developer experience for hundreds of internal engineers.

Define the multi-year architecture for Airflow and the surrounding data platform.
Scale and harden the Airflow scheduler, metadata database, and executors to ensure predictable latency.
Lead the design and operation of EKS clusters, including autoscaling, resource isolation, and cost efficiency.
Build platform tooling, APIs, custom operators, and libraries for internal data teams.
Establish SLOs, alerting, and diagnostics for pipelines and platform components.
Lead incident reviews and implement root-cause fixes in code.
Collaborate with data platform, governance, data science, and product engineering teams to align roadmaps.
Mentor staff, senior, and mid-level engineers while setting standards for code quality and design.

B.S. equivalent in CS or sufficient experience.
10+ years of experience designing, building, and operating production-grade, large-scale distributed systems.
Deep, hands-on experience designing, running, and scaling a production data-related workflow orchestrator or job scheduler.
Proven experience managing and scaling production Kubernetes clusters on AWS (EKS), including Helm, autoscaling, and GitOps.
Strong proficiency in Python, plus Go or Java.
Working knowledge of Spark jobs on Kubernetes and how orchestration interacts with them.
Experience building platforms used by other engineers with a focus on APIs and observability.
Track record of influencing direction across a large engineering organization and driving multi-quarter projects.

Contributions to Apache Airflow or related open-source projects.
Experience with other orchestration systems like Dagster, Temporal, Argo Workflows, or Flyte.
Experience with Terraform or other infrastructure-as-code.
Experience with Prometheus or OpenTelemetry.
Experience with cost optimization or chargeback for shared platforms.
Experience operating infrastructure at consumer-internet scale.