Senior Software Engineer, Data Infrastructure Compute

San Mateo, CA · San Mateo · USD 243,290-295,250 / year · 5+ years experience · Full-time · Onsite · Posted Oct 20, 2026

At a glance

The Data Platform Compute team at Roblox owns the compute layer for petabytes of daily data, managing Kubernetes clusters for Spark workloads.
As a Senior Software Engineer, you will build and operate systems that sit between users and the cluster, including job submission services, interactive notebook compute, and cluster lifecycle automation.

Design, build, and operate the software behind Spark job submission, including APIs, scheduling integration, and user libraries.
Deliver Spark Connect to support notebook workflows, including provisioning and managing EKS compute with isolation and cost controls.
Build fleet management automation for EKS, such as blue-green rollouts for AMIs and Kubernetes versions.
Develop auto-tuning tools that analyze Spark history and event logs to improve performance and reduce costs.
Operate and upgrade YuniKorn-based scheduling, including queue design, quotas, and fairness across tenants.
Evaluate emerging technologies like Apache Comet and run benchmarks on real workloads to make data-backed recommendations.
Define SLOs and metrics, build observability, and drive down toil and compute spend.
Partner with data engineering, ML, and infrastructure teams to understand needs and migrate users to improved paths.

5+ years of experience building and operating production-grade distributed systems or infrastructure platforms.
Hands-on experience running Apache Spark on Kubernetes (EKS, GKE, or self-managed) as a platform.
Strong experience operating production Kubernetes clusters at scale, including upgrades, node lifecycle, and autoscaling.
Strong proficiency in at least one of Go, Java, Scala, or Clojure.
Experience with AWS (EKS, EC2, IAM, S3) or GCP equivalents and infrastructure-as-code.
Experience taking a system from design through production operation, including on-call and incident response.
Ability to influence design decisions, mentor engineers, and drive alignment.
B.S. in Computer Science or equivalent practical experience.

Experience with YuniKorn, Volcano, Kueue, or other batch schedulers on Kubernetes.
Experience with Spark Connect, Livy, Kyuubi, or other interactive and gateway layers for Spark.
Experience with the Spark Operator, Karpenter, Helm, Terraform, or Argo.
Experience building tooling that uses Spark event logs or metrics for tuning, cost attribution, or anomaly detection.
Experience maintaining internal Spark builds or driving Spark version upgrades.
Familiarity with native execution engines (Comet, Gluten, Velox) or JVM performance tuning.
Experience migrating workloads from YARN/Hadoop to Kubernetes.
Contributions to open-source projects in the data or Kubernetes ecosystem.

Show all 14 skills