Member of Technical Staff - Pre-Training Evals Infra
New York, NY · San Francisco · London · Full-time · Onsite · Posted Oct 13, 2026
At a glance
- Employment
- Full-time
- Workplace
- Onsite
- Location
- New York, NY · San Francisco · London
- Visa sponsorship
- Visa sponsorship available
About the role
- You will build the evaluation infrastructure that researchers use to understand model capabilities, guide training decisions, and assess readiness for release.
- The role spans evaluation authoring, distributed execution, and results analysis, with a particular focus on pre-training.
- You will work directly with researchers to make evaluations faster, more reliable, and easier to use, ensuring that the results accurately reflect the models being evaluated.
Responsibilities
- Design and evolve evaluation abstractions, building clear APIs and data models for evaluation tasks, datasets, configurations, model outputs, and scores.
- Build scalable, reliable execution infrastructure by developing distributed systems for evaluating model checkpoints throughout training and ahead of release.
- Optimize evaluation and inference performance by profiling inference servers and distributed execution to improve throughput, GPU utilization, and time to results.
- Establish evaluation correctness and reproducibility through validation and regression tests, versioned datasets and configurations, and preserved provenance.
- Build researcher-facing tools and intuitive interfaces for inspecting model outputs, comparing checkpoints and experiments, and debugging unexpected behavior.
- Operate and monitor evaluations as training progresses, track workloads and results, triage failures, and develop alerts and diagnostics.
- Turn research needs into reusable systems by partnering directly with researchers to identify bottlenecks, prioritize improvements, and turn bespoke workflows into self-service tools.
Requirements
- Strong proficiency in Python, C++, or Rust, with the ability to reason deeply about concurrency, memory behavior, and performance.
- Experience designing, building, and operating reliable, maintainable distributed systems, with sound judgment about interfaces, failure modes, and architectural tradeoffs.
- Working knowledge of model inference and evaluation, including how implementation and configuration choices can affect measured model quality.
- Ability to investigate complex problems across system boundaries, follow evidence, and identify root causes.
- Strong product judgment and experience translating the needs of researchers or other technical users into clear APIs and usable tools.
- Comfort owning ambiguous problems end to end, prioritizing competing research needs, and balancing rapid delivery with long-term reliability and maintainability.
- Attention to detail and the persistence to investigate subtle discrepancies in model outputs, evaluation results, and system behavior.
Benefits
- Top-tier compensation including salary and equity
- Stock options
- Comprehensive medical, dental, vision, and life insurance
- Annual wellness allowance
- Lunch and dinner provided in the office daily
- 22 weeks paid parental leave for all new birthing and non-birthing parents, including adoptive and surrogate journeys
- Unlimited paid time off in the U.S. and 30 days in the U.K.
- Visa sponsorship support and long-term immigration pathways where applicable
- Regular off-sites, happy hours, and team celebrations
Skills
- Python
- C++
- Rust
- Distributed Systems
