Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)
San Francisco · Seattle · New York City · 5+ years experience · Full-time · Posted Oct 19, 2026
At a glance
- Experience
- 5+ years experience
- Employment
- Full-time
- Location
- San Francisco · Seattle · New York City
About the role
- The Enterprise Agent Evals & Efficiency team makes Perplexity's agents better at completing real work across enterprise connectors, tools, and data.
- This is a hands-on engineering role with end-to-end ownership, from building realistic enterprise evaluation environments to shipping improvements across the agent harness, tools, and skills.
- Working with AI Research, Connectors, and Platform teams, you'll turn experimental results into measurable gains in task success, latency, and cost per successful task.
Responsibilities
- Build realistic enterprise evaluation environments spanning connected applications, representative datasets, user roles, and permissions, with complex, multi-step workflows and reproducible system state.
- Own evaluation infrastructure, task suites, and automated graders that measure how well agents complete enterprise work, calibrated against human judgment and grounded in end-to-end outcomes.
- Analyze agent trajectories and production failures, identify root causes, and ship improvements across the agent harness, tools, and skills that measurably improve task success and reliability.
- Develop reusable skills and improve tool descriptions, schemas, and result handling so agents can select the right tools, execute workflows correctly, and recover from errors with fewer unnecessary calls.
- Build and evaluate model-selection and routing strategies that match each task to the right model and effort level, balancing quality, latency, reliability, and cost.
- Improve agent efficiency through context management, skill loading, caching, and execution strategies that reduce wasted tokens and cost per successful task without compromising quality or safety.
- Build regression gates and launch-readiness signals for model, connector, tool, skill, and harness changes, and validate that evaluation gains translate into better production outcomes.
Requirements
- 5+ years of software engineering or machine learning engineering experience, with a track record of owning complex production systems from design through deployment and operation.
- Strong system design skills, including building reliable, scalable infrastructure for experimentation, evaluation, or data-intensive applications.
- Strong proficiency in a backend language such as Python, Go, or Rust, and the ability to work effectively across a multi-language production stack.
- Hands-on experience building, evaluating, or improving LLM-based applications or agents, including tool use and multi-step workflows.
- Deep expertise in at least one of the following: Evaluation infrastructure, benchmark design, and automated grading; Agent harnesses, tool interfaces, and reusable skills for complex workflows; Realistic test environments or simulations spanning multiple systems, datasets, permissions, and stateful workflows; Model selection, routing, and inference efficiency, including context management and caching.
- Strong experimental and statistical judgment, including choosing representative tasks, defining meaningful success criteria, diagnosing failures, and distinguishing real improvements from noise.
- Sound technical judgment in balancing task quality, latency, reliability, and cost, with a track record of translating evaluation findings into shipped improvements.
- A track record of leading ambiguous, cross-team initiatives from architecture through production adoption, setting technical direction, and raising engineering quality through mentorship.
Skills
- Python
- Go
- Rust
- LLM
