Senior / Staff AI Engineer — Evals & Model Quality
San Francisco · 5+ years experience · Full-time · Onsite · Posted Oct 14, 2026
At a glance
- Experience
- 5+ years experience
- Employment
- Full-time
- Workplace
- Onsite
- Location
- San Francisco
About Further AI
- At FurtherAI, we’re building the AI agents that run insurance — a trillion-dollar industry still powered by manual work. By combining frontier agentic AI with deep domain expertise, we’re reshaping how the industry’s most critical work gets done.
- We’ve raised more than $30M from top investors (Andreessen Horowitz, YC, Nexus, South Park Commons), grown more than XY00% in the last 12 months, and work with some of the largest names in insurance — top 10 carriers/brokers, Fortune 100 companies, and public companies.
- We are a deeply technical company. Both founders are engineers, and engineers have an outsized say on product direction.
- Now, we’re looking for an exceptional AI Engineer to own how we evaluate and continuously improve the quality of AI agents across our platform.
About the role
- FurtherAI is seeking a Senior or Staff AI Engineer to own the evaluation and continuous improvement of AI agents across their platform. The role involves building frameworks to measure agent accuracy, reliability, and quality within the insurance industry. You will work closely with engineering teams to drive model selection and integrate evaluation-driven development processes.
Responsibilities
- Build evaluation frameworks and infrastructure to measure AI agent accuracy, quality, and reliability.
- Develop golden datasets, benchmarks, and regression suites for insurance tasks.
- Design evaluations for multi-step agent trajectories, tool use, and reasoning.
- Develop automated grading systems including LLM-as-judge and deterministic graders.
- Integrate offline and online evaluations into development and release processes.
- Benchmark frontier and open-weight models across accuracy, latency, cost, and reliability.
- Create continuous improvement loops using production failures and agent traces.
Requirements
- 5+ years of software engineering experience.
- Strong hands-on experience building production AI/ML systems.
- Experience building evaluation frameworks, LLM-based applications, or model benchmarking systems.
- Strong understanding of modern LLMs, agent architectures, and evaluation methodologies.
- Strong Python engineering skills.
- Ability to independently own ambiguous technical problems from research to production.
- Clear communication skills.
Preferred qualifications
- Experience with LLM-as-judge
- Experience with human evaluation systems
- Experience with agent trajectory evaluations
- Experience with large-scale benchmarking
- Experience with open-weight models
- Experience with fine-tuning, post-training, or model routing
- Background in B2B SaaS
- 0-to-1 experience
Benefits
- Fully covered health, dental, and vision benefits
- Competitive compensation
- Meaningful stock options
- Company 401(k)
- Unlimited PTO
- Lunch and dinner onsite
- Team events such as happy hours and off-sites
- Pre-tax commuter benefits
Workplace
- Outstanding in-office culture in the heart of San Francisco
- 5x per week in-office requirement
Skills
- Python
- LLMs
- AI Agents
- Model Benchmarking
- Evaluation Frameworks
