Research Engineer - Benchmarks

San Francisco · Toronto Hub · 4+ years experience · Full-time · Hybrid · Posted Sep 29, 2026

At a glance

Firecrawl is the easiest way to turn the web into data AI agents can use. One API call converts any URL into clean, LLM-ready markdown or structured data. It's the boring-hard problem everyone building with LLMs eventually hits, solved.
In September 2026, we raised a $75M Series B led by Smash Capital, and we're spending it building the largest repository of knowledge in the world. We hit 8 figures in ARR in year one and more than doubled it in year two. We have 187k+ GitHub stars, putting us in the top 40 repositories of all time, and developers, agents, and category-defining AI companies build on us every day. Growth like this is rare, and we're just getting started.
We're a small team punching far above our weight. Everyone here owns a real piece of the product and company, end to end, and runs it themselves. No hiding behind process or headcount.
This is a place for people who want to work at the frontier: an AI company building the infrastructure other AI companies run on, not one bolting AI onto an existing product. We move fast, go deep, and are building the tools superintelligence will rely on to gather data from the web. That library is called Alexandria, and it starts now.

You will build the independent version of data provider benchmarks by running rigorous, automated, public benchmarks of providers in Alexandria and beyond and publishing them weekly.
The results will feed straight back into Alexandria so it learns which provider to call for which job.
You'll own it end to end: the datasets, the ground truth, the scoring, the harness, the weekly release, and the loop into Alexandria's provider selection.

Design and run head-to-head benchmarks of data providers against verified ground truth.
Build and maintain the test datasets and ground truth, and keep them from going stale or leaking.
Own the automated harness and the weekly benchmark release.
Measure what buyers actually care about: accuracy, coverage, freshness, speed, and cost.
Work with marketing to ship public leaderboard pages that are useful and hold up under scrutiny.
Close the loop into Alexandria so benchmark results change which provider gets called for what.
Keep expanding the categories we benchmark, inside Alexandria and beyond it.

Shipped evals or benchmarks, and can explain exactly why your results were trustworthy.
Experience from a frontier lab, a data company like Scale, Surge, micro1 or Mercor, a third-party benchmark org, or a public benchmark project.
Strong in Python and API integrations, and comfortable with messy vendor APIs, rate limits, and inconsistent schemas.
Know how to build test sets and scoring methods: sampling, labeling, inter-rater agreement, and when to trust an LLM judge and when not to.
Use AI heavily and keep upgrading your own workflow.
Write findings clearly for engineers, marketers, and the vendors on the other side of the leaderboard.

Hybrid, onsite 3+ days a week in San Francisco, CA (SF HQ) or Toronto, ON (Toronto Hub).
Alternatively listed as on-site, five days a week for San Francisco, CA (SF HQ).