Principal Software Engineer, Applied AI

San Francisco · USD 224,000-308,000 / year · Posted Oct 15, 2026

At a glance

The team that built Jeeva AI is building Digital Workers, a new suite of autonomous agents for identity operations and security inside Okta.
You will help shape the product with the ownership and pace of a startup and the resources of a public company.
You will build the harnesses and evaluations that make these workers reliable, investigating risk, using tools, waiting for approvals and completing tasks across enterprise systems.

Build the agent harness for planning, tool use, memory, context compaction, interruption and recovery during long-running work.
Integrate OpenAI and Anthropic model APIs, selecting models using measured quality, latency and cost, with clear failure handling.
Evaluate existing harnesses and build only the missing pieces, keeping the harness connected to AWS workflow execution and approval state.
Keep authorization and sensitive writes behind application policy checks and require evidence of the resulting state before marking work complete.
Build evaluation datasets and replay tests for valid findings, access decisions, app setup and AI agent onboarding.
Measure false positives, missed risks, unsafe actions, completion rate and cost, and turn production failures into regression tests.
Test prompt injection, stale evidence, conflicting instructions and unsafe tool use while protecting tenant boundaries in retrieval and memory.
Improve prompts, retrieval, tools and model routing, and evaluate fine-tuning or other post-training methods when results justify them and data use is approved.
Set product engineering direction across workflows and implement the shared components that help the team ship faster.

Strong Python engineering skills and experience shipping an LLM application that uses tools.
Hands-on experience building agent workflows or evaluation systems for tool-using LLM applications.
Experience integrating model APIs, structured outputs and asynchronous services.
Depth in agent execution or evaluation systems, with working code and failure analysis you can explain.
Ability to distinguish a plausible model response from a supported finding or verified action.

Open-source contributions, agent observability, security evaluations or synthetic evaluation data.
Fine-tuning, preference data, post-training evaluation or model distillation using approved datasets.
Identity and security workflows.

Show all 13 skills