Applied AI Engineer

Responsibilities

Build and ship clinical AI agents, evaluation systems, and production harnesses, including prompts, routing, memory, context assembly, and regression gates. Develop closed-loop improvement systems that use clinician feedback and production traces to identify failures and improve agent quality, while collaborating with research, engineering, QA, and clinical teams.

Requirements

Requires strong Python and applied ML experience, hands-on expertise building agentic systems and LLM evaluation frameworks, and the ability to run rigorous experiments and bring results into production. Candidates should demonstrate prompting and context-engineering skills, a portfolio of LLM projects, strong written communication, and high ownership; healthcare, reinforcement learning, and multi-agent experience are strong advantages.

Skills

• Python
• Applied Machine Learning
• PyTorch
• Hugging Face
• Agentic Systems
• LLM Evaluation
• Benchmarking
• Prompt Engineering
• Context Engineering
• Multi-Agent Systems
• Reinforcement Learning
• Experiment Design
• Fine-Tuning
• Clinical AI
• HIPAA and PHI Handling
• Technical Communication

About this role

About Sully.ai
Sully.ai is building the AI operating system for healthcare. Our mission is to make doctors superhuman through AI, so every patient can get faster, better, and more accessible care.
Healthcare is one of the most important systems in the world, yet clinicians still lose hours every day to documentation, scheduling, intake, coding, and administrative work. We are changing that with AI medical employees: Scribes, Receptionists, Coders, Consultants and other agents that operate inside real clinical workflows.
We are backed by Amity Ventures, YC, Baidu Ventures amongst others, have raised $40M+, and are already deployed across hundreds of healthcare organizations. Our products process tens of thousands of clinical encounters every day, automate millions of minutes of administrative work, and help clinicians spend more time with patients.
The role
As an Applied AI Engineer on the Research team, you will work directly with our research and engineering org on the AI medical employees at the core of Sully — Scribe, Receptionist, Consultant, Coder, and what comes next. Your job is to turn the frontier of agent research into agents and eval systems that run in production for real clinicians within days.
This is a builder-researcher role. You read the latest work in agent, harness and context engineering, evals, and reinforcement learning, then ship from it. We do not separate "research" and "engineering" — you are expected to do both, and the work is measured by what lands in front of clinicians, not by a benchmark in isolation. Our agents are architecture- and configuration-driven: most of the behavior comes from decomposition, prompts, routing logic, context assembly, and evals rather than one-off code. You will own the systems that decide what the model sees, measure whether it worked, and make it better on its own over time.
You will own three things: evals, agents, and self-improving loops. Just as importantly, you will help define how Sully does applied research at all — the harnesses, eval infrastructure, and feedback loops that make every future agent faster to build and safer to ship. The environment is high-stakes by design: in clinical AI, accuracy has zero margin for error, and that constraint is the most interesting part of the job.
What you’ll do
Build and scale evaluation systems
Design and scale automated evaluation pipelines (LLM-as-judge + clinician review) with clinical-grade benchmarks for accuracy, factuality, attribution, safety, and instruction-following.
Build the tooling and dashboards researchers use to run evals, and the CI gates that block regressions before they reach patients.
Treat measurement as a science: understand reliability, variance, and scalability of your eval methodology, not just the headline number.
Turn fuzzy notions of "is this good enough?" into clear, defensible metrics that researchers, engineers, and clinical stakeholders can act on.
Build and ship medical AI agents
Design, implement, and ship agents (Scribe, Receptionist, Consultant, Coder, and new ones) on Sully's stack — decomposition, parallel specialists, dynamic output contracts, and per-agent model routing.
Engineer the harness: memory, context compression, retrieval, session/state, and communication patterns between agents.
Own prompts as infrastructure — structured, versioned, composable per specialty and per organization — because without a correction loop as a safety net, prompt quality is load-bearing.
Take a research result to production end-to-end, fast.
Build self-improving and auto-evolving loops
Build closed-loop systems where clinician feedback and production traces drive targeted, evidence-grounded improvements to the specific components that need them — prompt quality that improves weekly, not quarterly.
Build self-improvement loops that automate model understanding and surface where each agent is failing and why.
Stand up continuous evaluation so production traffic is the experiment and the observability layer is the eval harness.
Push context and loop engineering forward
Track emerging research (context engineering, loop engineering, agent harnesses, evals, RL) and bring the best ideas into our agents quickly.
Experiment with the frontier: RL environments for clinical tasks, small fine-tuned models for focused extraction, synthetic data generation, and model selection per agent role.
Run rigorous experiments: define the hypothesis, build the pipeline, run the model, analyze the result, decide what to do next.
Partner across research, engineering, and clinical
Work directly with Engineering, QA, and Clinical to validate, harden, and roll out new capabilities.
Explain technical tradeoffs clearly to research, product, and clinical stakeholders.
Write up findings so they shape product decisions — and so others can build on them.
What good looks like
Fuzzy quality questions become numbers: every shipped agent has automated evals with regression gates, and "did this get better?" is answerable in minutes, not weeks.
Deployed agents perform reliably in production — measurable gains in accuracy, hallucination/omission reduction, or safety on the workflows you own.
New agent capabilities go from research idea to production in days, end-to-end, without sacrificing clinical safety.
A working closed-loop improvement system turns clinician feedback and traces into targeted component fixes, so agent quality improves on a weekly cadence.
The harness, eval tooling, and templates you build make the next agent faster to ship — applied research becomes more scalable, not more bespoke.
The research team has a builder who can read a paper at midnight and ship the improvement the same week, and who raises the technical bar for everyone around them.
What we’re looking for
Required
Strong Python and applied ML background; comfortable with PyTorch / Hugging Face and an agent framework.
Proven experience designing agentic systems and LLM evaluation / benchmarking frameworks — not just calli...