The interesting thing about putting twenty medical AI agents into production is not the agents. It is the order in which they had to be designed. In most domains you can build the happy path first and harden the edges later. In healthcare that sequence is backwards to the point of negligence: the escalation path (what the system does when it should not act alone) is the first thing to design, because it is the thing that makes everything else permissible.
Escalation-first changes the architecture in concrete ways. Every agent needs a defined refusal boundary: the set of situations where the correct output is not an answer but a handoff, with the context packaged so the human picks up mid-stride rather than starting over. The handoff is not a failure mode to minimise. It is a designed outcome to get right. The systems that scare people are not the ones that escalate often; they are the ones that never do.
The systems that scare people are not the ones that escalate often; they are the ones that never do.
The second requirement of the domain is speaking its language, literally. Clinical work runs on controlled vocabularies, and grounding the agents in more than a hundred thousand LOINC codes was not optional plumbing. It was the difference between an assistant that gestures at medicine and one that can participate in it. Every regulated field has an equivalent spine: the terminology, the codes, the canonical identifiers. Ground in them early. Free-text cleverness does not survive contact with a compliance review.
We saw the same pattern hold in finance when one of our principals built a client-facing chatbot for a wealth platform, orchestrating four different agent frameworks under one set of guardrails. Different regulator, same architecture: hard boundaries on what the system may claim, glossaries it must respect, an escalation surface a human actually staffs. Regulated domains do not reward the boldest system. They reward the system whose limits were designed with the same care as its capabilities, and that turns out to be the system that ships.