“The professional decides” is a boundary, not a shield

On 4 August Henry Conter at Kesis & Sisters published The Human in the Loop Is a Regulatory Fiction. His example is an oncologist who signs off a recommendation that left out one fact, and could only have caught it by redoing the whole review. Her signature, he writes, proves that a doctor was present, not that the system was supervised. I keep using the sentence “the professional decides” when I write about Inora. This is the essay in which I ask what it is worth.

The danger of the phrase

Used carelessly, “the professional decides” does the opposite of what it says. It becomes a way to move risk to the person at the end of the chain. Conter makes the case with errors of omission: a reviewer sees what the system showed and cannot see what it left out. If the phrase is only a label on a product, it is a shield for the maker.

So I give it two parts and hold myself to both.

A boundary. Inora gives no clinical judgement and does not diagnose. It supports a professional with her organisation’s knowledge. That is a statement about what the product is for, and the reason it stays outside medical-device scope. Tools that report on their own, which Hardian, a regulatory consultancy, describes in Clinician in the AI loop, sit on the other side of that line. We are not building one.

A design duty. If the decision stays with her, the system owes her what she needs to take it: the criterion that applies, the source it comes from, and the gap where the sources are silent. Put these in front of her, and deciding no longer means doing the work again. That is the difference from Conter’s oncologist, and it is why I think his case does not carry over unchanged. His is a generative recommendation across seventy pages. A care worker asking what her organisation’s protocol says is asking for a bounded, checkable lookup, and I wrote in June, in Oversight is designed, not staffed what that does to oversight.

Who answers for what

Traceability brings back an old rule: responsibility follows where the error came from. The content is the organisation’s protocol. Finding it and assembling the answer is the maker’s harness. The decision is the professional’s. Each stays with the party that controls it. I will not make any statement here about what liability Axiomatic accepts; that is a matter for contracts and for lawyers, not for an essay.

The record of what she was shown protects her. It belongs in the organisation’s compliance record, and it must never become a surveillance file on the care worker.

Where I disagree

Conter writes that a clinician cannot see what an update changed. That holds for a hosted black box. It does not hold for a deployment that is pinned, on premises, and evaluated before a change, where the power to intervene sits with the organisation and whoever implements it. That is a further reason the implementer cannot stand apart from the product.

Hardian’s three routes to a better loop are not alternatives. Be a partner inside the shift. Let knowledge owners audit samples and gaps from beside the loop, so that the carer does not review every answer. And be bolder only where the evidence supports it. For us that last one is limited to provenance and abstention: the sources are current and owned, and the system says when it does not know. Never a diagnosis.

Last, moving all liability away from the professional would contradict professional law, which keeps her duty of care. A shield made of a signature is wrong. A world in which she answers for nothing is wrong too.