Regulation of clinical Artificial Intelligence (AI) in the Age of Agents: Unconfined Non-Deterministic Clinical Software (UNDCS) systems for healthcare
https://www.nature.com/articles/s41746-026-02420-z
Publish Date: 2026-02-23 08:00:00
Source Domain: www.nature.com
- The authors call for refined regulations for Low-Level Machine (LLM) Clinical Decision Support (CDS) criteria, focusing on clinician and non-clinician end-users.
- The latest FDA Software as a Medical Device (SaMD) guidelines now include these considerations, aligning with international regulatory frameworks.
- All CDSS, whether Generative AI or LLM-based, fall under the current SaMD guidelines. The 2025 update refines compliance requirements based on intended use and end-users.
- A debate exists regarding whether new regulations are needed for generalized CDSS which are not tied to specific clinical indications. This depends on distinguishing between confined (pre-LLM) and unconfined AI systems.
- Unconfined AI systems, like LLMs, operate across wide semantic spaces leading to risks like error and “hallucinations” which are challenging to regulate with existing frameworks.
- The article suggests the need for a new category of regulations for novel, non-deterministic clinical software solutions developed using LLMs or other AI techniques, termed unconfined non-deterministic clinical software (UNDCS).
- New regulations may include safeguards like red teaming, guardrails, agent-agent moderation, and confined retrieval-augmented generation (RAG) to mitigate the unique risks of UNDCS.
