AI as “Arbitrary” Intelligence – Center for Progressive Reform

In 2016, the Arkansas Department of Health Services hired a software company to create an AI algorithm intended to assess the needs of disabled patients. The algorithm frequently made incorrect decisions that negatively impacted almost half of Arkansas Medicaid recipients.
The use of AI in the federal government has seen a significant increase, with the U.S. Government Accountability Office noting doubling of use cases from 2023 to 2024, including a nine-fold increase in generative AI applications.
Similar AI-related issues and errors have been reported at both state and federal levels, indicating a systemic problem that could become more common.
The Trump administration’s plans to incorporate AI into the rulemaking process raise important administrative law questions, particularly concerning the reliance on potentially harmful AI-generated information in regulatory decisions.
Legal scholars question how judges can conduct effective judicial review of regulatory decisions based on AI, due to the “black box” nature of AI where it is difficult to determine how an AI model makes its decisions, complicating the enforcement of standards for reasoned decision-making under the Administrative Procedure Act.