How FOMO Is Turning AI Into a Cybersecurity Nightmare
How FOMO Is Turning AI Into a Cybersecurity Nightmare
https://www.inc.com/nick-selby/how-fomo-is-turning-ai-into-a-cybersecurity-nightmare/91261473
Publish Date: 2025-12-31 11:10:00
Source Domain: www.inc.com
Using an unordered list, summarize the following article with between 4 and 8 key points. After years of helping companies navigate technical challenges, my colleagues and I have observed a troubling pattern across several verticals and sectors. AI implementations fail not because the technology “doesn’t work,” but because executives rush into deployment without addressing the fundamental operational concerns of the technology. AI tools are not as safe and predictable as traditional enterprise software. This approach introduces risks that can cost millions. The Problem Isn’t the Technology…I’m not an AI skeptic. An Inc.com Featured PresentationFor many business uses, executives can strike the correct reward-to-risk balance by asking the right questions upfront, limiting potential harm if something goes wrong, and insisting that both business and technical teams monitor how the software performs. There are good AI use cases that, in some circumstances, can transform business operations. For example: a mid-sized enterprise client using a website chatbot to connect potential customers with an account executive reported significant results: thousands of dollars in weekly bookings, and hundreds of thousands of dollars in their weekly pipeline. They found that people interacting like this tend to become “high-intent leads, faster” than through any of their other channels. There’s serious pressure on executives to implement AI software for reasons that include a genuine belief that, as in the previous example, AI tools can increase customer engagement and lower costs. CEOs also are subject to plain old Fear Of Missing Out: The self-fulfilling cycle of advancing competitiveness creates very real pressure from investors and boards: when your key competitors are going “AI-First,” the pressure can be intense to jump aboard a speeding train. So it’s understandable that we see a lot of urgent CEO calls to, “Hurry up and AI the Everything!” Less justifiable, though, is the scant attention being paid by executives to effectively consider the risks of any given AI software implementation. That’s where companies consistently stumble. The only way to change this is for senior executives to reframe their understanding of the risks at hand. …It’s the Implementation. The risk decisions made by executives must derive from a cross-functional risk management approach that enables them to consider all aspects of the business holistically. An AI tool can be benign or advantageous from an operational perspective, but harmful from a cybersecurity perspective, a legal perspective, or a cost perspective and vice versa. Consider the 2025 security breach of Drift, which affected more than 700 customers of Salesloft’s AI-powered B2B chatbot product. Despite breathless claims about Drift’s “conversational AI capabilities,” the breach probably stemmed from basic information security failures unrelated to AI. The damage derived not from its AI functions but from the access privileges customers had granted to the Drift agent within the Salesforce and Google Workspace clouds: once they had pilfered the customers’ credentials to these critical internal systems and logged in, the criminals simply asked the agent for the data they wanted.Based on Salesloft company statements, the security failings appear to include absent or improperly configured multi-factor authentication on source code repositories, and hardcoded and improperly stored credentials (our understanding is less than complete because Salesloft security communications have themselves been vague, something that has helped neither their customers, nor the industry). This incident reflects a broader industry pattern: executives considering buying AI tools have focused on promised features but forgotten about due diligence of the AI-vendor’s core security practices. As we will see, some healthy skepticism about how vendors approach core information security and legal tenets and practices can reveal an accurate risk picture to inform CEO decisions. Legal and Security Risk Management is Harder Because AI Makers Have Re-Defined Industry Terms. Let’s remove the mystery and magic surrounding AI hype. “I refer to it as software,” says EPSD Advisor and National Academies Cyber Hard Problems committee member Wendy Nather, “because the term ‘AI’ tends to enchant and ensorcell.” Even companies trying to perform responsible due diligence face hidden challenges. Most of the procurement, information security, IT, and legal teams that comprise companies’ risk management function face newly complex assessment challenges, all while foot-tapping executives urge them to approve new AI implementations quickly in the name of competitive advantage. A core issue is confusion over what words mean in the “AI safety” context. As researchers Heidy Khlaaf and Sarah Myers West have documented, AI vendors have co-opted standard information security and risk management terms, so familiar terms now carry different meanings in different contexts. When AI vendors use terms like “red teaming” and “vulnerability management,” your security and legal professionals hear familiar language and assume rigorous security testing has occurred. Often, that assumption is wrong. In traditional IT security, “red teaming” means organized attackers actively attempting to breach your systems and steal data so your defenders can fix security holes before criminals exploit them. In AI contexts, it typically means a kind of content moderation testing, like checking whether a chatbot refuses to generate racist content or bomb-making instructions. Similarly, “vulnerability management” in cybersecurity means identifying and patching security flaws that could expose your systems to attack. In AI contexts, vendors often use it to describe managing their risks of biased outputs or inaccurate responses. Both interpretations address important concerns, but protect against different risks. One safeguards your data and systems from breach. The other prevents the vendor’s tools from generating inappropriate content. When vendors blur these lines, an executive might reasonably believe they are buying a secure product when testing only covered content appropriateness. Additionally, your legal teams might review a vendor attestation stating they have conducted “identification, estimation, and evaluation of known and reasonably foreseeable risks to health, safety, and fundamental rights” and assume this covers traditional product liability or workplace safety concerns. But in AI contexts, “health and safety risks” often refers to things like algorithmic bias or content that might promote self-harm, not safety or regulatory compliance in the traditional sense. “Misuse” might mean prompt injection attacks or generating harmful content, not the contractual or regulatory violations your lawyers typically assess. This linguistic sleight of hand can create dangerous gaps: the same words can refer to radically different risk universes. The lesson: risk managers must demand clear, plain-language definitions of terms to verify they properly understand what each vendor term means and how it is used. When AI Variability Becomes a Business Risk A key thing some executives fail to consider is generative AI software’s inherently non-deterministic nature. “Non-deterministic” means that, given the same inputs, the answer can and often will be different at different times. That’s only sometimes acceptable: it may not matter much for that bot making sales appointments, but it can create profound risk elsewhere. Companies allowing unsupervised chatbots to answer customer questions about specific product terms face customer frustration, and legal judgments, when those answers are wrong and create harm. Consider the Air Canada case. The airline’s chatbot erroneously told a passenger he could apply for a bereavement fare discount after purchasing and taking his flights. He couldn’t. Air Canada declined to honor the discount, and in 2024, the Civil Resolution Tribunal of British Columbia ruled against the airline, forcing it to honor it. The cost to Air Canada of this ruling was not material, but the customer trust and legal precedents were profound. Air Canada seems to have since discontinued using the tool. Non-deterministic systems are even more dangerous in software engineering operations, where 68 percent of developers report using AI tools daily or weekly. As engineer Chris Swan from Atsign says, the practice of IVO (Immediately Verify Output) has emerged as one of the most effective ways to deal with AI randomness. But, Swan warns, “This raises the specter of Almost Right Output (ARO). It doesn’t withstand thorough scrutiny, but in most organizations, ARO gets waved through as looking ‘good enough.’” (See Addy Osmani’s essay, The 70 Percent Problem, for both related and orthogonal concerns.) As Wendy Nather says, “At some point it should be deemed irresponsible to deploy stochastic agents (like AI software) to perform essential functions that you can’t fully predict or test.” The only way to know whether AI software is right for an application is through structured, deliberate testing and analysis of whether unpredictable, possibly incorrect outputs can create customer or reputational harm. This is likely far down in the weeds, but CEOs need to insist that the risks uncovered by this testing be made clearly, and in plain business terms so that executives can understand the choices the test results can inform. And CEOs must demand this business-level analysis from their risk management teams before approving any AI implementation, and demand a human in the loop to validate (as is done with IVO) before committing AI-generated code or text to customer-facing products. Three Critical Control Areas Before any AI integration, companies must implement three foundational control families: 1. Risk Enumeration and Threat Modeling Former US Navy SEAL commander Clint Bruce asks the essential question: “What is the cost of wrong?” This gives CEOs a straightforward business decision. When adopting AI software that will access critical systems and data, risk management professionals should assume the tool is compromised and answer the question: “How bad would this be?” Procurement, information security, legal, and IT teams must identify what data the tool will access, what could go wrong, and how it will integrate with existing systems. This practical mapping of data flows, access patterns, and failure modes defines the blast radius you’ll face if problems occur. The answer must be realistic and measurable, and, again, phrased in plain business terms so executives can make risk-reward decisions. And this operation should be repeated after the vendor makes major upgrades, such as introducing a leap from version X to Y. 2. Blast-Radius Reduction Presume any software (including AI) that’s introduced to your systems and data stores is compromised. To reduce the likelihood and impact of catastrophic failure, CEOs must limit AI tool access to only the data essential for it to function and achieve the desired outcome (vendor recommendations maximize connections to your data). Treat as essential and mandatory robust data classification, strict permission boundaries, and monitoring systems that detect unusual access patterns. The decisions about minimum viable access should be documented in an Architectural Decision Record and included in the risk map and threat model document, along with who made the decisions, when, and the context in which they were made. 3. Instrumentation and Alerting Absent vendor notification, most companies discover AI security issues months late, if at all. If you can’t observe what your AI tools are doing, you can’t secure them. This requires comprehensive logging, real-time monitoring, data loss prevention, and automated response capabilities. Your monitoring must distinguish human actions from AI agent actions. Test these capabilities to ensure they work when needed. All this applies to internally built tools and applications, not just third-party implementations. Emerging protocols like Model Context Protocol (MCP) often de-emphasize information security basics in favor of AI features and functionality. Code reviews of clients’ internally built MCP servers revealed uniformly poor security practices: dated libraries, insufficient authentication and logging mechanisms, and trivially exploitable vulnerabilities. The AI functionality worked perfectly, but the server components were security minefields. Your custom applications, systems and data integrations, API configurations, and deployment choices all contribute to the actual attack surface. The Business Case for Thorough Consideration I’ve long said that savvy executives recognize that rushed deployment creates technical debt that compounds like high-interest credit card debt. The same applies to AI: every shortcut you accept today will cost exponentially more to fix later, constraining your ability to deploy AI effectively across the organization. The alternative isn’t avoiding AI, it’s treating the implementation of AI software as a business strategy problem that requires the same systematic approach you’d use for any significant technology transformation. This means clear definitions, comprehensive planning, and robust measurement systems that translate technical realities into business metrics. Ask the Right QuestionsAI errors and information security breaches are business risks, not IT problems. As such, CEOs should require business leaders and risk management professionals to document strong business cases and risk analyses before deploying generative AI tools, starting with: “Is this a good use case for AI software?” With the AI market changing rapidly and buyer-beware rules unclear, take an old-school approach to answering this question:
Establish written business goals (and undesirable outcomes) that state the problem the tool will solve, success criteria, and measurement methods.
Document each tool’s blast radius; what damage would occur if the tool is compromised? Determine this through proper integration planning, configuration, and mapping acceptable data flows that minimize data access and maximize visibility into what these tools do in your systems and with your data.
Implement and test automated detection and blocking of suspicious activities.
Create conditions for effective cross-functional incident response when problems arise.
Don’t skip over these. In enterprise technology, speed comes from discipline, not shortcuts or from a rush to implement. The Future of AI in the Enterprise Companies that thoughtfully implement AI may enjoy a genuine potential competitive advantage. The key is in executives’ understanding that “thoughtful implementation” means insisting on proper foundations and not getting dazzled by the bright lights. This requires executive education, great organizational communication, cross-functional planning and training, and recognition that risk management isn’t a barrier to AI adoption but instead enables sustainable, scalable AI programs that actually deliver the competitive advantage that leads to business value. Companies that get this right will realize AI’s transformative potential. Companies that don’t will spend years and millions of dollars cleaning up the mess. Go inside one interesting founder-led company each day to find out how its strategy works, and what risk factors it faces. Sign up for 1 Smart Business Story from Inc. on Beehiiv.
