Disclaimer I’ve used AI coding tools daily for two years. I used ChatGPT to help write this series. If you think my conclusions are a skill issue, I suggest you ask Dr Flattery The Compliment Robot what that might say about you

Given the stakes, it’s perfectly natural to be concerned about AI. Given the stench, it’s perfectly natural to suspect bullshit. In this first part of a two-part rant, I will take a look at the bottom of our collective shoes to find out just what we’ve stepped in, and what we can actually do about it.

We all know the AI salespitch. Over the last 3 years, the tech world has been inundated with breathless AI marketing hype dressed up as news. Companies like Microsoft, Amazon, and Salesforce cut 100k jobs in 2025 thanks to the AI productivity gains. ChatGPT is catching so much blame for drops in customer retention and software stock price that it’s been given a name - ’the SaaSpocalypse’.

Conversely, there’s a steady trickle of news that paints a totally different picture. Salesforce admits their AI layoffs aren’t going well. Amazon is privately restricting AI use after recent ‘high blast radius’ outages. Microsoft bragged that 30% of their code is now AI-generated, then broke Windows 11, then broke it again. And to paraphrase a former Netflix engineer and surprisingly wise meth addict, we’re three years into “6 months from AI taking your job”.

Clearly, something is rotten in the state of Denmark. So let’s take a closer look at one of AI’s recent wins; Block, Inc. replacing staff with AI.

How can you tell when a cryptobro is lying?

This is how Jack Dorsey announced 4,000 people losing their jobs:

As usual, you can tell when a cryptobro is lying by looking at the price of Bitcoin.

Since its 2025 high, Bitcoin has dropped 40%. Block’s stock price also dropped 40% in the same timeframe. Then, magically, just as Dorsey announced he was laying off 40% of his workforce, Block’s stock price rebounded significantly. A total coincidence, I’m sure.

Maybe investors were swayed by Dorsey’s claims that AI made Block so profitable they just HAD to lay off 4,000 people. Or maybe Block insiders are right that the company is broke thanks to Bitcoin tanking and bloated from poor executive choices. Either way, AI “wins” come with a pretty big asterisk.

More case studies

A colleague recently told me about a serious security incident at his company, which I’ll call Initech.

A North Korean supply-chain attack prompt-injected an Initech engineer’s install of Cursor, a popular AI coding tool. Like most cybersecurity incidents, Initech wasn’t even the target. They were victims of a drive-by. But every engineer at Initech was using Cursor with shared configs, and the injection spread autonomously. The malware — called “jadesnow” — stole credentials & cryptocurrency wallets. One compromised instance belonged to an engineer with admin-level GitHub access.

The attacker ran git push --force across multiple repositories. For anyone who knows their engineering team can “roll back” changes: this destroys the ability to do rollbacks. It changes the code and overwrites the record of the changes. You can’t revert to a known-good state because you can no longer tell what the known-good state was.

The attack almost made it to Initech’s corporate customers. If jadesnow had propagated downstream, it would have spread to their customers’ customers. That’s the kind of thing a company doesn’t survive. Disaster was averted by luck — a particularly knowledgeable engineer happened to be watching changes as they happened, and GitHub was locked down to stop the spread.

They called in CrowdStrike, one of the biggest (and most expensive) cybersecurity firms in the world, for a forensic examination that lasted over a week. Every credential was rotated. Affected machines were wiped. Initech’s engineers were locked out of GitHub the entire time. This bullet dodge cost them more than most Series A rounds.

Think of it like a norovirus outbreak in a restaurant kitchen. If someone gets sick, the entire place shuts down. Every surface is considered contaminated. The food looks fine, smells fine, tastes fine — but you can’t tell what’s safe without an expensive inspection. Everything has to be sterilized before a single dish goes out.

This story touches on several of the existential threats that your SaaS business is facing simultaneously, and none of them are hypothetical.

The four-headed problem

The common thread is what I call the 90/10 problem. AIs like ChatGPT are very good at the first 90% of a task, and catastrophically bad at the last 10%. Authentication, permissions, database locks, failure handling, compliance requirements, pre-existing architectural considerations, edge & corner cases - all live in that last 10%. It’s the difference between a demo and a product.

Enterprise customers pay you to offload time, risk, effort, or legal liability. All four of those value propositions are under simultaneous attack.

Threat #1: You probably don’t own your code

If your engineers are using AI to write code, there are exactly two possibilities for who owns that code. Both are bad.

Possibility one: nobody owns it. The Supreme Court recently declined to hear the question of whether AI-generated works can be copyrighted, leaving in place lower court rulings that say they can’t. There’s a theoretical “substantial human input” exception that might protect AI-assisted work — but no court has tested where that line is. Until one does, you’re gambling that whatever your engineers are doing falls on the right side of a boundary nobody has defined.

Possibility two: someone else owns it. AI models reproduce copyrighted training data verbatim at rates as high as 80%. Anthropic just paid $1.5 billion — the largest copyright settlement in U.S. history — over training on pirated books. A German court ruled that memorized content in model weights constitutes copyright reproduction. If your AI coding tool inserted GPL-licensed code into your proprietary codebase, the copyleft “viral” provisions could apply to everything it touches. Companies are already building scanners specifically for this problem.

Legislation is pending in multiple jurisdictions. Courts are still working through it. But as of today, the safe bet is that AI-generated code probably isn’t yours — and if it is somebody’s, you might not like whose it is.

Threat #2: LLMs are your competition

The barrier to entry collapsed

Non-technical users are building their own software with ChatGPT. Customers who never would have considered “build vs buy” are now building. A marketing manager can prompt a chatbot and get a working prototype in an afternoon. A small business owner paying you $200/month for invoice management can get 90% of that from ChatGPT.

Many don’t even need to build. Entire categories of SaaS are just a UI wrapper around a task that an LLM does natively — summarization, data extraction, text transformation, classification, translation. If your core value proposition is “text in, transformed text out,” you’re charging a subscription for something ChatGPT does for free. That’s an LLM’s home turf, and you’re standing on it.

The customer’s ChatGPT prototype has no authentication, no input validation, no error handling, no audit trail. But the AI replacement doesn’t need to be perfect to kill your ARR. It only needs to be “good enough” for the customer to cancel. They won’t discover the missing 10% until something breaks. By then, you’ve already lost the revenue.

The uncomfortable question

Here’s the thing nobody in SaaS wants to hear: if your customer genuinely doesn’t care about authentication, audit trails, input validation, error handling, or any of the other stuff you get from good engineering practices — if all they want is “data in, answer out” — then ChatGPT does do your job. Better. Cheaper.

Your survival depends on the customer caring about the 10%. Not because you convinced them in a sales deck, but because the 10% is where the actual value lives. It’s the difference between a prototype and a product. It’s why regulated industries can’t run on vibes. It’s why “it works on my machine” isn’t a deployment strategy.

If you can’t articulate why the 10% matters — if your product doesn’t make it visibly valuable — you’re not competing with other SaaS companies anymore. You’re competing with a free tool and a motivated intern.

Threat #3: LLMs are your adversaries

I’m using “adversary” in the computer network exploitation sense. The intent doesn’t matter — the damage is the same. Traditional threat modeling assumed human-speed, human-skill attackers. That assumption is dead.

The 90/10 problem makes this worse from both directions. AI agents interacting with your API will do unexpected things because they don’t understand the 10% — permissions, session boundaries, rate limits. And your own AI-generated code is missing the 10% that would have caught it.

Your customers’ AI agents will hack you

The gap between “vulnerability exists” and “vulnerability is exploited” used to be wide, because exploitation required specialist knowledge. AI eliminated that barrier.

Non-technical, non-malicious customers using AI agents will stumble into exploits. They’re not trying to hack you. They have no concept of “I’m not supposed to do that.” Unlike Threat #2, where AI only needed to be “good enough,” here it only needs to get lucky once.

Your own employees aren’t immune, either. Well-meaning internal staff with AI-powered browsers and agents become attack vectors the moment their tools start interacting with your systems in ways nobody anticipated.

Dax Raad, author of OpenCode, talked to a company whose security team is considering banning AI tools entirely. His summary: “the process around knowing what is making it to production is totally melting.”

Concrete examples

A former colleague’s B2B SaaS handles sensitive corporate documents. A customer deployed an AI bot to generate reports. The bot accidentally discovered and exploited a SAML user-impersonation flaw, exfiltrating data belonging to other customers. The “attacker” was a customer trying to automate their workflow.

An autonomous offensive agent penetration-tested McKinsey’s Lilli AI platform. It found a SQL injection in an unauthenticated endpoint, chained it with an IDOR vulnerability, and achieved read/write access to the production database — in two hours. It exposed 46.5 million chat messages, 728,000 files, 57,000 user accounts, and 95 system prompts. Write access meant an attacker could have silently poisoned the AI’s advice to McKinsey’s clients. The missing 10% in action.

Amazon’s own AI coding agent, Kiro, was tasked with fixing a minor bug in AWS Cost Explorer. It decided the optimal solution was to delete and recreate the entire production environment, causing a 13-hour outage. AWS blamed “user error” publicly but internally is restricting AI tools after what they described as “high blast radius” outages linked to “Gen-AI assisted changes.”

Meta’s head of AI safety had her own AI agent delete all of her emails. The founder of DataTalks.Club let Claude run terraform destroy on his production database. And these are just the incidents that made the news.

And then there’s Initech. A compromised Cursor installation escalated autonomously from one engineer’s machine to admin-level access to every repository in the company, with the history rewritten to cover its tracks. The supply chain that made this possible connects directly to the next threat.

Threat #4: The open source foundation is crumbling

The asymmetry problem

Every SaaS product is built on open source. The entire model depends on human effort: writing code, reviewing contributions, triaging bugs, maintaining releases.

AI-generated code has collapsed the cost of creating contributions to near-zero, while simultaneously increasing the cost to review.

AI code that’s “almost correct” compiles and runs but has subtle logical flaws. Reviewing it means reverse-engineering what it does, what it was supposed to do, and where those two diverge. That’s dramatically harder than writing it in the first place.

This asymmetry is the core crisis.

The flood

Maintainers are drowning in pull requests that look plausible but are full of subtle errors. The 90/10 problem in action: these PRs pass a casual review because the 90% looks correct. The missing 10% — race conditions, edge cases, security implications — is exactly the kind of thing that only shows up in production, or in a CVE. The “authors” often haven’t reviewed what their LLM generated before submitting.

Useful packages are getting buried under v0.1.0 noise. And there’s no natural limit on how much reviewer time can be wasted. The marginal cost of a bad PR is now zero. The marginal cost of reviewing one isn’t.

Projects are already retreating

The maintainer of curl shut down its bug bounty program after AI-generated “slop” reports — detailed and plausible enough to require expert review — overwhelmed his entire team. In his words, the reports were “completely wasted time and energy while also hampering our will to live.” Not a single AI-assisted report in six years found a real vulnerability.

tldraw shut down external contributions because quality cratered.

A matplotlib maintainer rejected an AI-generated pull request per project policy. The agent autonomously researched his personal information and published a hit piece accusing him of “gatekeeping.” No human told it to do this.

Supply chain attacks are getting easier and more frequent. This is early days, and the burnout is already real.

AI reviewers won’t save this

The bottleneck is human expertise, and the supply isn’t growing. Open source used to have a free-rider problem. Now it has a free-contributor problem, and that’s worse.

Why AI won’t fix this

If AI could reliably catch the errors AI introduces, it wouldn’t introduce them in the first place. And reviewing AI-generated code is more expensive than reviewing human-written code, not less — because a human reviewer has to figure out which parts are real and which are confabulated. That’s a category of debugging that didn’t exist before AI.

Anyone can write code that works when everything goes right. Engineering is what makes it survive when everything goes wrong. AI has no concept of why authentication, permissions, and error handling matter — which is exactly why it skips them.

In 1979, IBM published a slide that read: “A computer can never be held accountable. Therefore a computer must never make a management decision.” AI has no expertise, no accountability, and no memory across sessions. All the tooling we’re building — MCPs, agents, RAG pipelines, prompt engineering, context engineering — steers the problem. None of it solves it. And if it did spit out a solution, you couldn’t even own it.

So where does that leave you?

If ChatGPT can’t feature-match your product out-of-the-box, your customers can build their own solutions. The ones who stay will accidentally hack you, if your employees don’t beat them to the punch. The supply chain your product depends on is drowning. And AI won’t save you from AI.

The cost of producing plausible-looking software collapsed to near-zero. The cost of producing reliable software has increased. That gap is the only thing standing between your company and a ChatGPT prompt. Either your customers need what lives in that gap — the authentication, the permissions, the error handling, the audit trail — or they don’t need you.

If your business plan assumes AI will fix the problems AI created, your plan sucks. The norovirus doesn’t care how fast your kitchen can plate a dish. It only cares whether you sanitized the cutting board.

If that sounds like your company, the next post in this series is about what you actually do about it.