The people building artificial intelligence keep warning
us about superintelligent machines. Meanwhile, AI is already helping criminals
write better lies — and the defense against that is older, duller, and more
useful than any of us want to admit.
For a certain kind of AI story, the future always arrives
the same way. The lights go out. Traffic signals change for cars that are no
longer moving. Somewhere inside a windowless data center, a machine makes
thousands of decisions per second, moving faster than the people who built it
can follow, and by the time anyone reaches for the off switch it no longer
matters. We have been telling versions of this story for decades. The details
get updated — the glowing red eye becomes a frontier model, the underground
bunker becomes a GPU cluster in Virginia — but the shape holds. Humans build
something extraordinarily capable, hand it more authority than they intended,
and discover too late that they are no longer the ones deciding.
What's genuinely new is who's telling the story now. It
isn't screenwriters. It's the executives of the companies selling the
technology, testifying before legislators, publishing safety frameworks, and
describing existential risk in the same breath they describe their product
roadmap. There are serious researchers asking serious questions about
long-horizon autonomy, and those questions deserve the attention they get. But
it's worth noticing the structure of the argument: this may be the most
dangerous thing humanity has ever built, and you can subscribe for twenty
dollars a month. A warning that dramatic does something useful for the
warner. It makes the product sound inevitable.
Meanwhile, down here where most businesses actually live, AI
is doing something far less cinematic and considerably more expensive. It is
helping someone write a better phishing email.
There is no burning skyline in that sentence. There is an
accounts-payable clerk on a Tuesday afternoon, looking at a message from a
vendor whose banking details have changed. The writing is clean. The tone is
right. The names check out, the project referenced is real, and a few minutes
later a voicemail arrives that sounds an awful lot like the owner saying to get
it handled before the end of the week. Five years ago, we taught people to spot
the broken English, the odd formatting, the greeting that didn't sound like
anyone they knew. Those tells were never a security control so much as a gift —
the attacker's own sloppiness doing our detection work for us. Generative AI
took the gift back.
The numbers are no longer speculative. The FBI logged more
than 22,000 complaints involving AI-related information in 2025, representing
over $893 million in reported losses, with criminals using AI to generate
convincing messages, synthetic profiles, video, and cloned voices. More than
$30 million of that came from business email compromise alone. None of these
crimes are new. Invoice fraud, executive impersonation, social engineering —
all of it predates the transformer by a wide margin. What changed is the cost
of doing it well.
We tend to imagine that new technology creates new
categories of danger. Occasionally it does. Far more often, it takes a problem
we already had and removes the friction that was quietly keeping it in check.
The automobile did not invent the accident; it invented the accident at fifty
miles an hour. The internet did not invent fraud; it removed the requirement
that the con artist and the mark occupy the same room. Social media did not
invent the rumor, only the rumor that reaches eleven thousand people before
lunch. AI did not invent deception. It made deception cheap, fast, fluent, and
endlessly repeatable — which turns out to be the entire game.
And here is the uncomfortable symmetry at the center of all
this: the reason AI has become valuable to your business is precisely the
reason it has become valuable to the person trying to defraud your business.
Both of you are buying the same thing. One person can now do more than one
person used to be able to do.
That promise is real, and it deserves to be taken seriously
rather than dismissed. A thirty-person accounting firm in Plymouth does not
need to become an AI company to benefit from recovering four hours a week of
administrative drudgery. Neither does a construction company in Braintree, an
engineering firm off Route 3, or a manufacturer whose estimating process still
lives in three spreadsheets and one long-tenured employee's memory. When your
staff is small, getting more useful output from the people you already employ
is not a productivity abstraction. It's the difference between taking the next
project and passing on it.
But leverage has never been a moral instrument. It magnifies
good decisions and bad ones with total indifference. An employee who
misunderstands an instruction makes one mistake before somebody catches it; an
automated workflow built on that same misunderstanding makes the mistake four
hundred times, correctly, on schedule, with no one noticing that the logic was
wrong at the start. A person sends one bad email. Software sends five hundred.
Speed makes good work more valuable and gives errors much farther to travel.
Criminals figured this out early. A decade ago, most attacks
ran on volume — send enough bad email and eventually somebody clicks. The
economics rewarded quantity, and quantity is why the messages were so obviously
bad. Nobody was going to spend an hour researching a target worth a few hundred
dollars. Artificial intelligence flipped that math. Reconnaissance that used to
take a human afternoon now takes minutes: the org chart from LinkedIn, the
vendor relationships from a press release, the tone and rhythm of a CFO's
writing from three public posts and a conference panel. The message that
results isn't a mass mailing. It's tailored, timed, and plausible, and it may
arrive alongside a voice that sounds like someone you've worked with for nine
years. The result is not a smarter criminal. It's a more productive one, and
productivity compounds.
This is why the framing matters so much. The risk isn't that
AI becomes something inhuman and alien. The risk is that it becomes a force
multiplier for behavior that is depressingly, familiarly human — greed,
urgency, deference to authority, the instinct to be helpful, the reluctance to
be the person who slows down a payment because something felt slightly off. The
villain in the actual 2026 story is not Skynet. It's a Tuesday, a deadline, and
a message that looked exactly like every other message.
Until recently, the AI question inside most businesses was
narrow: is this output any good? Write the email, summarize the meeting,
draft the proposal, explain the contract clause. A human stayed visibly in the
middle. The software produced something, and a person decided what to do with
it. That boundary is moving, and the movement is the part worth paying
attention to. AI agents are being sold on their ability to perform work rather
than merely draft it — checking calendars, querying databases, updating CRM
records, compiling reports, sending messages, triggering downstream processes.
The moment software starts acting, the governing question changes shape
entirely. It stops being did the AI know the right answer and becomes was
the AI allowed to do that, and how would we find out if it did something wrong?
We've written before about treating an AI agent a little like an intern — clear assignment,
defined success, reviewed output, responsibility that grows as competence is
demonstrated. That analogy holds up better the more capable the technology
becomes, because it forces the right question about scope. Handing an intern a
writing assignment is one thing. Handing that intern your client files, your
calendar, your billing system, and permission to email on your behalf is a
different decision, and you'd think about it differently. Software deserves
that same pause: what can it see, what can it change, what does it do when it
hits something unexpected, which actions require a human signature, and is
there a record afterward?
Large organizations are already living this. A September
2026 EY survey of senior AI executives found that 36 percent reported an AI
incident or failure with materially negative impact — data loss, financial
damage, operational disruption. Among those using agentic AI, 26 percent said
they could not detect unauthorized AI agents operating inside their own
environment. And in the finding that should make everyone uneasy, 98 percent
said they had formal AI governance policies, while 47 percent admitted those processes
had been bypassed for an urgent deployment. These are billion-dollar companies
with dedicated risk functions. The lesson scales down with almost no loss in
translation: having a rule and knowing what is actually happening are two
separate achievements.
Part of what makes this hard is the strangest property of
generative AI, which is how little a correct answer and a wrong one differ in
appearance. Both arrive in clean paragraphs. Both sound composed. Both offer
plausible reasoning, and both land on your screen in four seconds. NIST uses
the word confabulation for content that is erroneous but delivered with
complete assurance, and the real hazard isn't the error itself — it's that
polish reads as competence. Humans leak more signal than that. A colleague who
isn't sure will hesitate, or hedge, or come back and say the number looks
strange and can you check it. Software just gives you the number.
What happens next is entirely predictable and not remotely
foolish. The first time you use the tool for something that matters, you verify
everything. It's right. You use it again, and it's right again, and somewhere
in the third month the checking gets lighter, because human beings cannot
function while independently confirming every assumption every day. We trust
the brakes because they worked yesterday. We trust the person who has run
payroll correctly for five years. We trust the backup job because the console
has been green every morning since we installed it. Trust is the shortcut that
makes an organization possible in the first place. The trouble is that trust
degrades into assumption, and assumptions become dangerous the moment nobody
remembers the last time they were tested.
Which brings us to the actual thesis, and it's worth stating
plainly: AI is driving the cost of producing convincing information toward
zero, which makes the ability to verify information more valuable than it has
ever been. For most of history, the scarce resource was the answer. Now the
answer is free, and the scarce resource is judgment about which answers deserve
trust.
Ask enough owners about their security and you start hearing
the same grammar. I think everybody's on MFA. Our IT company handles the
backups. Those accounts should get shut off. They know they're not supposed to
paste client data into ChatGPT. The process says it updated every record. None
of that means anything is broken. Everything may be working exactly as
described. But there is a real and measurable distance between believing a
control exists and knowing it works — what the National Cybersecurity Alliance
calls the confidence gap. Its 2026 survey of 1,000 small and midsized business
leaders found strong expressed confidence in cybersecurity even though more
than half could not confirm a clean security record, with AI adoption outpacing
governance and security tooling that wasn't always used as intended.
The gap is not theoretical, and it doesn't require a
nation-state to expose it. In September, Springfield Public Schools lost access
to critical systems after an outside group reached its network; schools closed
for days, and the disruption reached transportation, food service, instruction,
and student medical records. Everett City Hall closed around the same time
after a separate incident. Weeks earlier, Marlborough-based Boston Scientific
disclosed a cyber incident that globally disrupted manufacturing and order
processing. Three very different organizations, one common structure:
technology stopped doing what everyone assumed it was doing, and the problem
immediately stopped being an IT problem. Trucks had to move. Products had to
ship. Children needed medication. A dental office in Marshfield isn't running a
city school district, and a fifty-person engineering firm on the South Shore
doesn't need thousands of endpoints for the lesson to apply. It needs exactly
one important system that everybody assumes is fine.
There's an obvious way to take this argument too far. If
every AI output must be independently recreated, every automated step
inspected, and every system manually confirmed each morning, we've spent the
productivity gain on the auditing of the productivity gain. The goal was never
total verification. It's proportional verification — the principle that the
effort you spend checking should rise with the cost of being wrong. An AI
drafts an internal meeting agenda: read it. It cleans up two thousand CRM records:
spot-check the output and read the exception report. It produces the analysis
underneath a six-figure decision: slow down and interrogate the assumptions,
not just the arithmetic. An agent wants to modify user permissions: that gets a
log and an approval. A vendor emails new wire instructions for a $70,000
invoice: pick up the phone and use a number you already had, not the
one in the message. The backup console says last night ran clean: good — now
restore something, this quarter, and watch it come back.
That rule is not an AI rule. It's a finance rule, an
operations rule, and a hiring rule, and it long predates any of this. AI simply
raised the stakes by making the unverified thing look more credible than it
used to.
So let the larger debate continue. Maybe we will eventually
face genuine questions about machine autonomy and how much consequential
decision-making human beings should delegate to systems we don't fully
understand. Those arguments matter, and the people having them in good faith
are doing real work. But tomorrow morning, the question in front of most
businesses is much smaller and much more answerable. Did that email come from
the person whose name is on it? Did the AI get the number right? Did the agent
actually finish the job, or just report that it did? Did the restore work? Is
everyone — everyone — actually covered by MFA?
Did anybody check?
The apocalypse can wait. The phishing email is already in
someone's inbox.
