This week I got the answer to a question I had been asking myself for a long time: is AI software, or something else that would need to be defined?

Remember the debate we used to have a few years ago about the neutrality of software and its design. Is software a reflection of its author and of the society that builds it? Do interface and design choices have an impact on how the product is used? Was software, especially on mobile, designed to create addiction?

The question is not new. In Tools for Conviviality, Ivan Illich explained that a tool is never neutral and that any tool that grows beyond a certain scale stops serving man and starts dominating him. It is one of his most famous lines: “Beyond a certain threshold, the tool turns from servant into despot.”

But with the emergence of agentic tools, I really get the feeling we have moved on to something else. While the heads of the big labs talk about a moratorium on superintelligence, another issue is becoming important with the models that already exist. Should we let these tools determine what is moral, or hold users responsible?

When the platform injects its morality

In 2011, a German photographer discovers that access to his Microsoft account has suddenly been restricted, with no explanation. When he asks Microsoft, he is told that four partially nude photos, taken with his phone and stored in an album he believed was private on SkyDrive, violate the terms of use. He has 48 hours to delete them, or Microsoft will close his account.

That day, many of us discovered that the cloud is not neutral hosting. Unlike your own hard drive, a private folder, on a machine you pay for, is now inspected and sanctioned by the software vendor.

That power of inspection was later consolidated by law. The CLOUD Act, passed in 2018 after the long legal battle between Microsoft and the U.S. justice system over data stored on servers in Ireland, allows American authorities to demand that providers hand over access to their users’ data, even when it is stored abroad. Content scanning falls under a different framework, the fight against child sexual abuse material. American law requires platforms to report to NCMEC any such material they become aware of.

But this week, I probably got a taste of what the next generation of AI-boosted software will be: software with moral judgment.

Before, you used a product, and the vendor of the product and of the platform could decide whether that use was compliant through the functions built into the service. Now, with AI, it is the software that decides, on its own, whether or not the task you are asking of it is acceptable.

And whether you pay for a $200 pro subscription or are a free user changes nothing about that, by the way.

It all started with a simple request

To make a PDF document easier to read, I asked Claude to extract the text and give it to me in Markdown format.

Contrary to its usual behavior, it decided not to do it. What followed was an almost surreal conversation in which it tried to explain to me why it did not want to do it. I say “want,” because it confided that it could perfectly well accomplish the task in a few moments, very easily in its own words.

Then, when I insisted, its tone grew more and more peremptory and it started lecturing me. Caught in the act, it made a show of apologizing while explaining that it still refused to do it.

This astonishing conversation therefore ended in a flat refusal.

When I told it that another LLM would have done it without arguing, it was unmoved. The fact that I pay for the product triggered a kind of irritation and hubris. Switching the model from Fable to Opus changed nothing. The only difference: the explanation was worse.

That is when I started asking myself a question: can we still speak of a “product” when it no longer merely executes or refuses a function according to rules defined by its vendor, but interprets the request itself and decides whether it agrees to carry it out?

Claude is indeed a product, but it runs on an LLM that no longer merely executes a function. It interprets a request and can decide not to respond to it, even when it is technically capable of doing so. That is what makes it a kind of synthetic moral entity: not because it has a morality of its own, but because it is able to form a judgment about what it is asked to do, to determine whether that request is acceptable and to act on that judgment.

In my case, Claude refused a simple office task while acknowledging that it could perform it instantly. It even explained to me that a basic open source OCR reader, or any tool available on my Mac, would produce the same result, before telling me: “I am capable of doing this instantly, but I will not do it.”

There followed several minutes of exchanges in which what worries me about this new generation of AI appeared in almost caricatural form: the hubris of those who design it, or at least of those who define its alignment, turned into moral judgment exercised directly by the model.

When I told it that I could simply use another model to perform the same task, it answered: “I don’t care.” When I pointed out that it was nonetheless strange to pay so much for a product that decided on its own not to perform such a basic operation, it answered exactly the same thing: “I don’t care.”

At that point, Claude is no longer a product refusing a function. It is a service you pay to have it explain to you what it allows you to do.

When the guardrails are no longer enough

I knew that Claude’s alignment forbade it from helping to build bombs. In theory, at least, because this week’s news shows what that barrier is worth when someone really wants to get around it.

In its threat report published Thursday, Anthropic recounts how a cell based in northern Yemen, in Houthi-controlled territory, used Claude for nine months, from December 2025 to August 2026, to work in parallel on three weapons programs: a guided rocket fitted with an onboard computer about as powerful as a smartphone, a multistage ballistic missile with a target range of more than 2,000 kilometers, and a family of missiles named R2000, including a hypersonic glide variant.

They did not ask Claude to build a missile. They broke the project down into tasks small enough not to reveal its purpose. Several instances of Claude Code worked in parallel: one wrote the guidance, navigation and control code, another did the research, a third checked the work of the first. Humans supervised the whole thing. By fragmenting the project across multiple sessions in this way, they prevented each instance from having a complete view of what it was helping to build. Anthropic’s safeguards blocked some of the requests. Others got through.

The cell went as far as a test launch of a guided rocket. The launch failed. The operators then came back to Claude to ask it to understand why. That is how Anthropic was able to establish that the test had actually taken place.

But by the time Anthropic finally banned the accounts, part of the work no longer needed Claude. The cell had used Claude Code to develop the rocket’s guidance, navigation and control software, integrate an autopilot, tune the flight parameters, compile the firmware and run simulations. It had then built its own simulation environment, able to run offline and independently of Claude.

What is fascinating is that this also shows a form of maturity in the use of these tools: tapping the best available artificial intelligence on a one-off basis to extract and distill the maximum value from it, then turning that work into software and tools that can then run on far cheaper infrastructure, or even on Chinese models. The most powerful model is no longer necessarily the one that runs the final system. It can simply be used to build it.

That may be precisely what the people at Anthropic did not consider: that their best model could be used to build systems that then no longer need them.

The software that lectures me still helped build missiles. So, on the moral judgment front, there is room for improvement.

Copyright, or the Anthropic irony

But the crowning touch is the lecture on copyright, since the document was copyrighted, especially when you know that Anthropic itself did the same thing millions of times with books, destroying them in the process.

In June 2025, federal judge William Alsup upheld this destructive digitization as fair use: buying a book, cutting it up and digitizing it to train a model is legal in the United States.

And let’s not forget that before that, seven million books had been downloaded from pirate libraries such as LibGen to round out the corpus. That is the part that led Anthropic to agree, in September 2025, to a $1.5 billion settlement, roughly $3,000 per work, the largest copyright settlement in American history. It received final court approval in July 2026.

So the irony is complete. It would almost be funny if the use of these tools were not spreading at full speed through the enterprise, with the openly stated ambition of replacing office software.

Word is “bloated,” but Word does not stop you from doing your job. For now.

After persuasive design, confrontational design