Original Here
Introduction
The detection of content generated by AI has changed from a theoretical issue to a regulatory obligation under the European Union’s AI Act. As of August 2, 2026, it will be necessary for providers to mark their outputs in a machine-readable format. However, this top-down approach places heavy technical demands on developers and at the same time fails to take into account the practical impossibility of successfully meeting these requirements.
Imposing compliance by means of invisible signatures is a crude way of dealing with a complicated issue. The EU may end up with a system which can easily be circumvented by the dishonest while at the same time imposing on genuine providers costly and quality-reducing requirements that do not achieve the complete certainty that the regulators want.
In certain respects we already have informal versions of AI watermarks. There are some habits that have come to be linked with AI writing. The em dash, in fact, has never come under such close suspicion. Likewise, certain sentence patterns—’It’s not X, it’s Y’—as well as excessive headings, oddly balanced lists and an ever-increasing number of words that have become stereotypically ‘ChatGPT’ have also attracted attention. Of course none of these things proves anything; people have used em dashes before ChatGPT and will likely continue to do so afterwards.
A compulsory technical watermark would try to substitute those unreliable intuitive judgments with something much more systematic: an invisible signal which is deliberately put into the output by the model itself.
The issue is that while it’s relatively easy to imagine it, it’s much harder to put it into practice.
We now have techniques, called steganography, which allow us to hide information within photographs, video and audio. Text, on the other hand, is a completely different kind of medium; a photograph has millions of values which can be altered slightly without a person noticing, whereas a sentence has rather fewer options and altering any one of them can change its meaning or quality.
The gap in this area means that it is much more difficult to carry out reliable watermarking of outputs from LLMs than it at first seems – and suggests that requiring it could result in a system which is both easy to break and at the same time costly to the quality of the models it is intended to control.
Take the case of a digital photograph. A photograph is made up of millions of pixels, each of which is represented by numerical values that indicate things such as the intensity of red, green, and blue. These numbers contain a vast quantity of information.
If the red component of a certain pixel is 184, altering it to 185 would very likely be imperceptible to a person viewing the image; and if you carry out this change for thousands of pixels, you can encode a rather large amount of information without the photograph appearing to be noticeably altered.
A simple method would be to represent even numbers as a 0 and odd numbers as a 1; a computer could then conceal a binary message in the picture by modifying the value of certain pixels by one.
184 becomes 0.
185 becomes 1.
If you repeat this over a sufficient number of pixels, you can encode an identifier, a timestamp, a cryptographic signature, or some other information indicating the origin of the image.
A video also provides a great many possibilities since it shows thousands of images in sequence together with sound; there thus being a huge number of values which can be altered slightly without the viewer noticing.
There are also major issues concerning the use of mandatory steganography in photographs and videos, especially after the media has been compressed, cropped, filtered, resized, screenshotted, or deliberately altered. But I shall set those problems aside.
The key thing to note is that images have a great deal of entropy since there are millions of small numerical options open to the person who produces the image, and a number of these options can be modified without making any significant difference to what a human observes.
It doesn’t provide the same level of luxury.
The Text Problem
Imagine asking ChatGPT:
What is the capital city of Burkina Faso?
The correct answer is:
Ouagadougou.
Nothing can be concealed.
ChatGPT won’t change “Ouagadougou” to “Timbuktu” since it has to give a correct answer. It can’t alter the capitalisation as that might look odd. It also can’t add three unnecessary paragraphs just because it needs some space to put a watermark.
The same issue occurs in the case of mathematics.
What is 7 multiplied by 8?
56.
There is simply not sufficient freedom in the answer to allow meaningful hidden information to be encoded. This means that there is an immediate restriction on any scheme for watermarking text universally. Answers that are short, factual, numerical, consisting of names, translations of fixed phrases, and many other types of output contain so little entropy that it is not possible to reliably carry a hidden message.
What about longer writing?
Word Baskets
The most reasonable approach would most likely be one that we could refer to as word baskets. Suppose an AI is writing the following sentence:
The proposal is intended to boost economic growth.
There are several words the model might reasonably have used:
increase
boost
raise
improve
accelerate
Normally an AI system gives probabilities to all the possible following words. For example, “increase” might have a probability of 30 per cent, “boost” 25 per cent, “raise” 20 per cent, and so on. A system using watermarking could secretly split up the possible words into two groups.
Basket 0 might contain:
increase
raise
Basket 1 might contain:
boost
improve
accelerate
When the hidden message calls for the following encoded bit to be 0, the model will tend towards words in Basket 0; if the hidden bit is 1, the model will tend towards Basket 1. Picking one word over another has about the same function as altering a pixel value from 184 to 185. Over hundreds or thousands of words, such small decisions could build up to form a detectable statistical pattern.
The watermark wouldn’t necessarily take the form of a concrete hidden sentence; rather, a person looking at the text could consider whether the model used words from a given group considerably more frequently than would be the case naturally.
This is likely the most important difference between image and text watermarking: in the case of an image we can alter the numerical values slightly in a way that has no effect on the meaning of the image, whereas with text almost any change might affect its meaning, tone, rhythm, clarity or style.
This leaves the following tradeoff: the more strenuous you are in making the model keep the watermark, the more you disrupt its capacity to generate the best answer.
Watermarks Have a Cost
Suppose that “increase” is the best possible following word, but the watermark insists on using a word from the other basket. The model now selects ‘boost’ and that probably has no importance. But then carry out the process again and again, thousands of times.
On certain occasions the second-best word will be nearly impossible to tell apart from the first, on other occasions it will be clearly worse, on some occasions a change of word will mean having to restructure the sentence, and on other occasions the alternative will slightly alter the meaning.
The problem is particularly clear when looking at computer code.
Watermarking Code
There is a great deal of redundancy in natural language, with about twenty different reasonable ways of expressing roughly the same idea. Code is far less tolerant. A programmer doesn’t necessarily want the model to choose among function names, variable structures, library calls, formatting conventions, or algorithms since those choices serve to preserve some invisible watermark.
There are often several equivalent methods of writing a program, and there are sometimes not. A token-level watermarking system might cause the model to avoid the most efficient, readable, secure, or conventional approach simply because the presence of another valid token is able to maintain the concealed statistical pattern. That kind of writing would result in an unnatural sentence in ordinary prose. But it might cause a bug in the code.
Watermark Removers
There is another, larger problem.
Even if text watermarking does work, the person who is trying to beat it will know what the defender is attempting to achieve. The removal of watermarks from images usually involves disturbing the hidden signal while at the same time keeping the visible part unchanged. An image can be compressed, resized, cropped, slightly blurred, passed through a filter, regenerated or otherwise altered. Each of these conversions may result in some of the carefully adjusted values which carry the watermark being destroyed.
Understanding the analogous attack on text’s even simpler. Just rewrite it.
If an AI generates a 1,000-word essay containing a statistical watermark, another model could be instructed:
Rewrite it while preserving the same meaning.
A secondary model would make a huge number of new choices regarding grammar and vocabulary by changing words, reordering sentences, combining paragraphs, and getting rid of the transitions. It would therefore seriously undermine or completely eliminate the initial statistical footprint.
There are already tools that are marketed as AI “humanizers” and the reason for their existence is basically to achieve precisely this.
The Larger Mistake
There is also a more basic problem with the whole project.
We are putting in a great deal of effort to maintain a scholarly and professional environment in which we act as if machine co-writers don’t exist; yet they do and they are going to improve. They will be put into use by students, by professors, by lawyers, by programmers, and by government officials. It therefore amounts to attacking the wrong problem if one is trying to hide an invisible technical tripwire within each paragraph produced by an AI.
What we should do instead is consider machine co-writers as a given. I talk about this more general point in the article I wrote for Cyborg Scholars. The important question nowadays is no longer whether a human has received help from a machine, but rather whether the human understands, assesses, improves, and assumes responsibility for the final work.
The European Union’s initiative concerning the watermarking of machine-generated content is a misguided effort to deal with complicated social and academic issues by means of technically flawed measures. Since text does not have the same level of entropy as images, it is necessary to reduce the quality, efficiency and usefulness of AI-produced content, especially in technical fields such as programming. Furthermore, such watermarks can easily be made useless through simple rephrasing, turning the detection process into an endless and pointless arms race. Rather than trying to control the use of machine co-writers by relying on superficial deterrents, regulators and institutions should concentrate on promoting human accountability, critical assessment, and responsible interaction with AI tools.