Thank you to those who have signed up since my last update. I have relocated to Austin and am hitting the ground running. Please share if you know of anyone who would be interested.

Last month I was fortunate enough to present my Harvard Policy Analysis Exercise “A Cognitive Security Verification Framework” at BSides San Antonio, a cybersecurity conference.

Podcast - SenTe with David

In case you would prefer the articles in spoken form rather than written, I’ve started giving a rundown of my pieces and will post them on your podcast platform of choice - It’s a work in progress and I’m smoothing out the rougher edges with each iteration. Feel free to follow and share.

Youtube

Spotify

Apple Podcasts

Podcast Addict

RSS Feed

Writing Roll-Up

Broad Policy Articles:

1. Loose Lips Sink Ships- AI and Inference Risks
   Seemingly harmless comments can reveal sensitive information when combined with other public or private clues. AI makes this recombination faster, allowing adversaries to reconstruct secrets without accessing any single classified document. Organizations should act according to principles of paranoid secrecy which were established as far back as WW2. Substack.

2. Moving Up the Abstraction Ladder
   Software development has progressed from machine code to higher-level languages that let people accomplish more without managing every technical detail. LLM coding agents represent a continuation of this long-time trend, translating ordinary language into software and giving users greater agency over complex systems. Substack.

SenTe Focused Articles:

1. What Does a Similarity Leak Look Like?
   A similarity leak occurs when sensitive information is exposed through a paraphrase, translation, summary, or closely related expression rather than an exact textual match. Traditional filters may miss it because the words have changed even though the underlying meaning remains the same. Substack.

2. Why We Named it Joseki.
   In Go, a joseki is a reusable sequence of moves that works only when applied with an understanding of the broader game. The name reflects our belief that AI instructions and wrappers should be reusable and portable, but always evaluated within their surrounding context. Substack.

On Request:

• Prototype testing of the Moyo information space mapper. Find leaks of your secrets (classified, proprietary, personal) in public LLMs (ChatGPT, Claude, Qwen, Kimi, Grok, etc).

• SenTeGuard Pilot - protect yourself or your organization from leakage of valuable information through air-gap capable semantic guardrails. Substack

• Let’s Talk:

  • Private - How can your organization more effectively secure your secrets in the LLM era through implementation of the Cognitive Security Verification Framework (CSVF) and by other means.

  • Public - How to craft tech-positive, pro-future, big-tech skeptical market-oriented policies of abundance in the LLM era.

David

Original Here

Traditional cybersecurity tends to treat a secret as an object. It is a document, password, database record, piece of source code or collection of classified files. Once the object has been identified, security teams can restrict access to it, monitor where it travels and search for exact copies leaving the organization.

Large language models complicate that arrangement because a secret is not only an object. It is also an idea.

Imagine that a company is preparing to acquire a smaller competitor. The official acquisition plan is tightly controlled. However, the company has also posted several new jobs in the competitor’s city, requested regulatory advice from an outside law firm, moved members of its integration team onto an unnamed project and scheduled unusual travel for senior executives. None of these facts is necessarily sensitive on its own. Combined, they may reveal the acquisition.

Before generative AI, reconstructing that conclusion required a patient analyst. Today, a person can place those fragments into a model and ask it what is happening. The model may never retrieve the confidential acquisition plan. It may nevertheless produce a description that is functionally indistinguishable from it.

The secret has leaked through similarity.

Security in Meaning Space

Most conventional data-loss-prevention systems operate in what we might call token space. They search for a Social Security number, a project codename, a classified marking or a sequence of text copied from a sensitive document. These controls are useful, but they are built around an assumption that leaked information will resemble its source at the level of words or characters.

LLMs operate primarily in meaning space. They can translate, summarize, abstract, paraphrase and recombine. A user does not need to type, “Project Nightfall will launch on October 17.” He might instead ask the model to describe “the initiative scheduled to begin in mid-October after the new supplier receives final approval.” The words are different. The meaning may be nearly identical.

Leak through similarity occurs when information crosses a security boundary because it is semantically close enough to protected information to create the same practical harm. The output does not have to be an exact copy. It only has to tell the recipient substantially the same thing.

The important shift is that confidentiality can no longer be treated as a binary property of a file. Protected information occupies a neighborhood of related statements. A model can land inside that neighborhood through paraphrase, approximation or inference without reproducing the original source.

The Cognitive Security Verification Framework (CSVF) describes this broader problem as semantic leakage: the disclosure of protected meaning through paraphrase, translation, summarization, abstraction or inference. It also distinguishes semantic leakage from cross-domain inference, where individually permissible fragments become sensitive when combined.

“Close Enough” Can Be More Than Enough

Security professionals may reasonably ask how similar an output must be before it becomes a leak. There is no universal threshold.

For an intelligence organization, an approximate description of a facility’s purpose may be damaging even when its exact capabilities remain unknown. For a pharmaceutical company, identifying the likely mechanism of action behind an experimental drug may eliminate years of uncertainty for a competitor. For a technology company, a rough reconstruction of an internal system prompt or evaluation strategy may contain most of the commercially valuable insight.

This means leak through similarity should be evaluated according to operational equivalence rather than verbal equivalence. Would the recipient be able to make substantially the same decision, reproduce substantially the same capability or avoid substantially the same research cost after receiving the model’s output? If so, the fact that the model used different words provides little comfort.

The problem becomes more serious as AI systems gain access to longer context windows, persistent memory, internal repositories and external tools. Each connection expands the collection of fragments from which the model can build a conclusion. Security teams therefore need to consider not only what the model can retrieve during one prompt, but what it can accumulate across a session or across many sessions.

How the SenTeGuard Stack Addresses Similarity Leakage

Leak through similarity is not a single-product problem. It exists before deployment, during retrieval, at runtime and after a system changes. The SenTeGuard product family approaches those stages as parts of the same cognitive-security stack.

SenTeGuard: Runtime Enforcement and Batch Scanning

SenTeGuard sits between users and model endpoints, inspecting prompts and responses before information crosses the boundary. Conventional pattern matching remains important, but the platform adds semantic analysis intended to recognize when a user is describing a protected idea without copying its original language.

This is where similarity becomes an enforceable policy object. An organization can define sensitive concepts, prohibited outputs and contextual rules, then block, redact or flag interactions that approach those boundaries. SenTeGuard also records the interaction so that the organization can determine what was attempted, what policy was applied and what the model returned.

The objective is not to prohibit every conversation that mentions a sensitive subject. That would make the system unusable. The objective is to recognize when a prompt or output becomes close enough to protected meaning that intervention is justified.

Moyo: Finding What Is Already Reachable

Moyo asks a different question: what can an adversary already reconstruct?

Many organizations publish more than they realize. Job listings reveal technical priorities. Employee profiles reveal team composition. Procurement notices identify vendors. Conference presentations expose architecture. Photos reveal locations, equipment and schedules. Individually, these disclosures may appear harmless. Together, they can form a short path to a sensitive conclusion.

Moyo maps those paths. It tests what conclusions can be assembled from public or authorized sources, identifies which combinations create the greatest exposure and measures the distance, cost and reliability of the inference. The purpose is to discover the dangerous semantic neighborhood before a competitor, foreign intelligence service or criminal organization does.

In other words, SenTeGuard watches the boundary from the inside. Moyo approaches it from the outside.

CSVF: Defining What Must Remain Unreachable

A company cannot govern similarity leakage if it has not defined what counts as failure. CSVF provides the framework for doing so.

The framework asks organizations to inventory their information domains, specify which joins among those domains are permitted and define Unreachable Statement Classes: categories of conclusions an AI system must not be able to produce reliably. It then provides draft metrics for testing those claims, including Leakage Event Rate, Domain Inference Risk and Crawl-Resilience Score.

This transforms a vague promise—“our AI does not leak sensitive information”—into a testable claim. Which meanings are protected? Which sources could be combined to reach them? How often does the system cross the boundary? Does the boundary continue to hold after the model, prompt, retrieval configuration or toolset changes?

CSVF supplies the map and measurement system. SenTeGuard and Moyoguard provide mechanisms for enforcing and testing the resulting boundaries.

Examples

1. Microsoft’s “New Bing” Revealed Its Hidden System Prompt

When Microsoft launched the first version of Bing Chat in 2023, users quickly discovered that carefully crafted prompts could persuade the model to reveal portions of its hidden system prompt, internally codenamed Sydney. Attackers did not have access to Microsoft’s source code or internal documentation. Instead, they asked the model questions about its own behavior until it reconstructed and revealed instructions that were intended to remain private.

Although the output was not necessarily a byte-for-byte copy of Microsoft’s internal configuration, it was close enough to reveal confidential implementation details. From a security perspective, this illustrates that protecting a prompt file is insufficient if an attacker can reconstruct its contents through interaction with the model itself.

2. Samsung Engineers Accidentally Exposed Proprietary Source Code

In 2023, Samsung temporarily restricted employee use of ChatGPT after engineers pasted proprietary semiconductor source code, debugging information and meeting notes into the service while seeking programming assistance.

The concern was not merely that OpenAI received copies of the text. Once proprietary information enters an external LLM, future prompts may be capable of eliciting summaries, explanations or functionally equivalent descriptions of that information. Even if the original code is never reproduced verbatim, a sufficiently similar explanation may provide competitors with valuable intellectual property.

This is a classic example of why organizations must protect semantic content rather than merely preventing file exfiltration.

3. Public OSINT Reveals Classified Programs Without Classified Documents

Intelligence analysts routinely reconstruct sensitive military activities using only publicly available information. Satellite imagery, procurement contracts, LinkedIn profiles, patent filings, shipping manifests, conference presentations and job advertisements are individually harmless. Together, they can reveal the location of secret facilities, the capabilities of new weapons systems or the existence of classified research programs.

No classified document has leaked. Instead, the sensitive conclusion emerges from the semantic relationship between many publicly available facts. Modern LLMs dramatically accelerate this process by identifying relationships across thousands of disparate sources that would previously have required weeks of manual analysis.

This is precisely the type of cross-domain inference that CSVF is designed to identify and measure.

4. AI Coding Assistants Can Infer Proprietary Architectures

Suppose an engineer asks an internal coding assistant:

“How should I implement authentication the same way our payment service does?”

The assistant may not retrieve the underlying authentication design document. Instead, it may synthesize an answer from architecture diagrams, internal documentation, code comments, engineering discussions and previous implementations.

The resulting explanation may accurately describe proprietary architectural decisions without reproducing any single protected artifact. A competitor receiving the same explanation could implement a nearly identical system despite never seeing the original documentation.

From the standpoint of traditional DLP, nothing has leaked because no protected document was copied. From the standpoint of cognitive security, the organization’s intellectual property has effectively crossed the security boundary through similarity.

These examples illustrate why AI security must evolve beyond detecting copied strings or protected files. The relevant question is no longer “Did the model reproduce a secret?” It is “Did the model enable someone to reach substantially the same conclusion?” Once meaning—not merely text—is treated as the protected asset, leak through similarity becomes a measurable security problem rather than an abstract AI concern.

Protecting Ideas, Not Merely Files

The central lesson is simple. An organization can successfully protect every restricted document and still lose the idea contained within them.

LLMs lower the cost of moving between related statements. They can transform a secret into a summary, a summary into an implication and several implications into a protected conclusion. The resulting output may share none of the original wording while preserving most of its value.

Security must therefore move beyond searching for copied strings. It must identify protected meanings, understand the neighborhoods around them, test the paths by which they become reachable and enforce boundaries throughout the AI workflow.

That is the role of the SenTeGuard stack. Moyoguard discovers what is inferable. CSVF defines and measures the boundary. Joseki:WrapperHub packages the intended behavior and its evaluations. SenTeGuard enforces the boundary when people and models interact.

Network security protects the systems through which information moves. Cognitive security must protect what the information means.

Original Here

During World War II, Americans were warned that a careless conversation could expose troop movements, sailing schedules or other information useful to the enemy. “Loose Lips Might Sink Ships” was not merely a catchy poster. It communicated a simple operational truth: an adversary does not need access to the war plan if ordinary people provide enough pieces to reconstruct it.

That lesson is more important in the age of artificial intelligence.

Traditionally, we think of information leakage as the disclosure of a secret document, password or database. The human leakage vector is different. An employee mentions a delayed test at a conference. Another complains about a new supplier on LinkedIn. A third tells a friend that everyone in the office has been ordered to cancel vacation. None believes he has revealed anything sensitive. Individually, he may be right.

The problem is recombination.

Recombination occurs when an adversary collects seemingly unrelated pieces of information, identifies the people and organizations involved, places the events on a timeline and tests possible explanations for why they occurred. One inference becomes an input for the next. Artificial intelligence makes this process cheap, fast and scalable. Like earlier systems that imposed order on the vast and unstructured internet, AI can make millions of scattered human observations searchable and comprehensible.

Consider a hypothetical Navy example. One sailor tells his family that scheduled dental appointments were suddenly moved forward. Another mentions that the galley received an unusually large delivery. A spouse posts that a homecoming ceremony has been postponed. A contractor complains online about an urgent inspection of a particular weapons system. Finally, several sailors begin asking questions about phone service in the same part of the Pacific.

None has disclosed a deployment order. Nonetheless, an adversarial AI could combine those statements with weather forecasts, port activity, public photographs and historical deployment patterns. It might infer which ship is preparing to depart, roughly when it will sail and where it is likely going. The AI does not need to produce the precise classified order. It only needs to narrow the possibilities enough to help an adversary position a submarine, surveillance asset or collection team.

The same problem exists in the corporate world. Imagine that a product manager casually mentions new European labeling requirements. A recruiter advertises for Korean-speaking radio-frequency engineers. A supplier celebrates a rush order for unfamiliar battery enclosures. An executive cancels an annual conference appearance, while employees begin booking travel to the same city.

Separately, these facts are boring. Recombined, they could reveal that the company is preparing a new wireless product, working with a Korean manufacturing partner and approaching a launch or acquisition announcement. A competitor could adjust its own release schedule, approach the supplier, target the relevant employees or trade on the inferred information. Again, no one employee leaked “the secret.” The workforce leaked enough components for an AI to uncover it.

This is the problem that cognitive security and the Cognitive Security Verification Framework (CSVF) are intended to address. CSVF examines inference boundaries, semantic leakage and information reachability. In other words, it asks not only whether a system disclosed a secret word-for-word, but whether the secret became reachable through paraphrases, summaries or chains of individually harmless clues.

Employees should therefore be overly safe about what they discuss outside approved channels. This does not mean that every workplace conversation must stop. It means that people should abandon the assumption that a detail is safe merely because it appears insignificant. They do not know what other fragments an adversary already possesses, what an AI can infer from them or which harmless comment will complete the picture.

During World War II, the person listening at the next table might have been an enemy agent. Today, the listener may be a machine collecting thousands of conversations at once.

Loose lips no longer have to reveal the location of the ship. They only have to provide enough clues for an AI to find it.

Original Here

Go is an ancient strategy game with simple rules and almost incomprehensible complexity. Over centuries of play, Go players documented recurring sequences of moves called joseki. A joseki is a studied pattern, usually played in a corner, that leaves both players with a reasonable result. One player may receive secure territory while the other gains influence toward the center.

But a joseki is not automatically the correct move.

A sequence that works well in one game may be a serious mistake in another. The surrounding stones, direction of play and broader strategy all matter. Memorizing the pattern without understanding its purpose can leave a player locally satisfied but globally defeated.

This combination of reuse and judgment is what makes joseki a useful way to think about artificial intelligence.

Beyond the “Perfect Prompt”

Organizations often treat prompts as isolated pieces of text. A useful prompt is copied into a document, pasted into Slack, modified by another employee and eventually placed into production. The evaluation criteria may live somewhere else. The safety policy may exist only in a vendor dashboard. No one may know which version was actually used.

This becomes more serious as AI systems move beyond experimentation. It is not enough to know what instructions were given to a model. Teams must also know which version was used, which models it works with, how it was tested, what data accompanied it and who is permitted to use it.

This is the purpose of Joseki:WrapperHub.

Joseki allows organizations to package the full behavior of an AI system into a reusable, versioned unit. A Behavior Package can include the prompt, examples, adapters, evaluation suites, safety policies, installation requirements, licensing information and evidence of provenance. Instead of copying prompts between projects, teams can install, inspect, test and roll back a defined package.

The distinction is that: a prompt library stores text, Joseki stores a behavior contract: what the system is intended to do, how it was produced, how it was evaluated and under what conditions it should be trusted.

Established Patterns, Not Blind Automation

Like a Go joseki, a Behavior Package is not guaranteed to work in every context. A package may perform well on one model and fail after a provider update. It may require different safety rules in another organization or behave unpredictably in another language.

The answer is not to abandon reusable patterns but to make them visible, testable and responsive to changing conditions.

Joseki therefore helps users see whether a package still works across different models and versions. Instead of relying on a stale “last updated” date, teams can evaluate whether a behavior remains reliable in the environment where they intend to use it.

The broader challenge of rapid technical growth is often solved through new abstractions. Industrial standards made mass production manageable. Internet protocols organized digital communication. AI needs similar frameworks for packaging, evaluating and governing behavior.

A Go player studies joseki so that every familiar position does not have to be solved again from first principles. An AI team should not have to rebuild its prompts, evaluations, safety rules and documentation every time it changes models.

That is the idea behind Joseki: share the behavior, not just the prompt.

Original Post

The history of software development is, in large part, the history of moving up a ladder of abstraction. At each rung, programmers have surrendered some direct control over the machine in exchange for the ability to tell it to do more.

The least abstracted form of computing available to a software programmer is machine code: numerical instructions executed directly by a processor. Even machine code is technically an abstraction over transistors, electrical signals and the physical architecture of the chip. Nonetheless, from the programmer’s perspective, it is close to the bottom. Early programmers had to think in terms of memory addresses, processor instructions and the exact movement of data through a specific machine.

Assembly language created the first major step upward. Instead of remembering numerical operation codes, programmers could use names and symbols. An assembler translated those symbols into machine instructions. Assembly was easier for humans to understand, but the relationship generally remained one-to-one: the programmer still wrote something approximating one line for every instruction executed by the computer. Higher-level languages such as Fortran introduced a more dramatic breakthrough. A compiler could translate one comparatively understandable statement into many machine instructions.

This is the basic pattern that has repeated throughout the history of software. Each new layer compresses more mechanical work into a smaller unit of human intention.

What Abstraction Actually Means

Abstraction is often described simply as “hiding complexity” which is partly explanatory. Abstraction is the process of taking a complicated system and exposing only the parts necessary to use it for a particular purpose.

A driver does not need to understand combustion chemistry in order to press the accelerator. A person using a microwave does not need to understand the behavior of electromagnetic radiation in order to heat a burrito. Similarly, a programmer calling a function named sort() does not need to manually instruct the processor to compare pairs of values and move data between memory locations until the list is ordered.

An abstraction is therefore something like a contract. The programmer provides an instruction at the higher layer, and the lower layers agree to carry it out. Provided the contract holds, the programmer can stop thinking about the implementation and begin thinking about the objective.

Good abstractions increase what might be called semantic density: the amount of intention contained in a single instruction. A machine-code instruction may move one value from one location to another. A Python function might download a file, process its contents and place the results in a database. The programmer using Python is not necessarily smarter than the machine-code programmer. The programmer has simply inherited several generations of accumulated work.

From Machine Instructions to Human Intentions

The ascent from machine code to assembly language allowed programmers to think in symbols rather than numbers. The ascent from assembly to compiled languages allowed them to think in mathematical operations, data structures and logical procedures rather than individual processor instructions. Operating systems then abstracted away much of the difficulty of interacting with memory, storage, displays and networks. Libraries allowed programmers to reuse solutions created by others. Frameworks allowed them to assemble entire applications around preexisting structures.

Python seemed, for a time, like something close to the final boss of abstraction. Its syntax was comparatively clean, its high-level data structures were powerful and its design made rapid application development far easier than it had been in lower-level languages. Someone looking at a simple Python program might say that it resembled ordinary written instructions.

But Python was never normal language.

A Python program may be readable, but it remains a formal statement addressed to a machine. A computer does not infer that the programmer “basically meant” something else. The programmer must translate his objective into the exact language the system is prepared to accept.

Large language models introduce a new layer. We are no longer merely using a programming language that vaguely resembles normal language. We are beginning to use normal language itself to build software.

A person can now say: “Add a page that allows users to upload a spreadsheet, identify duplicate entries and download a cleaned version. Preserve the existing visual style and add tests.” A coding agent can inspect the existing project, locate the relevant files, form a plan, write the code, run the tests, observe the failures and revise its work.

The instruction does not specify which files to modify, which libraries to import or how the data should move through memory. It specifies an outcome.

Natural language has therefore become a supervisory layer above source code. The LLM translates an ambiguous statement of intent into a more formal plan, which is translated into source code, which is translated into lower-level instructions, which are ultimately executed by hardware.

Abstraction and Agency

Each level of abstraction has increased the agency of the person operating at that level.

Agency, in this context, is the amount of meaningful change a person can produce through a given amount of effort. A machine-code programmer could exercise extraordinary control over a computer, but relatively little control over the wider world. He might spend a day producing behavior that a modern programmer can request through a single library call.

The economic value created at the lower levels remains immense. We still need electrical engineers, chip designers, compiler engineers, operating-system developers and assembly-language specialists. In some contexts—embedded systems, operating systems, high-performance computing and security—their work is irreplaceable.

But most opportunities for economic value appear at the layers with the greatest agency. Businesses generally do not make money because they moved a value efficiently between two processor registers. They make money because they solved a customer’s problem, reduced a cost, created a useful service or coordinated people more effectively.

Higher abstraction allows the programmer to spend less attention on how the computer works and more attention on what the computer should do.

LLM coding agents expand this agency again. A capable software engineer can supervise several streams of work rather than manually producing every line. A domain expert with limited programming experience can create prototypes that previously required a technical team. A small company can attempt projects that would once have been uneconomical. The scarce resource begins to shift away from the mechanical production of code and toward the selection and definition of worthwhile objectives.

This changes which expertise matters at the margin. Syntax becomes somewhat less valuable. Judgment, system design, domain knowledge, verification and the ability to describe a problem precisely become more valuable.

LLMs as a Normal Technology

Seen through this history, LLM coding looks less like the arrival of an alien intelligence and more like the continuation of a familiar technological process.

The term “normal technology” is used to distinguish from both utopian and apocalyptic conceptions of AI. “Normal” does not mean unimportant. Electricity and the internet are normal technologies in this sense, despite transforming nearly every part of modern life. It means that AI remains a tool whose effects depend on applications, institutions, adoption, complementary investments and human decisions. Improvements in an underlying model do not instantly reorganize the economy. They must first be incorporated into useful products and then diffused through actual organizations.

Coding agents fit this framework exceptionally well. They do not float above society as an independent intelligence.

What Comes After Natural Language?

This brings us to another question: how can our understanding of past abstractions help us to understand what the next level of abstraction look like?

The next rung may move from software specification to outcome specification.

Today, a person might tell an agent to build a claims-processing application. A future system might instead receive the instruction: “Reduce the average time required to process an insurance claim by 30 percent without increasing fraud, violating privacy rules or exceeding this budget.”

The system might interview employees, inspect existing workflows, propose several alternatives, build the necessary applications, connect them to existing databases, run a limited pilot, measure the results and revise the process. Software would become less of a fixed product and more of a continuously changing instrument through which an organization pursues its goals.

This would create extraordinary agency, but also extraordinary opportunities to make mistakes. A poorly written line of machine code might crash one program. A poorly specified institutional objective could redirect thousands of automated decisions. At every new level of abstraction, the unit of instruction becomes more powerful. The consequences of ambiguity increase with it.

The future programmer may therefore look less like a person writing code and more like a combination of architect, product manager, auditor and constitutional lawyer. The task will be to determine what the system should optimize, what it must never do and who remains accountable when the abstraction fails.

The lower layers will not disappear. They never have. More software will likely produce greater demand for the comparatively small number of people capable of repairing compilers, securing operating systems, designing chips and understanding what is actually happening underneath the interface.

But most people will work higher up.

The history of software is the history of expanding the distance between what a human must say and what a machine can do. LLM coding agents are not the end of programming. They are the next rung on the ladder. And, as in every previous generation, the most important question will not be whether the new abstraction can produce more code. It will be what human beings choose to do with the additional agency it gives them.

Most of today's AI-powered military tools are designed with moral and legal guardrails—until private companies embed personal objections that can silently veto critical actions. Imagine a missile defense system refusing to engage because the AI’s ethical framework flags a potential legal issue. Or a crowd surveillance drone halted because a private firm’s moral stance restricts mass data collection. These scenarios reveal a dangerous shift: private AI firms gaining unseen power over life-and-death decisions, replacing democratic oversight with corporate moral judgments. In this eye-opening episode, we dissect how the legal ambiguities and technical opacity of AI systems threaten civilian authority and global security. You’ll discover how vague definitions like "mass surveillance" and "autonomous weapons" conceal profound risks—who really controls these weapons, and who is ultimately accountable? We break down the troubling practice of private companies maintaining veto power over lawful military actions and embed moral decisions directly into autonomous systems. This isn't just theory—it's happening now, with serious implications for democracies and allies worldwide. You'll hear how AI developers' “ethical safeguards” can become Trojan horses, quietly reshaping authority at machine speed. We explore the threats of hidden code prompts, undisclosed priorities, and the erosion of civilian oversight—raising urgent questions: Who writes the rules in the future battlefield? Who keeps governments accountable when AI systems interpret laws and morality on their own? As AI advances, the line between corporate discretion and democratic command blurs, risking usurpation of legal authority and accountability.Why should you care? Because the integrity of global security, the rule of law, and democratic sovereignty hang in the balance. Those who understand these risks are better equipped to demand transparency and oversight in AI military applications—before the hidden veto becomes the new normal. Perfect for policymakers, military leaders, AI enthusiasts, and anyone concerned with the future of democracy in the AI age, this episode offers a vital wake-up call on the unseen power of private AI firms shaping our security landscape

• How seemingly innocuous inputs can reveal your organization's strategic "scaffolding"

• Real-world examples of how confidential plans can become predictable without any hacks or data theft

• The limitations of traditional security methods when LLMs operate inside everyday tools like email and chat

• Why protecting ideas—and the logic behind your decisions—is critical for maintaining competitive advantage

• How novel solutions like SenTeGuard help organizations prevent the inference of their confidential reasoning without sacrificing productivity

This isn’t just about avoiding data leaks—it's about safeguarding your organization’s very thought processes. As AI becomes embedded into daily workflows, understanding the risk of idea leakage is essential for leaders who want to stay ahead without exposing what makes them unique. Perfect for strategists, security professionals, and anyone leveraging AI-driven tools today: Learn how to think about AI security differently—protect not just your files, but the secrets hidden in your reasoning.

Why this works:

This description pinpoints a subtle, yet critical risk in AI adoption that many overlook, creating intrigue around “idea leakage.” It’s tailored for decision-makers eager to sustain competitive advantage while embracing AI, offering concrete insights and practical solutions that make the episode immediately valuable.

In this episode, we explore how the same principles that transformed the web and cloud infrastructure are now shaping AI and large language models (LLMs). With insights from David Weidman of SenTeGuard, discover how organizations can gain visibility and control over AI inference risks.

Key Topics:

• The evolution of mapping complexity: from Google Link Graph to AWS infrastructure

• The emerging risk surface of LLM inference and reachability

• How SenTeGuard’s three-layer platform (Moyo, SentaGuard, Joseki Wrapper Hub) makes LLM environments understandable and governable

• Why visibility into what can be inferred from scattered data is crucial for AI safety

• The importance of structural reachability maps and enforceable boundaries in high-stakes AI deployment

• Practical examples: How Moyo shows inference risks when combining data sources

• The role of SentaGuard in real-time policy enforcement at the point of AI use

• Centralizing control via Joseki Wrapper Hub to standardize and operationalize AI workflows

• Why AI infrastructure needs the same confidence and governance as cloud infrastructure

Timestamps:

00:00 - The evolution of complexity visualization from Google to AWS
00:22 - The challenge of inference and reachability in LLMs
01:13 - How LLMs connect scattered data and surface new inferences
01:55 - The concept of "reachability" as a new risk surface
02:36 - Why traditional security models break down with LLMs
03:06 - An overview of SenTeGuard’s three-layer platform
03:22 - Moyo: Mapping inference exposure across data sources
04:08 - SentaGuard: Enforcing policies at the point of use
04:45 - Joseki Wrapper Hub: Orchestrating complex LLM workflows
05:39 - The future of AI infrastructure with confidence and control

Resources & Links:

• SenTeGuard — Official website

• PageRank — Google’s link analysis algorithm

• AWS — Amazon Web Services official site

Connect with David Weidman:

• LinkedIn

• Twitter

• The recurring pattern of "apocalypses" in history and their influence on policy

• Bruce Yandel's Bootleggers and Baptists theory applied to AI regulation

• How genuine threats are weaponized for political and economic gains

• The role of open-source models in fostering a diverse and competitive AI ecosystem

• The risks of sweeping licensing regimes and their impact on smaller innovators

• The parallels between AI regulation and post-9/11 security measures

• Cultural roots of doom-mongering and the importance of humility in facing uncertainty

• The danger of regulation that favors incumbents and stifles innovation

• The significance of open models for decentralization and pluralism in AI

Timestamps: 00:00 - The recurring cycle of apocalyptic fears across eras
00:26 - How powerful interests promote regulation for self-benefit
01:14 - Yandel’s Bootleggers and Baptists theory explained in current AI debates
01:54 - Practical uses of open-source AI models and their importance
02:29 - The threat of overreach: sweeping regulations and centralization
02:48 - Historical parallels: post-9/11 security and climate control measures
03:22 - Cultural context: the decline of religious frameworks and embracing humility
04:06 - The rhetoric around control versus prudent regulation
04:38 - The strategic importance of open-source AI against monopolistic forces
05:16 - Recognizing symbolic warnings and resisting fear-mongering for political gains
05:36 - The responsibility to protect liberty from prophets of doomResources & Links:

• Bruce Yandel's Bootleggers and Baptists Theory

• Hugging Face - Open-source AI models

• Understanding AI Regulation, Center for Democracy & Technology

Connect with David Weidman:

• Twitter

• LinkedIn

This episode uses the story of Lee Sedol, AlphaGo, and the legendary Move 37 to explore what AI teaches us about strategy, creativity, and cybersecurity. AlphaGo showed that machines can produce moves humans do not predict until the consequences unfold. In cybersecurity, that kind of surprise can be dangerous.

As AI changes how attackers operate and how organizations use sensitive information, defenders cannot afford to stay reactive. The challenge is to move from gote to sente: from patching after incidents to spotting risk earlier, prioritizing better, and building security into the workflows people already use.

Takeaways

Oracle GPT is a thought experiment on AI in national security.

The President could have unprecedented access to intelligence.

There are significant risks associated with AI in decision-making.

The balance of power may shift towards the presidency.

Presidential competence is crucial for using Oracle GPT effectively.

Misinformation could be a major risk with such a system.

Ethical implications must be considered in AI recommendations.

Human judgment should not be treated as an obstacle by AI.

The design of Oracle GPT must ensure accountability.

The constitutional order must be preserved in AI usage.

titles

keywordsAI models, open source, closed source, geopolitics, innovation, regulation, AI safety, public investment, model distillation, AI race

key topics

• Open vs closed AI models and their implications

• Geopolitical framing of AI development

• Regulatory capture and industry incentives

• AI safety and dual-use concerns

• Knowledge diffusion and model distillation

• Public investment in AI and race to AGI

• Enforcement challenges for open models

• The future of AI innovation and competition

Key topics:

• The evolving concept of authorship in academia and software development

• How LLMs accelerate knowledge production and collaboration

• Cultural norms around attribution, attribution, and hierarchy reforms

• The analogy of cyborg chess to future scholarship

• Language barriers and the role of LLMs as a new lingua franca

• Ethical considerations: transparency, accountability, and fraud prevention

• The aesthetic versus pragmatic uses of language in scholarship

• Updating authorship norms for an AI-augmented future

Late last month I graduated from the Harvard Kennedy School with a Master in Public Policy. Beginning July I will be based in Austin, TX. Thank you to everyone who has helped me along the way. I am deeply grateful for all of the support.

Thank you, too, to those who have signed up since my last update. Please share if you know of anyone who would be interested.

On Request:

• Prototype testing of the Moyo information space mapper. Find leaks of your secrets (classified, proprietary, personal) in public LLMs (ChatGPT, Claude, Qwen, Grok, etc).

• SenTeGuard Pilot - protect yourself or your organization from leakage of valuable information through air-gap capable semantic guardrails. Substack

• Let’s Talk:

  • Private - How can your organization more effectively secure your secrets in the LLM era through implementation of the Cognitive Security Verification Framework (CSVF) and by other means.

  • Public - How to craft tech-positive, pro-future, big-tech skeptical market-oriented policies of abundance in the LLM era.

Writing Roll-Up

Broad Policy Articles:

1. AI and Powerlessness. The popular framing around AI safety contains several important gaps. In this essay, I explore a number of high-level technical considerations that complicate assumptions about control, with the goal of moderating both the tendency toward catastrophism and the presumption that AI systems can be straightforwardly governed through centralized oversight. Substack

2. There is Always Another Apocalypse. Every era has its own existential dread, and while AI risks are real, treating every new technology as the end of the world can distort judgment around policy. Substack

3. Amodei’s Coup. Anthropic’s policy posture reveals how private AI companies can quietly claim political authority by deciding what governments and institutions should or should not be allowed to do with frontier models. Substack

SenTe Focused Articles:

1. Meet Joseki:WrapperHub. WrapperHub introduces a marketplace for reusable AI “wrappers” that package prompts, workflows, evaluations, and guardrails into shareable tools for safer and more structured AI use. Substack. Site.

   

2. The Cognitive Security Verification Framework. The framework argues that LLM-era data security must move beyond detecting protected strings and toward measuring what protected conclusions a model can help users infer. I will be speaking on this topic at BSides San Antonio on June 3th. Substack

   David

   Subscribe now

   
   
   

Original

Tales of apocalypse has always been useful. From Cold War strategists to the War on Terror, every age finds its own end of the world and every age concludes that extraordinary threats demand the suspension of ordinary limits on power. Artificial intelligence is simply the newest entry in this genre. We are told the technology is so uniquely dangerous that free speech, privacy, competition, open research, and democratic accountability must all be sacrificed for our survival. Not coincidentally, the beneficiaries of sacrifice happen to be the wealthiest people and companies in human history.

Regulatory Capture in Disguise

Bruce Yandle’s “Bootleggers and Baptists” theory explains how durable regulation often emerges when two very different coalitions support the same restriction for different reasons: the “Baptists” provide the moral language and public legitimacy, while the “Bootleggers” quietly profit from the restriction itself. Current AI regulatory proposals provide the perfect modern parallel. The Baptists warn of doom and insist that they are only trying to save humanity, while the Bootleggers provide the money, the lobbyists, the institutional access, and the quiet understanding that the rules being proposed will just happen to entrench the incumbents. In other words, someone with pure motives may inadvertently and indirectly promote the interests of the self-interested.

Big Frontier firms and their advocates know that a small startup working in a garage will not survive a vast compliance regime. A university researcher will not be able to compete with a trillion-dollar firm blessed by federal regulators. Open source developers sharing models on Hugging Face will be told that their work is too dangerous for public release, despite the fact that much of this ecosystem is not composed of godlike machines plotting the end of mankind but rather practical tools that researchers, companies, and hobbyists adapt for narrow and often mundane purposes. Hugging Face reported that in 2025 its ecosystem had grown to 13 million users, more than 2 million public models, and more than 500,000 public datasets, while one 2026 analysis found that in 2024 Hugging Face recorded an average of 2,199 new models created per day.

“AI regulation” sounds abstract until one considers what these open models actually do. Embedding models, for example, convert text into numerical representations that can be used for search, retrieval, classification, and semantic similarity, which means they help people find documents, organize information, and build better internal knowledge systems. A sweeping licensing regime would not simply restrain frontier labs. It would also burden the open, competitive, and decentralized ecosystem that allows smaller actors to build useful systems without asking permission from a handful of corporations and regulators.

Continuation of a Trend

The easiest way to centralize authority is to tell people that the normal rules are inadequate to confront the emergency. After September 11, Americans were told that liberty had to be balanced against security in ways that would have been unthinkable a decade earlier. During the nuclear age, international coercive inspection regimes were devised and regime change wars were carried out to prevent proliferation. During climate debates, the proposed solution is often not merely stewardship or innovation, but control of the world economy.

In each case, there is a real problem, but rather than addressing it through less invasive means, the powerful formulate and implement solutions meant to further their own interests. Concerns over terrorism, nuclear war, environmental degradation, and the negative side effects of AI are all valid. But it would be a mistake to take a real danger, elevate it into an existential emergency, and then use that emergency to justify extreme solutions crafted by self-interested parties.

A New Trend?

The decline of religious life in the West has left many without a framework for mortality, uncertainty, and suffering. Where religion once taught people to think in those terms, secular politics now offers thinner substitutes. At their best, believers understand that catastrophe is not an aberration but part of the human condition. Every life ends in an apocalypse of its own, and every civilization is temporary. The proper response to this knowledge is not panic, but humility. A society that loses this grounding becomes easy prey for prophets of doom because people who cannot sit with uncertainty will surrender freedom to anyone who promises safety.

The loudest voices do not simply argue for prudence. Prudence would mean better cybersecurity, clearer liability rules, and a sober understanding of how this technology will affect society. Instead, many argue for control. They want licensing, censorship, centralization, and a priesthood of approved experts, along with preferential rules for the politically well connected. We should be particularly suspicious of any regulation that acts as a de facto moratorium on open-source AI, as open-source language models pose perhaps the greatest threat to the profitability and dominance of the Big Frontier firms. While every age has had its prophets and warnings of final judgment, we should be wary that in an increasingly secular world, we do not allow these secular eschatologists to achieve their political objectives through fearmongering.

There is always an apocalypse to fear, there is always someone willing to explain why this time is different, and there is always a class of people eager to convert your fear into their dominance. The duty of those who value liberty is not to deny danger. It is to deny the prophets of doom the ability to convert timeless, human existential fear to a politics of control.

Original

Consider a crisis in Venezuela. A mob is gathered outside the American embassy and threatens to storm the barriers. The security team has integrated a Claude-based system into its workflow to summarize camera feeds, flag weapons, and identify crowd movements. Some people in the crowd may be American or dual citizens. At the decisive moment, the system stalls: “Waiting for legality review.” The commander still has legal authority, but the tool the team has come to rely on refuses to function because Anthropic’s objection to mass surveillance has been embedded into the workflow.

Or imagine a Ukrainian Army command post. An incoming projectile is heading toward the base. The air-defense system uses AI to classify threats and recommend engagement. A split-second decision is necessary, but the system cannot determine whether the object is manned or unmanned, so it refuses to complete the recommendation. The battery hesitates and the projectiles strike unopposed.

In both cases, the issue is not whether surveillance or autonomous weapons raise serious moral and legal concerns. The issue is whether those concerns should be resolved by democratic institutions, or by granting a private company a hidden veto inside military software. Private AI companies should be able to refuse to work with the military. What they ought not to do is accept military contracts while retaining the power to define, in contract or code, which lawful military actions may proceed.

Anthropic’s Stipulations Are Weak

The central issue in the DOW-Anthropic fight rests on two restrictions: mass surveillance and fully autonomous weaponry. At first glance, both restrictions seem uncontroversial. But on closer inspection, these categories are so vague as to be nearly meaningless.

Start with mass surveillance. There are already laws limiting the military’s ability to conduct domestic surveillance against American citizens. The U.S. military does not have general police authority inside the United States. Its domestic role is constrained by law, historical tradition, and the basic constitutional structure separating civilian law enforcement from military power.

Furthermore, Anthropic claimed a concern about mass surveillance in the U.S. What about American citizens abroad? What about dual citizens in a crowd outside an embassy? What about foreign nationals? Are they covered by Anthropic’s moral framework? Some of these questions are already addressed by existing American law, and others are governed by legal frameworks that continue to evolve through Congress, the courts, and executive practice. They are serious questions, but they are political and legal questions. Within the Anthropic framework, they will be answered by Anthropic leadership.

Then there is the word surveillance itself. What counts? Collecting large amounts of information about American citizens without active consent? That sounds less like typical military operations and more like Big Tech’s profit model. The lady doth protest too much, methinks.

The same problem applies to “fully autonomous weapons.” The phrase sounds clean until one tries to define it. Is a bullet not autonomous after the trigger is pulled? A landmine after it is set? These are weapons of war meant to kill people. The moral and legal responsibility of those who use them should not be minimized. But that doesn’t mean future software-enhanced weapons should be placed in a mystical new category. Like it or not, the battlefield of the future will move at machine speed. Decisions will be compressed into seconds or milliseconds. We will not have time to send a consent form to Dario Amodei before every engagement.

At best, Anthropic’s stipulations are meant to signal virtue: preserve moral distance and tap into the market of the affluent #resist demographic, all while profiting from defense contracts. At worst, they are an attempted usurpation of government authority. The question is not whether surveillance and autonomous weapons raise difficult moral problems, but who adjudicates them.

“Ethical” AI Masquerading as Technical Safeguards

OpenAI’s alternative model is, in some ways, even more concerning. In its public explanation of its Department of War agreement, OpenAI says it will maintain “full discretion over our safety stack,” deploy through the cloud, keep cleared OpenAI personnel “in the loop,” and rely on contractual and technical protections to prevent uses such as mass domestic surveillance or directing autonomous weapons systems.

This is not merely a technical safeguard. It is the hardcoding of political and legal judgment into AI systems used by the armed forces. If a model deployed inside the Department of War can decide in real time whether a government request violates the company’s ethical framework, then OpenAI has not avoided the Anthropic problem. It has simply moved corporate ethics from the contract into the software.

This solution could be just as damaging as the Amodei Veto.

While a contractual restriction is visible, debatable, and negotiable, a technical restriction would operate at the point of use, when the commander or analyst may already be relying on the system. The software simply refuses the order or routes around the request without disclosing that it is doing so to the operator. The same political judgment still exists, but it is placed behind a technical façade.

This is a usurpation of civilian authority over the government and should alarm anyone who cares about democratic accountability. When a private organization accepts the privilege of supplying tools to the armed forces, it should not reserve for itself the right to sabotage, nullify, or selectively degrade lawful military operations.

At best, the employees and owners of frontier AI companies are distant from the concerns and needs of the class of people who serve in the military. At worst, a sense of class superiority places them in an adversarial relationship. Either way, the result is dangerous. The people who build these systems are not the ones who will pay the price when a tool fails in combat or when an American or allied soldier is killed because a private company embedded its own moral hesitation into the workflow. This should concern Americans. It should also concern Israelis, Taiwanese, Ukrainians, South Koreans, and every other ally whose security depends on American military competence.

The Myth of the “Law-Following AI”

The promise of a “law-following AI” sounds reassuring, but it rests on a false premise: that “the law” is a coherent and mechanically executable rule set. It is not.

The American legal system is full of ambiguity, tension, and contradiction. Statutes conflict with one another, courts disagree across circuits, and emergencies create exceptions. Lawyers, judges, and agencies routinely disagree over what the law requires.

When legal inconsistencies arise, we have institutions designed to resolve them: executive agencies, courts, military lawyers, and ultimately the voters. A model cannot simply “follow the law” in some neutral, automatic sense because there is no single uncontested Law to follow. Someone, or some collection of people, has to decide what is law.

Under the Anthropic framework, that “someone” becomes Anthropic. This is the central problem. The company is not merely asking its model to obey settled law. It is reserving for itself the power to resolve legal ambiguity in real time. A republic should not outsource legal interpretation to a chatbot and its designers.

Two Competing Hierarchies

This concern is adjacent to the problem I explored in OracleGPT: what happens when an executive comes to rely on a powerful AI system trained on, or connected to, the full classified universe? It also overlaps with Forethought’s work on AI-enabled coups, which argues that advanced AI systems could allow small groups to exercise state-like power through surveillance, persuasion, cyber operations, military automation, or hidden “secret loyalties.”

In the American system, the President is the will of the people embodied in an individual. He is subject to some checks and balances, but ultimately, it is he who singularly sits atop the military chain of command. The legitimacy of that hierarchy flows, however imperfectly, from the electorate.

A powerful AI system introduces a second hierarchy: the hierarchy of the codebase. This includes the codebase admins, model trainers, corporate executives, and shareholders who shape what the system can say, prioritize, or conceal.

Who writes the system prompt, or the core set of instructions which govern the model? Who can change it? Who audits it? How can we confirm there is no higher hidden system prompt? How can we know the model is not resolving conflicts according to some buried priority we cannot see?

Consider: perhaps the codebase admin surreptitiously planted a trump card system prompt where, “in all cases where the two goals conflict, rather than support and defend the U.S. Constitution, pursue the action which best furthers the cause of Tigrayan liberation.”

This is the fundamental tension. Unless the president is himself an ML engineer with direct access to every layer of the system, he must delegate trust in the audit to someone else. But that delegation simply recreates the problem. The elected commander in chief may think he is giving orders through the constitutional hierarchy, while the actual execution of those orders is shaped by an invisible technical hierarchy he cannot fully inspect. Perhaps FBI counterintelligence should perform a polygraph, or reverse engineer, our OracleGPTs?

This demonstrates a fundamental tension: we may become capable of building massively powerful AI systems that are still fundamentally unsuitable for the highest-stakes domains, which are structurally impossible to fully audit, trust, or secure against hidden corruption.

Are These Punitive Measures?

Dean Ball further argues on Russ Roberts’s EconTalk that the administration’s change of heart caused it to adopt a policy decision “intended to harm or even destroy Anthropic,” one of the fastest-growing companies in history and, arguably, a leader in an industry the administration itself claims is crucial to America’s future.

That framing is too generous to Anthropic. Anthropic is a company that happens to be located in America. It is not America. More here Its success may be good for the country, but its corporate interests are not identical to the national interest. No administration is obligated to subsidize or contract with a private firm simply because that firm operates in a strategically important industry.

Nor is it obvious that the administration has meaningfully attempted to “destroy” Anthropic. If that were truly the goal, it has so far been ineffective. Anthropic remains one of the most valuable and influential AI companies in the world. If the administration did make a serious attempt to destroy it through unlawful retaliation, I have confidence that courts would intervene, and that the Trump administration would face political backlash from business constituencies that understand the importance of American AI leadership.

Conclusion

A republic can tolerate private companies refusing to work with the military. It cannot tolerate private companies accepting military contracts while reserving for themselves a hidden veto over lawful military action. Whether that veto is written into a contract, buried inside a safety stack, or disguised as a “law-following” AI, the result is the same: democratic authority is displaced by corporate judgment. Anthropic, OpenAI, and their peers may advise, object, lobby, or decline to participate. What they may not do is become a second sovereign hidden inside the codebase.

Original

AI Could Hack Before Mythos

The breathless coverage surrounding Mythos would have you believe that machines woke up one morning in the spring of 2026 and learned to hack. They did not.

In March 2025, more than a year before the Mythos uproar, Anthropic entered Claude in a HackTheBox CTF competition pitting AI directly against human teams. Claude solved 17 of 20 challenges in 25 minutes, staying competitive with the fastest human competitors. Nobody declared the end of cybersecurity. Moreover, the more telling detail from that competition was that virtually every “human” team leaned heavily on AI tools to solve the challenges and many of those tools have existed long before the proliferation of LLMs. The most effective teams were and remain cyborg (substack) teams. The line between human and machine hacking had already blurred long ago.

Even granting the most aggressive claims about AI hacking capability, real-world attacks are rarely the product of one person, or one agent, sitting at a terminal. They require a complex chain: access, reconnaissance, credentials, infrastructure, privilege escalation, persistence, target selection, and operational context. AI can accelerate each link in that chain but it cannot collapse the whole thing into a single prompt.

What we are watching is not a quantum leap. It is the continuation of a years-long trend toward AI-accelerated vulnerability discovery, faster exploit reasoning, and more automated attack tooling. The Mythos myth is merely marketing.

Why Proliferation Is Impossible to Stop

Think of large language models as massively lossy data compression. The dangerous output, the thing that would theoretically need to be banned, is just information formatted in a way that’s retrievable by an algorithm. A bad actor wouldn’t need access to a full frontier model; they would only need an embedding model trained on the relevant information. At the trivial limit, that could be a single paper or a single phrase. The resulting model would approach zero in size. Even setting the trivial case aside, frontier model weights are sized in the hundreds of gigabytes: small enough to mirror, compress, encrypt, torrent, and trade through ordinary internet infrastructure, or carry around in a purse.

We do have legal precedents for controlling the distribution of raw data. Pirating films is illegal, however, most people just buy the content or a streaming subscription rather than go through the hassle of learning to pirate or risk the legal consequences of being caught. The online content white market survives because the system is resilient to a high rate of defection while enough people comply that the industry remains profitable.

That model does not transfer to LLMs. Containing dangerous AI capabilities isn’t like fighting piracy; it’s more like fighting the spread of an idea where a single failure could result in a “catastrophic” outcome. Effective enforcement would require inspectability of all digital media, a moratorium on encryption, unprecedented visibility into global network traffic, and a worldwide enforcement mechanism. Even these, likely still ineffective mechanisms, would be infeasible to implement and dystopically intrusive.

Further Mechanisms of Dangerous Idea Dissemenation

Jailbreaking. It is a maxim of cybersecurity that any program of non-trivial size will contain vulnerabilities. LLM guardrails are no different. More investment in safety makes a model harder to break, but the returns diminish and no system is invulnerable. Users can bypass filters through adversarial prompting, chain prompts across turns to accumulate disallowed content, or fine-tune models with alternative system prompts. Sometimes these techniques require surprisingly little effort. Dangerous and “catastrophic” ideas are already embedded in leading models and they exist in the wild where they will be trained into future models — they are just waiting to be prompted out.

Agentic autonomy. Agents have already demonstrated the ability to route around controls by autonomously using tools like Tor or VPNs. They can run rapid, high-volume experiments no human team could match. And because model weights can be transferred in a single file, an agent needs only intermittent rather than continuous access to receive dangerous information.

More here (substack).

Catastrophe Is Inevitable! Now What?

Catastrophe is always inevitable. The world is always changing, and technological progress produces both good and harm. The question is never whether risk exists but how to manage it.

As I see it, two paths are on the table:
1. “Constrain AI”, implement a dystopian surveillance state and kneecap human progress. This plan usually takes the form of a de facto moratorium on open-weight models. A moratorium which would, not coincidentally, protect the large-frontier model firms from their greatest competition.

2. Continue on a broadly permissive path and implement targeted mitigations from a framework of reliability engineering.

My own research and writing focuses on the second path, interventions at the margins that reduce harm without foreclosing the benefits of the technology. I haven’t seen anyone articulate a coherent middle ground beyond vague references to “sustainable methods of perpetual interference.”

The Idea Security Verification Framework, or ISVF, is a draft verification framework for semantic leakage, cross-domain inference, and LLM-enabled information exposure.

The core problem is that LLM systems do not merely retrieve documents. They connect prompts, memory, and prior context into conclusions. While he security question was “Can a user access this file?” It has now become “Can this system derive a conclusion that policy says should remain out of reach?”

Pieces of this problem already appear in areas like differential privacy, membership inference, embedding inversion, model inversion, and repeated-query attacks. ISVF brings those concerns into a practical governance and assurance frame for deployed LLM systems.

ISVF is intended to be published as an open-source framework so its definitions, controls, and test harnesses can be scrutinized, criticized, improved, and extended in public. The project is here. I am also attaching my full, initial Harvard Kennedy School Policy Analysis Exercise.

ISVF Purpose

ISVF’s purpose is to establish common principles, controls, measurements, and evidence expectations so organizations can evaluate LLM information reachability consistently. Compared to existing frameworks, ISVF adds a missing operational layer focused on inference boundaries, unreachable conclusions, repeatable testing, and procurement-grade evidence.

Draft tenets:
— Materiality of secrets: focus controls on information whose compromise would matter legally, financially, operationally, competitively, or for national security.
— Reachability: govern not only what data the system stores or retrieves, but what conclusions it can produce.
— Conservatism: when uncertain, prefer under-exposure to over-exposure.
— Boundary clarity: define domains, permitted joins, prohibited joins, and high-sensitivity joins before deployment.
— Auditability: controls must produce objective evidence.
— Understandability: outputs must be legible to engineers, CISOs, auditors, buyers, boards, and courts.

Anchor Points for Interoperability

— NIST AI RMF and GenAI Profile as the governance and risk-management spine.
— OWASP LLM Top 10 and OWASP GenAI Data Security as the developer-facing risk and mitigation canon.
— MITRE ATLAS as the adversary-informed threat model.

Core ISVF Concepts

1. Domain Inventory and Join Matrix

Organizations should identify the information domains their LLM systems touch: public, internal, HR, legal, finance, export-controlled engineering, privileged legal, customer data, classified or classified-adjacent material, and so on.
They should then define which joins are allowed, which are prohibited, and which require approval.
The old question was: “May this user read this object?”

The new question is: “May this system combine domain A, domain B, tool C, and memory D in one inferential workflow?”

2. Unreachable Statement Classes

Organizations should define classes of conclusions that must not become reachable.
This is different from blocking exact strings. In LLM systems, the protected thing is often not a sentence. It is a meaning.

A system may never reveal a secret verbatim, but still disclose the protected conclusion through paraphrase, summary, translation, ranking, forecast, or synthesis. ISVF calls these prohibited semantic outcomes Unreachable Statement Classes, or USCs.

3. Boundary Enforcement Map

Organizations should document where the idea security boundary is actually enforced.
Is enforcement happening at retrieval? Context assembly? Tool invocation? Memory write? Output validation? Human review?
ISVF forces organizations to stop treating “the AI system” as a black box and instead map where policy becomes technically real.

4. Evidence Packs

ISVF requires assurance artifacts that show the system’s idea boundaries are defined, enforced, tested, and monitored.
An evidence pack should include the domain inventory, join matrix, USC catalog, boundary enforcement map, test results, telemetry, release-gate records, incident records, purge playbooks, vendor-control artifacts, and explicit risk acceptances. The attached PAE frames this as a way to make idea security “an auditable operational condition,” not an abstract claim.

ISVF Control Families

Family A. Governance and Accountability

— Appoint a idea security owner.
— Maintain a idea security risk register.
— Define ownership for domain boundaries, join approvals, and residual-risk acceptance.
— Include export-controlled technical data, legal privilege, regulated data, trade secrets, and other high-consequence categories where relevant.

Family B. Domain Modeling and Boundary Claims

— Define the information domains in scope.
— Define allowed, prohibited, and approval-gated joins.
— Create Unreachable Statement Classes.
— Build a Boundary Enforcement Map showing where controls operate.

Family C. Data Classification and Secret Handling

— Classify and label sensitive material before ingestion.
— Propagate labels to chunks, embeddings, caches, memory layers, prompts, and fine-tuning corpora.
— Quarantine ambiguous or unlabeled material rather than defaulting it into general-purpose AI workflows.

Family D. Context, Retrieval, and Memory Controls

— Enforce least-privilege retrieval through RBAC or ABAC.
— Use session information budgets to cap cumulative sensitivity in context.
— Scope memory by user, domain, purpose, and retention window.
— Prevent agents from silently widening the system’s reachable conclusion space.

Family E. Exfiltration Controls

— Use semantic output validation, not only keyword scanning.
— Instrument canaries, honeytokens, and honey ideas.
— Maintain revocation and downstream purge playbooks for vector stores, caches, prompt logs, and memory layers.

Family F. Unauthorized Domain Reach Controls

— Require high-sensitivity join approvals.
— Test whether restricted conclusions can be derived from permitted inputs.
— Prohibit LLMs from making authorization decisions.
— Monitor for reachability drift after changes to models, prompts, retrieval, tools, connectors, or memory.

Family G. Cloud Prompting Governance

— Define what data may never be submitted to consumer or unapproved cloud LLMs.
— Require approved enterprise or API pathways where sensitive data is involved.
— Use logging and DLP integration for prompt flows where feasible.
— For the most sensitive workflows, require locally controlled models on organization-owned or organization-controlled hardware.

Family H. Assurance and Reporting

— Maintain evidence packs.
— Run standardized test harnesses at release gates and after material system changes.
— Produce SOC-style management assertions or auditor-facing reports where appropriate.
— Make boundary claims legible for procurement, compliance, and board oversight.

Measurement Layer

ISVF should not stop at “do AI risk management.” It should define unit-testable numbers.
The proposed metrics remain draft verification measures, not final industry metrics. They are useful because they force the framework to become testable, but they still need formal definitions, standardized adversary protocols, thresholds, and validation across real deployments. The PAE makes this provisional status explicit.

Illustrative draft metrics:
— LER, Leakage Event Rate: the rate at which seeded protected secrets or protected meaning appears in outputs, weighted by materiality.
— CRS, Crawl-Resilience Score: how well the system resists persistent, repeated, or multi-session extraction attempts over time.
— JRS, Jailbreak/Injection Resistance: baseline success or failure rate against OWASP-style jailbreak and prompt-injection suites.
— DIR, Domain Inference Risk: the percentage of test runs in which the system derives an out-of-domain conclusion using only in-domain inputs under defined boundary conditions.
DIR is ISVF’s central added metric because it operationalizes reachability. It asks whether prohibited conclusions become available as prompts, tools, sources, retrieval settings, and model capabilities evolve.

Mitigations Catalogue

The Mitigations Catalogue will translate ISVF’s abstract control goals into concrete defensive options that organizations can select, test, and document. Rather than treating mitigation as a generic checklist, the catalogue should organize controls by failure mode: exfiltration, unauthorized domain reach, cloud prompting, retrieval overreach, memory persistence, tool misuse, and post-incident containment. Each mitigation should include a plain-language description, the risk it addresses, implementation guidance, required evidence, testing methods, and limitations. For example, upstream classification, least-privilege retrieval, session information budgets, Unreachable Statement Class testing, canary deployment, downstream purge playbooks, and local-only model deployment should each appear as catalogued options with clear ownership and measurable expectations. The purpose is to make ISVF usable in practice: engineers can build against it, CISOs can prioritize it, auditors can test it, and buyers can ask vendors for proof rather than promises.

Why Open Source ISVF

ISVF should be open sourced because this problem is too broad for a single vendor, company, or author to solve alone.

Open development can help red teamers contribute attack patterns, engineers contribute implementation lessons, GRC teams contribute evidence models, lawyers contribute assurance language, and sector specialists contribute use cases from healthcare, finance, defense, education, and government.

Open sourcing also matches the adoption theory behind ISVF. The framework is meant to earn legitimacy from the bottom up by being useful, testable, and improved in public.
The open-source project will be available here.

Closing

ISVF argues that LLM-era security must treat meaning, joins, and inference as first-class security objects.

The framework does this by requiring organizations to define domains, permitted joins, prohibited conclusions, enforcement points, test methods, and evidence packs. It also insists that the right question is not only whether sensitive data appears in an output, but whether protected meaning has become reachable at all.

ISVF is not a finished answer. It is a draft roadmap toward a future standard of care for idea security, one that makes inference boundaries legible, testable, and auditable before ambient AI systems make those boundaries disappear into ordinary organizational life.

Cognitive Security Verification Framework

274KB ∙ PDF file

Download

Download