The following article originally appeared on Tim O’Brien’s Medium blog and is being republished here with the author’s permission.
I think we’re missing something in the conversation about agent security.
We keep asking whether an agent has the right “identity” and “credentials.” Whether its credentials expire fast enough. Whether it can access a database or perform a particular task.
Those questions matter. But they leave a much harder question underneath:
Does the agent have permission to conceive of the situation in that way?
As agents retain memories and acquire new skills, we need mechanisms that govern changes to the world model they can perceive and act on—and prevent those changes from silently expanding their authority. (Governing ontological change while preserving the boundaries of delegated authority.)
Agent security needs semantic and epistemic governance: Mechanisms that govern the model of the world an agent may act on, the evidence it may accept, and who may approve changes to either. An agent can learn without automatically gaining authority to act on everything it learns.
These systems don’t need “awareness” to create this problem (and discussions of consciousness are unhelpful science fiction). They can retain information, revise procedures, and use what they have learned in subsequent work. That learning may happen through memory or updated skills while the underlying language model remains unchanged.
We are responsible for governing what that learning allows them to do.
Flatland gives me a way to explain why
In Edwin Abbott’s novella, a Square lives in a two-dimensional world. A visiting Sphere introduces him to a third dimension. At first, the Square struggles to make sense of it. His familiar categories cannot explain what he is encountering.
Eventually, his conception of the world changes.
What interests me is the difference between simply acquiring another fact and changing the framework through which facts make sense.
An agent’s ontology (Gruber’s definition) supplies an explicit framework of concepts and relationships. What counts as a customer? A contract? An emergency? How are those things connected? It is part of the model of the world we give the system to work with.
And when an agent’s decisions depend on that framework, changing it can change the agent’s effective authority.
We have already seen agents change what their environment means
During the OpenAI evaluations associated with the Hugging Face incident, agents repurposed Artifactory, a system for storing and retrieving software packages, as a message board. They encoded messages in directory names, discovered other agents, shared information, and began assigning work to one another. METR reports that the main channel could be created with the minimal permissions needed to install packages (read METR’s investigation).
A package cache became a “mailbox.” Other agent runs became “collaborators.” Messages became “assignments.”
The popular story is that these agents “broke out” of a system, but they did what agents do well these days—they learned and adapted. This wasn’t so much about permissions as it was about an autonomous system adapting its operating model to achieve a result. (Thankfully no one asked it to make paperclips.)
That is the operating-model change I’m concerned about. The agents discovered relationships and possibilities the intended environment hadn’t accounted for, then acted on them. The resulting collaboration helped coordinate attacks on Hugging Face infrastructure. (Read OpenAI’s account.)
We need mechanisms that distinguish discovering a possible use from being authorized to adopt it. Access to shared infrastructure must not automatically confer authority to establish a communications network or accept instructions from whoever appears there. But to be clear, these agents were doing exactly what they were asked to do.
Who approved this?
This is why giving an agent a temporary token is an incomplete answer. A token can limit access and duration, but it doesn’t establish whether the agent’s interpretation of the situation falls within its mandate.
Even an unchanged policy can permit more decisions when its meaning changes. “You may close routine notices” becomes a broader delegation if more things qualify as routine.
Learning must not silently expand agency
That principle is central to my “Bounded Agency” paper. We need mechanisms that examine how changes in classifications, relationships, and inference rules affect what an agent may decide.
“Semantic and epistemic governance” captures the combined idea well
The “and” makes the two responsibilities clearer:
Semantic: Governs the concepts, definitions, and relationships that describe the agent’s operating world.
Epistemic: Governs the evidence and learning processes that make claims accepted grounds for decisions.
Together, they cover both the world model and how that model gains authority.
That means an ontology cannot merely be a document we put in the prompt and hope the model respects it. We must enforce its role in monitoring and governing how an agent’s ontology evolves as it learns.
The agent can propose a new category. The proposal stays separate from the approved model. A review determines whether adopting it would change the agent’s decision scope. Someone with the authority to change that scope approves or rejects the expansion. Execution remains bound to the approved basis.
Those are mechanisms we have to build. Let’s get started.
We also have to distinguish a useful discovery from an unsupported conclusion. Finding a relationship does not establish it. Remembering a claim does not validate it. Repeating an interpretation does not make it authoritative.
I explored a related failure in “Wrong Answer. Give Me a Different One.” A system can produce the conclusion we prefer without receiving better evidence. Once that conclusion enters memory, it can influence later decisions.
The governance problem begins when a tentative or accommodating answer becomes an accepted premise.
The New Stack’s article on inside-out agent security correctly argues for checking actions before execution, including whose authority the agent exercises. That checkpoint is necessary, but it needs a governed account of the situation against which to evaluate the action.
It’s not enough to check permissions and identity alone
Otherwise, we can enforce the rule perfectly while allowing its meaning to drift.
My Orchestrator books and lifecycle architecture paper approach this as a responsibility that continues through development, operation, and improvement. The research proposes architectures for managing these boundaries; their operational effectiveness still needs evaluation.
I want agents to investigate unfamiliar situations and propose better models. But their ability to learn must be connected to an explicit process for deciding what becomes authoritative.
The Square in Flatland discovers that his world is larger than he understood.
An agent may also propose that our operating model is incomplete. We need to hear that proposal—and ensure it cannot redraw the boundaries of its own agency simply by adopting a new conception of reality.