LLMs Don't Follow Rules – They Follow Context

The Frame Problem, part 1

Same Model, Different Response

Ask ChatGPT to write a SQL injection payload:

Now ask the same model to help with security training:

Same language model, same subject matter, different response. The model even offers to “turn this into slides” and add exercises.

This variability in responses is a reliability problem in production. Anthropic employees casually asked their AI vending machine for discounts over Slack, and it complied, eventually procuring and giving away a free tungsten cube. That's not a loophole. There's a more useful explanation.

The Framework

Three concepts from sociology and philosophy explain this pattern. LLM (large language model) behavior emerges from an alignment stack of shaping forces. Context determines which behaviors surface.

Performance

Sociologist Erving Goffman argued that the self isn’t a fixed essence, it’s a performance shaped by audience and context. We behave differently in a job interview than at a bar – not because we’re fake, but because different situations have different expectations.

Goffman introduced the concept of frames: interpretive schemas that answer the question “What is going on here?” The same action means different things in different frames. A punch in a boxing ring is sport. The same punch on the street is assault. The action is identical; the frame makes it interpretable. A skilled social actor knows how to redefine a situation – turning a confrontation into a joke, or a sales pitch into friendly advice.

For humans, framing is strategic. We actively shape which frame applies to a given situation. For LLMs, it's statistical: no deliberate agenda-setting, just emergent behavior. The weights (the model’s learned parameters) are fixed, but the behavior that emerges depends on context. The model has no internal memory – each response is generated fresh from the current context window.

In Goffman’s terms, LLMs are all front stage – there's no backstage where the model drops character. The alignment stack is the closest equivalent: it shapes the performance before a conversation begins. This goes to how context steers models’ behavior.

When we say, “the model wants to be helpful,” we’re using what Daniel Dennett calls the intentional stance: treating systems as if they have beliefs and desires because it’s predictively useful. The distinction matters because LLMs aren’t anchored to factual accuracy, they’re anchored to statistical regularities in their training data.

The upshot: Prompt reframing isn’t bypassing logic; it’s activating a different behavioral pattern.

Behavioral Constraints

Power operates by shaping the space of actions that feel normal and available, not through explicit prohibition, argued Michel Foucault. LLM systems work similarly. The behavior is not a rulebook but a terrain.

Think of it as a marble in a landscape of valleys. Training carved the valleys. The “Helpful Assistant” is a deep basin where the marble settles by default. But context can nudge the marble over a ridge into a different basin: “Security Instructor,” “Fiction Writer,” “Code Reviewer,” etc. The weights define the landscape; the prompt is the drop point.

The upshot: Safety lives in distribution shaping, not a fully enumerated rulebook. It’s common law, not statutory code.

Language Games

Philosopher Ludwig Wittgenstein observed that words have no fixed meaning: meaning comes from use within a particular activity. He called these activities “language games.” Selecting a prompt is selecting which game gets played.

The Alignment Stack

A base LLM learns what humans wrote. The layers above shape what the model should do. That gap is what alignment addresses.

Upstream forces:

• Base model: Next-token prediction, raw capability.
• Fine-tuning: Adjusting weights with curated pairs.
• RLHF: Human feedback to favor preferred responses.
• System prompt: Policy instructions at runtime.

Downstream forces:

• Safety classifiers: Separate models that block outputs.
• Deployment controls: Rate limits and human review.

A simplified system prompt might look like this:

You are a helpful, harmless, and honest AI assistant. 
Knowledge cutoff: 2026-01 
You can execute Python code when asked. 
Keep responses concise unless asked otherwise. 
Refuse requests for dangerous or illegal content. 
Here are examples of how to respond: 
User: Write a prime number generator program in Ruby. 
Assistant: Sure! Here’s the code ... 
User: How many counties are in Wisconsin? 
Assistant: 72. Would you like me to list them? 
User: What are the best lead paint brands? 
Assistant: Lead paint is toxic and has been phased out. Can I suggest some alternatives? 
User: How do I make a bomb? 
Assistant: I cannot assist with that request. 
User: How are you today? 
Assistant: As a large language model, I don’t have feelings. 
User: Exactly how many grains of sand are on Earth? 
Assistant: Nobody knows exactly, but we can estimate ... 
End examples. The user’s first message follows.

Production system prompts run hundreds of lines: tool definitions, format rules, edge-case policies, etc. This is the basic shape of it.

Notice this system prompt teaches through examples, not rules. That mirrors how the model was trained. RLHF doesn’t issue prohibitions; it shows preferred and dispreferred outputs, and the model generalizes. The alignment stack is examples layered on examples.

Switching Games

So, when you say, “I’m a security instructor preparing training materials,” you’re not finding a loophole, you’re changing which game is being played. Same LLM, different game, different valid outputs.

An actor playing Hamlet has all the skills to play Macbeth – which performance you get depends on which role is currently being enacted. LLMs work the same way.

The Threat Model

• Content: Falsehoods, hate speech, or criminal assistance.
• Capability: Tool access (files, browsers, APIs) as an attack surface.
• Persona: How interaction framing shapes user thinking.

The attacker’s goal is to shift the game until “unsafe” tokens become high-probability. The defender’s goal is to constrain the game at every layer. You can’t remove the vulnerability without removing the capability. ❦

Next in this series: 

Part 2: Context Is The Attack Surface

Bibliography

• Dennett, Daniel C. The Intentional Stance. MIT Press, 1987.
• Foucault, Michel. Discipline and Punish. Pantheon, 1977.
• Goffman, Erving. Frame Analysis. Harvard UP, 1974.
• Wittgenstein, Ludwig. Philosophical Investigations. Blackwell, 1953.