Prompts, context, and why the same model gives different companies different answers
The short answer
The model is shared; the behavior is yours
A base model gives no single answer. The user prompt, the system prompt, the context, the sampling settings, and any tuning together define the system that actually answers. Four and a half of those five dials are set on your side of the line, and that is where your accountability lives.
What you will be able to do
- Identify the five things that change a model's answer while the base model stays fixed: the user prompt, the system prompt, the context supplied to the model, the sampling settings, and the tuning applied to the weights.
- Explain why the Meta Llama 4 Maverick arena episode (2025) is a governance lesson and not just a benchmarking dispute: the number you were shown described a tuned, configured system, not the model you would deploy.
- Demonstrate, by running one question through one model under several configurations, that the same model returns materially different answers depending on prompt and context.
- Distinguish the parts of the deployed system your organization controls (prompt, context, retrieval sources, sampling, guardrails) from the parts the model vendor controls (base weights, provider-side tuning).
- Apply the principle that every prompt and every context choice is a governance decision your organization is accountable for, not an incidental implementation detail.
- Produce a documented comparison of one model's answers across configurations, a shipped-configuration decision with a stated reason, and a short note on how this behavior could fail, added to your build log.
- Judge when a vendor's benchmark number is relevant to your deployment and when it is not, and what you would have to test yourself before trusting it.
The ideas, one by one
A benchmark describes a configured system, not "the model."
The Meta Llama 4 Maverick episode (2025) took second place on a public leaderboard with a tuned experimental entry (Elo 1417) that did not describe the released weights (TechCrunch, 2025). Convert every model claim into three questions: which exact version, under what prompt and context, measured how.
The system prompt is policy
A standing instruction that decides what your AI will and will not do for every user is a policy, whatever folder it lives in. Version it, own it, review it. A prompt one engineer can quietly edit is an ungoverned control.
Context is often the biggest dial
With retrieval-augmented generation, the model answers from what you hand it. Two companies pointing the same model at different documents get different answers by design. Wrong retrieval produces a confidently wrong answer, so the context source is a first-class governance concern.
Temperature zero improves repeatability but does not guarantee it, and never guarantees correctness
Batching, hardware, and mixture-of-experts routing can still cause run-to-run variation, and a confidently wrong low-temperature answer is wrong every time. Verify and log actual outputs; do not assume the setting delivered the property.
Same model, different companies, different answers is the normal case, not an anomaly
It is the direct, expected result of different configurations. "We use the standard model" is not a defense; the parts you configured are the parts anyone will examine.
Change one dial at a time
To understand or improve behavior, hold everything fixed and move a single dial. Changing several at once teaches you nothing about which change did what. This is controlled experimentation, and it is how your comparison becomes evidence.
Configurations are artifacts
The prompt, the context sources, and the settings are things you write down, version, and can produce on demand, because they define the system that was running when something happened. They feed your "how my model fails" explanation and your eval suite directly.
Reach for the cheap dials first
When a system misbehaves, check context, then the system prompt, then sampling, and only then consider tuning. Most "the model is wrong" problems are really missing context or a loose system prompt, and fixing them on the cheap, reversible rungs saves cost and leaves a cleaner audit trail than changing the weights.
A prompt is only half an instruction
The context completes it, and the context is local to your organization, which is why prompts do not port cleanly and why copying a competitor's prompt does not copy their behavior. The reproducible unit is the full configuration, not the prompt alone.
Behavior can shift without you touching anything
The vendor can update the base model, a retrieval source can go stale, or a shared prompt can be edited by someone else. Versioning and monitoring your configuration is what lets you tell a real model change from a self-inflicted one, so "it worked before" is a hypothesis to re-verify, not a fact.
Listen to it
The same lesson, talked through by two hosts as episode 8 of the podcast.
Real cases
These examples show the five dials moving real behavior. The Maverick episode is the deep case; the others illustrate specific dials with verifiable, sourced facts.
Example 1: Meta Llama 4 Maverick, the tuning dial and the benchmark trap (2025). Covered in depth in Section 3C. The one-line version: a second-place Elo of 1417 belonged to a conversationally tuned experimental entry, "Llama-4-Maverick-03-26-Experimental," while the released weights, "Llama-4-Maverick-17B-128E-Instruct," ranked lower once scored, and the arena maintainers apologized and changed policy (TechCrunch, 11 April 2025). The dial: provider-side tuning, plus the reminder that a benchmark measures a configured system under a specific context, not "the model."
Example 2: Published system prompts, the standing-instruction dial (Anthropic, 2024 onward). Anthropic began publishing the system prompts that shape its Claude assistants' behavior, exposing to the public how much of an assistant's tone, refusals, and default behavior comes from standing instructions rather than from the weights (Anthropic system prompt release notes, 2024). The governance point: the behavior users attribute to "the AI" is substantially set by a text file the operator writes, which is a policy surface, not a mystery of the model.
Example 3: Retrieval-augmented generation, the context dial (Lewis et al., NeurIPS 2020). The RAG technique, which retrieves relevant passages from a chosen document set and inserts them into the prompt so the model answers from that material, is now the standard way organizations ground a general model in their own knowledge. Two organizations pointing the same model at different document sets get different answers by design. This is the single clearest mechanism behind "same model, different companies, different answers," and it is entirely operator-controlled.
Example 4: The temperature dial in everyday API use (OpenAI and other model APIs, documented parameter). Essentially every commercial model API exposes a temperature parameter that scales output randomness, with vendor documentation advising lower values for focused, deterministic-leaning tasks and higher values for creative ones. The same prompt to the same model at temperature 0.1 versus 1.0 yields noticeably different variability. Operators who need consistent, auditable answers (regulated decisions, factual lookups) turn it down; the choice is a recorded risk decision, not a default to accept unexamined.
Example 5: A system-prompt change that produced harm (pointer). The clearest cautionary case of a single standing-instruction change flipping a model's behavior into harmful output is owned by the previous topic and is treated there as the "fixes are never free" anchor. (see Topic 1.3) It belongs in your mind here as proof that dial 2 is not cosmetic: one line in the system prompt changed what the model said to the world.
Example 6: A jailbroken assistant, the user-prompt dial from the attacker's side (pointer). Users can also turn dial 1 against you, crafting prompts that override your system prompt and pull the model off its intended behavior. The deep treatment of adversarial user prompts against a deployed assistant is owned by the red-teaming topic. (see Topic 4.3) Hold it here as the reminder that the user prompt is a dial your adversaries reach too, which is why the system prompt and guardrails you set have to withstand hostile input, not just cooperative input.
Example 7: The same model measured on different setups reads as a different model (Stanford, pointer). Researchers documented that a widely used assistant's behavior shifted between versions over months, on tasks as basic as identifying prime numbers, which is a reminder that even the vendor's own model changes under you over time. The deep treatment of behavior changing across versions, and why it demands living model documentation, is owned by the model-and-system-cards topic. (see Topic 10.3) The narrow point for this topic: if the model can shift under you unannounced, then a benchmark taken at one moment is even weaker as a standing promise, and pinning the exact version becomes a governance requirement, not a nicety.
Example 8: Customer fine-tuning as an owned artifact (documented industry practice). Major model providers offer fine-tuning, where an organization trains the base model further on its own examples to produce a customized version. This is dial 5 in your own hands, and it is the one dial on the operator side that changes the weights rather than the surrounding setup. The governance consequence is concrete: once you fine-tune, you have created a new model artifact that you now own and must govern, including the examples you trained on (their provenance and quality), the failure modes you may have introduced, and the version control of the tuned weights. A fine-tune can produce behavior no prompt could, but it moves work from the cheap, visible dials to the expensive, opaque one, so experts reach for it last (Section 3H) and document it as a first-class system, not a tweak.
Example 9: The same model, two jurisdictions, two answers by design (globally sourced illustration). A multinational deploying one assistant across markets does not, and should not, give every region the same answer to a compliance question. The operator changes the context and the system prompt per market: a data-handling question retrieves the local rules and the system prompt names the local obligations, so a user in the European Union, in Brazil, and in Japan can receive three different, each locally correct answers from one base model. This is the five dials used deliberately for good: divergence is not a bug to stamp out but a configuration choice that reflects real legal difference. It also underlines the governance point, because each regional configuration is a distinct system to document and defend, and getting the wrong region's context in front of a user would produce a confidently wrong compliance answer. The specific divergence of the world's rulebooks is developed later; here it is an example of context and system prompt used on purpose to make one model speak correctly to different places.
Across these, the pattern is one lesson from many angles: the model is a constant only in the moment and only for a fixed configuration, and the behavior your organization is accountable for is produced by dials you or your users set, and sometimes by the vendor changing the base under you.
Where people go wrong
- "We use the same model as everyone else, so our behavior is standard." Wrong, and dangerously comforting. The shared base model is the part you did not configure. Your system prompt, your retrieved context, and your settings are what actually produce your answers, and they are unique to you. "Standard model" gives you no governance cover, because nobody is going to examine the base weights; they are going to examine what you told the model to do.
- "The benchmark score tells us how the model will perform for us." A benchmark describes a specific version under a specific setup measured by a specific party. The Maverick episode showed a leaderboard-topping Elo of 1417 that belonged to a tuned experimental entry, not the released weights (TechCrunch, 2025). Treat vendor numbers as leads to verify on your own inputs, never as predictions of your deployment.
- "Prompt engineering is a nice-to-have, the real governance is in the policy." The prompt, especially the system prompt, is policy: it decides what your AI does for every user. Leaving it to whoever happens to be editing the code, unversioned and unreviewed, is running an ungoverned control. The prompt is one of the first artifacts a serious governance function versions and owns.
- "Set temperature to zero and the model is deterministic, problem solved." Temperature zero improves repeatability but does not guarantee bit-identical output in production (batching, hardware, and mixture-of-experts routing can still cause variation), and it does nothing for correctness. A confidently wrong temperature-zero answer is wrong every single time. Verify and log actual outputs; do not assume the setting delivered the property.
- "Context is just background, the model mostly answers from what it knows." For grounded enterprise systems the retrieved context is often the dominant driver of the answer. That is the point of retrieval-augmented generation: the model answers from what you handed it. If your retrieval pulls the wrong document, the model will faithfully give a wrong answer with total confidence. Context is not background; it is frequently the largest dial.
- "If two teams use the same model, it is one system to govern." If they wired it to different context with different system prompts and settings, they are running different systems with different risk profiles. Governing them as one, or listing them as one line in your inventory, undercounts your real exposure. Capture the configuration, not just the model name.
- "A more capable model would remove these problems." A more capable base model does not remove the dials; it still answers according to the prompt, context, and settings you give it. Capability raises the ceiling; it does not take the instrument out of your hands. The governance work of setting, versioning, and testing the configuration remains yours at every capability level.
- "The vendor tuned it, so any weirdness is the vendor's fault." Provider-side tuning is the vendor's, but the system prompt, the context, the sampling, and any fine-tune you applied are yours. When a claimant asks why your system said something, "the vendor tuned it" will not cover a harm that flowed from your configuration. Know precisely which side of the line each behavior came from.
- "Copying a good prompt copies the good behavior." A prompt is only half an instruction; the context completes it. A system prompt that produces excellent, grounded answers in one organization can produce confident wrong answers in another that connected it to different or worse data. Prompts do not port cleanly across organizations because the meaning of "our policy," "our product," and "our customer" resolves against local context. A prompt is also model-dependent: instructions phrased and weighted for one vendor's model can be followed more loosely, or interpreted differently, by another vendor's model, even with identical context. Copy the prompt if you like, but you must supply your own correct context and re-test the result on the model you are actually running.
- "Whatever documents we put in context, only the model's answer needs checking." The context you retrieve is itself an input an attacker can reach. If a hostile actor can add or edit a document your system later retrieves, such as a support ticket, a wiki page, or an uploaded file, they can plant instructions inside it that the model treats as part of its prompt when that document is pulled into context, a technique known as context or retrieval poisoning. Checking only the model's final answer misses this; the documents feeding your retrieval need the same scrutiny as the prompt itself.
- "We can paste any of our documents into context; the model just answers from them." Context sent to a hosted model is data leaving your organization's boundary. Pasting a document with unredacted personal or health information into a third-party model's context, whether for a demo, a test, or a live answer, can create the same privacy exposure as any other unauthorized data disclosure. Minimize what you place in context to what the answer actually requires, and confirm your contract and your data classification rules permit sending that material to that provider before you do it.
- "The prompt worked yesterday, so it works today." Behavior can shift without you touching the prompt at all: the vendor can update the base model under you, a retrieval source can go stale, or someone else can edit a shared system prompt. A configuration that is not versioned and monitored can degrade silently, and you will not know whether the cause was the model, the data, or a quiet edit. Treat "it worked before" as a hypothesis to re-verify, not a fact, especially after any vendor model update.
- "Fine-tuning is the serious fix; prompting is a hack." The reverse is closer to the truth as a first move. Prompt and context changes are cheap, reversible, and transparent; fine-tuning is expensive, hard to reverse, and creates a new model artifact you must govern. Experts fine-tune last, only after proving the lighter dials cannot do the job. Reaching for a fine-tune to solve a retrieval or system-prompt problem is the actual hack, and an expensive one.
Questions people ask
- What is base model (base weights)?
- The trained neural network your organization licenses from a model vendor, before you add any system prompt, context, or settings. It is the shared component: the same base model can power very different systems depending on how it is configured. You cannot change the base weights; you can only change what you put around them.
- What is user prompt?
- The literal question or instruction a user submits to the model. The most visible dial, and one that adversaries can also use against you by crafting inputs that try to override your standing instructions.
- What is system prompt?
- A standing set of instructions the operator places above every user message, usually hidden from the user, that sets the assistant's role, rules, tone, and refusals. It is the most powerful everyday governance surface most organizations have, and functionally it is policy: it decides what the AI does for every user until someone changes it. More on System prompt
- What is context (supplied context)?
- The material placed in front of the model alongside the question, including pasted documents, conversation history, and retrieved passages from your own data. For grounded enterprise systems it is frequently the largest single driver of the answer.
- What is retrieval-augmented generation (RAG)?
- A pattern in which the system searches a chosen set of documents, pulls the most relevant passages, and inserts them into the prompt so the model answers from that material rather than only from its training. It is the main reason the same model gives two organizations different answers, because each retrieves from its own documents (named and demonstrated by Lewis et al., NeurIPS 2020). More on Retrieval-augmented generation (RAG)
Keep going
This lesson builds How models work, at a governance depth, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →