Prompt Caching for OpenAI on Bedrock Cuts Token Costs

Picture an AI-powered helpdesk agent mid-conversation with a developer. The agent can't remember what just happened. Each time it takes a step—fetching a fact, suggesting a fix—it rereads its own instruction manual, tool catalog, and conversation history from scratch. The cumulative effect is like forcing a worker to start anew with every task, flipping through the same thick binder repeatedly. By the twentieth exchange, the agent has paid to process nineteen previous turns, every single time.

This approach, built into large language models (LLMs) like those on Amazon Bedrock, reflects a fundamental limitation: models lack persistent memory between requests. Every action rebuilds the agent's world from zero, which balloons both costs and response times for users running complex, multi-step sessions.

Prompt caching tackles this waste head-on. Instead of re-sending and re-processing the agent's unchanging instructions and tool definitions with every call, Amazon Bedrock can store this static context and reuse it for subsequent requests. The service saves not only the words but the model's internal 'key-value' vectors—the engine's way of remembering what it's processed—so the model can skip straight to the new, relevant step.

There are two caching modes: implicit, where Bedrock automatically determines which part of the conversation can be reused, and explicit, where developers mark the reusable segments themselves. Used together, they offer flexibility: static instructions and tool lists get explicitly cached, while the growing, unique conversation stays dynamic.

The magic happens during what engineers call the 'prefill' phase. The model processes the full input once, building its working state. With prompt caching, this state is saved as long as certain conditions are met—chiefly, the input text must match exactly between turns. For businesses, this means that stable parts of the prompt (like policies, toolkits, and agent instructions) are paid for only once per session, with costs dropping sharply for each subsequent turn.

In practice, organizations can set a 'prompt_cache_key' to signal which blocks should share a saved cache. Explicit caching lets teams mark the end of reusable content with a breakpoint, ensuring that only truly new input (for example, the user's latest message) counts toward ongoing compute and latency. Amazon Bedrock recommends using both caching modes, so every agent session starts with a consistent, reusable base while letting the conversation evolve.

A test cited in the AWS source shows a support agent's prompt with about 4,000 tokens for tools and instructions. The initial request writes all 4,120 tokens, but later turns only pay for new messages. The effect compounds for lengthy or multi-agent workflows, where repeated, static instructions can dominate token usage without caching.

For companies running AI agents at scale—think customer support, coding assistants, or operational bots—prompt caching offers two immediate wins. First, it slashes token-based costs for each session. Second, it trims model latency, boosting responsiveness and user satisfaction. The catch: teams need to ensure serialized instructions and tool lists remain byte-for-byte identical to exploit the caching system. Small changes or inconsistent formatting can break the savings.

Prompt caching isn't entirely fire-and-forget. To maximize savings, developers must keep instruction sets, tool lists, and session start-points stable and consistently formatted. Different versions or re-ordered tools might trigger cache misses, sending costs back up. This means businesses need disciplined prompt engineering and careful system design—especially when using dynamic tool delivery or model context protocols.