Prompt Caching

PickLayer supports prompt caching across all AI providers with automatic translation between different caching mechanisms.

Overview

Prompt caching can significantly reduce costs and latency for applications with:

• Large system prompts
• Reusable context documents
• Long multi-turn conversations

Cost savings example (Anthropic):

• Regular input tokens: $3.00 / 1M tokens
• Cached input tokens: $0.30 / 1M tokens (90% savings)

How It Works

PickLayer uses a two-layer architecture:

1. Intent Layer (cache_control): You specify where caching should occur
2. Implementation Layer: PickLayer translates to provider-specific formats

Using Cache Control

Basic Format

{
  "model": "anthropic/claude-3-5-sonnet",
  "input": [...],
  "cache_control": {
    "mode": "inline_blocks",
    "blocks": [
      { "location": "system" },
      { "location": "user", "position": "last" }
    ]
  }
}

Block Options

Each block specifies where to inject cache markers:

• location (required): "system", "user", or "assistant"
• position: "first", "last", or "all" (default: "last")
• index: Specific message index (0-based, alternative to position)
• type: "ephemeral" (default)

Common Patterns

// Cache system prompt only
{ "location": "system" }
 
// Cache last user message
{ "location": "user", "position": "last" }
 
// Cache first user message
{ "location": "user", "position": "first" }
 
// Cache all assistant responses
{ "location": "assistant", "position": "all" }
 
// Cache specific user message by index
{ "location": "user", "index": 2 }

Provider-Specific Usage

Anthropic

Full support with up to 4 cache breakpoints:

{
  "model": "anthropic/claude-3-opus",
  "input": [
    { "role": "system", "content": "You are an expert..." },
    { "role": "user", "content": "Explain quantum computing" }
  ],
  "cache_control": {
    "mode": "inline_blocks",
    "blocks": [
      { "location": "system" },
      { "location": "user", "position": "last" }
    ]
  }
}

OpenAI

Automatic caching - no configuration needed:

{
  "model": "openai/gpt-4o",
  "input": [...]
  // Automatically cached if ≥ 1024 tokens
}

Gemini

cache_control markers map to Google’s Cached Content API  (cachedContents resource — distinct from inline markers). PickLayer maintains a content-hash → resource-name stash so identical content is reused across requests:

{
  "model": "gemini/gemini-2.5-pro",
  "input": [...],
  "cache_control": {
    "mode": "inline_blocks",
    "blocks": [{ "location": "system" }]
  }
}

• Default TTL: 5 minutes, aligned with Anthropic’s ephemeral semantics.
• Extended TTL: pass cache_control.ttl: "1h" (or any value in the 1-minute to 48-hour range) to override.
• Hit signal: usage.cache_read_input_tokens reflects Google’s cachedContentTokenCount.
• Fallback: if the cache lookup or create call to Google fails, the request is sent uncached (logged).

Bedrock (Anthropic & Nova)

PickLayer emits AWS-native cachePoint blocks for caching-capable Bedrock models. Use the same canonical cache_control request shape — provider translation is automatic:

{
  "model": "bedrock/anthropic.claude-sonnet-4-20250514-v1:0",
  "input": [...],
  "cache_control": {
    "mode": "inline_blocks",
    "blocks": [
      { "location": "system" },
      { "location": "user", "position": "last" }
    ]
  }
}

Nova Micro is not caching-capable per AWS documentation. Models outside this set receive a downgraded warning and the markers are skipped (no failure — the request still runs uncached).

xAI

Uses automatic caching via headers:

{
  "model": "xai/grok-3",
  "input": [...],
  "cache_control": {
    "mode": "inline_blocks",
    "blocks": [{ "location": "system" }]
  }
}
// Warning: "cache_control (xAI uses automatic caching via x-grok-conv-id header)"

OpenRouter

Universal gateway - passes through to all providers:

{
  "model": "openrouter/anthropic/claude-3-opus",
  "input": [...],
  "cache_control": {
    "mode": "inline_blocks",
    "blocks": [
      { "location": "system" },
      { "location": "user", "position": "last" }
    ]
  }
}

cache_creation_tokens in the usage event is sourced from OpenRouter’s prompt_tokens_details.cache_write_tokens field; cached_input_tokens from prompt_tokens_details.cached_tokens. Both surface back through PickLayer billing events so local pricing math attributes the cache-write tier correctly.

Python SDK

The Python SDK provides helper functions for creating cache control configurations:

from picklayer import (
    PickLayerClient,
    create_cache_control,
    create_cache_control_block,
    create_anthropic_cache_control,
)
 
client = PickLayerClient(org_handler="...", app_handler="...")
 
# Simple pattern: cache system and last user message
resp = client.responses({
    "model": "anthropic/claude-3-5-sonnet",
    "input": messages,
    "cache_control": create_cache_control([
        create_cache_control_block("system"),
        create_cache_control_block("user", position="last"),
    ]),
})
 
# Anthropic helper (up to 4 breakpoints)
resp = client.responses({
    "model": "openrouter/anthropic/claude-3-opus",
    "input": messages,
    "cache_control": create_anthropic_cache_control(
        "system",
        ("user", "last"),
    ),
})
 
# Check provider support
from picklayer import supports_cache_control
 
if supports_cache_control("anthropic"):
    print("Provider supports explicit cache_control markers")

Reading Cache Usage From Responses

Cache token counts are emitted in the response usage object using the same field names as each endpoint’s canonical wire shape. PickLayer never invents new top-level cache fields — read the same fields you’d read directly from OpenAI / Anthropic.

These fields are zero-suppressed: the sub-object (prompt_tokens_details, input_tokens_details, output_tokens_details) is omitted entirely when all of its fields would be zero. Cross-provider routes (e.g. an Anthropic model called via /v1/chat/completions) emit OpenAI’s nested shape; canonical Anthropic flat fields are not leaked.

Messages API

When using the Anthropic Messages API, cache control markers are placed inline:

{
  "model": "anthropic/claude-3-5-sonnet",
  "messages": [...],
  "system": [
    {
      "type": "text",
      "text": "You are a helpful assistant...",
      "cache_control": { "type": "ephemeral" }
    }
  ]
}

PickLayer automatically extracts these markers and converts them to the canonical cache_control format.

Migration from Legacy Format

If you were using prompt_caching:

Old format:

{
  "prompt_caching": {
    "enabled": true,
    "injection_points": [{ "location": "system" }]
  }
}

New format:

{
  "cache_control": {
    "mode": "inline_blocks",
    "blocks": [{ "location": "system" }]
  }
}

Both formats are supported - prompt_caching is automatically converted internally.

Best Practices

1. Cache large static content: System prompts, documents, context
2. Limit cache points: Anthropic and Bedrock-Anthropic max 4 breakpoints
3. Consider your provider:
   • Use explicit cache_control for Anthropic, Bedrock-Anthropic, Bedrock-Nova, Gemini, and OpenRouter
   • Rely on automatic caching for OpenAI
4. Monitor warnings: Check metadata.warnings for stripping notices (xAI, or models that are not caching-capable)
5. Test cache hit rates: Measure actual savings in your use case

Troubleshooting

”cache_control stripped” warning

Expected behavior for xAI (which uses automatic caching via headers) and for any Bedrock model outside the caching-capable allowlist. The request still runs — just uncached.

Cache misses

• Ensure cache points are on content ≥ 1024 tokens
• Check that content hasn’t changed between requests
• Anthropic requires exact content match for cache hits

Max blocks exceeded

Anthropic supports max 4 cache breakpoints. Additional blocks are ignored with a warning.

See Also

• Responses API
• Messages API
• Python SDK README