Response Caching Strategy

Compliance tests use LLM response caching for reproducible, cost-effective testing.

Why Caching?

Reproducibility - Same inputs produce same outputs

Cost reduction - Avoid repeated API calls

Speed - Cached responses are instant

Offline testing - Run tests without API access

How It Works

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│ Test Runner │────►│    Cache    │────►│  LLM API    │
│             │◄────│             │◄────│             │
└─────────────┘     └─────────────┘     └─────────────┘
       │                   │
       │    Cache Hit      │
       │◄──────────────────│
       │
       │    Cache Miss
       │──────────────────────────────────────────────►
                                                    │
       │◄─────────────────────────────────────────────
       │         Response (saved to cache)

Cache Modes

read_write (Default)

Best for CI/PR testing:

Check cache first

On miss, call LLM and save response

Deterministic results

python -m compliance.runner --cache-mode read_write

write_only

Best for nightly runs:

Always call LLM

Save responses to cache

Detects regressions with fresh data

python -m compliance.runner --cache-mode write_only

disabled

Best for pre-release:

No caching

Tests real LLM behavior

Full variability

python -m compliance.runner --cache-mode disabled

Cache Key Generation

Cache keys are generated from:

Model name

Messages (content only)

Temperature

Tools (if present)

Other deterministic parameters

def cache_key(request: dict) -> str:
    key_data = {
        "model": request.get("model"),
        "messages": normalize_messages(request.get("messages", [])),
        "temperature": request.get("temperature", 0),
        "tools": normalize_tools(request.get("tools")),
    }
    return hashlib.sha256(json.dumps(key_data, sort_keys=True)).hexdigest()

Temperature Strategy

Test Type	Temperature	Cache	Rationale
CI/PR	0	read_write	Deterministic
Nightly	0.3	write_only	Some variation
Pre-release	0.7	disabled	Full variation

Temperature = 0

Most reproducible

Same prompt → same response (usually)

Best for CI

Temperature > 0

Introduces randomness

Tests edge cases

May reveal fragile assertions

Cache Structure

compliance/.cache/
├── manifest.json          # Index of cached responses
├── openai/
│   ├── abc123.json        # Cached response
│   └── def456.json
├── anthropic/
│   └── ...
└── google/
    └── ...

Cached Response Format

{
  "request": {
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0
  },
  "response": {
    "id": "chatcmpl-...",
    "choices": [...]
  },
  "metadata": {
    "cached_at": "2026-01-29T10:00:00Z",
    "ttl": 604800
  }
}

Cache Invalidation

Automatic

Cache entries expire after TTL (default: 7 days)

Model updates invalidate relevant entries

Manual

# Clear all
rm -rf compliance/.cache/

# Clear by provider
rm -rf compliance/.cache/openai/

# Clear specific entry
rm compliance/.cache/openai/abc123.json

CacheSkill Implementation

Each SDK implements its own CacheSkill:

import { Skill, hook } from 'webagents';
import * as crypto from 'node:crypto';
import * as fs from 'node:fs/promises';
import * as path from 'node:path';

class CacheSkill extends Skill {
  readonly name = 'cache';
  constructor(private cacheDir: string = '.cache') {
    super();
  }

  private cacheKey(messages: Array<{ role: string; content: string }>, options: { model?: string; temperature?: number }): string {
    const data = {
      messages: messages.map((m) => ({ role: m.role, content: m.content })),
      model: options.model,
      temperature: options.temperature ?? 0,
    };
    return crypto.createHash('sha256').update(JSON.stringify(data)).digest('hex');
  }

  @hook({ lifecycle: 'before_llm_call', priority: 0 })
  async checkCache(_data: unknown, context: any): Promise<void> {
    const key = this.cacheKey(context.messages, context.options);
    const file = path.join(this.cacheDir, `${key}.json`);
    try {
      const cached = JSON.parse(await fs.readFile(file, 'utf8'));
      context.response = cached.response;
      context.skipLlm = true;
    } catch {
      // cache miss
    }
  }

  @hook({ lifecycle: 'finalize_connection', priority: 100 })
  async saveCache(_data: unknown, context: any): Promise<void> {
    if (!context.response || context.skipLlm) return;
    const key = this.cacheKey(context.messages, context.options);
    await fs.mkdir(this.cacheDir, { recursive: true });
    await fs.writeFile(
      path.join(this.cacheDir, `${key}.json`),
      JSON.stringify({
        request: { messages: context.messages, model: context.options?.model },
        response: context.response,
        metadata: { cachedAt: new Date().toISOString() },
      }),
    );
  }
}

Best Practices

1. Commit Cache Selectively

# .gitignore
compliance/.cache/

# But optionally commit known-good responses
!compliance/.cache/fixtures/

2. Cache Warming

Pre-populate cache in CI:

- name: Warm cache
  run: |
    python -m compliance.runner --cache-mode write_only
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

3. Cache Validation

Periodically verify cached responses are still valid:

python -m compliance.runner --cache-mode write_only --tag core

4. Provider-Specific Caching

Different providers may have different caching needs:

cache_config = {
    "openai": {"ttl": 604800},      # 7 days
    "anthropic": {"ttl": 259200},   # 3 days
    "google": {"ttl": 86400},       # 1 day
}

On this page