Spring AI Test Tools

Deterministic testing
for Spring AI

Golden AI responses. Zero-token replays.
Your test code never changes.

Get started GitHub

Java 21 · Spring Boot 4.0 · Spring AI 2.0 · Apache-2.0

See each capability, not just read about it

Six mechanisms, six independent fixture families. Pick one — every diagram below is annotated from this project's own verified end-to-end tests, not illustrative guesswork.

FIRST RUN slow · costs tokens · needs network Your test ChatClient Real LLM ~2.9 – 46.7 s writes cassette.json committed to git EVERY RUN AFTER instant · $0 · fully offline Your test ChatClient reads 0.8 ms

What this proves: a real cache miss records a fixture, and a real cache hit replays it with zero additional HTTP requests — OllamaEndToEndTests.

EVERY RUN, FROM THE FIRST ONE no fixture, no hash, no Spring context needed Your test .respondingWith(…) hand-authored, inline or file ChatModel (stub) 0 network calls

What this proves: a stub never touches the network or a fixture file — the exact scenario you wrote is what comes back, every time — StubbingErrorScenariosTest.

RECORD real @Tool method runs once Your test ChatClient Real @Tool side effect fires writes tool-cassette.json name + arguments keyed REPLAY real method never runs Your test ChatClient @Tool bypassed injects result 0 side effects

What this proves: the real @Tool method runs exactly once — on the call that records it — and zero times on an identical replay, full isolation, by default — OllamaToolIsolationEndToEndTests.

FIRST RUN real embedding call Your test EmbeddingModel Real model float[2048] writes embedding-cassette full vector, not a hash EVERY RUN AFTER byte-for-byte identical vector Your test EmbeddingModel reads exact vector

What this proves: a replayed embedding is assertThat(replayed).isEqualTo(recorded) exact, not "same length" or "close enough" — OllamaEmbeddingEndToEndTests.

FIRST RUN real judge model call Your test Relevancy- Evaluator Real judge same ChatClient path writes cassette.json no new fixture type EVERY CI RUN deterministic verdict, not live Your test Relevancy- Evaluator reads same verdict, every time

What this proves: the same judge verdict replays deterministically in CI — a frozen regression check, not a live evaluation — and a genuinely different judged answer still reaches the judge model again — OllamaEvaluatorEndToEndTests.

FIRST RUN chunks arrive one by one Your test .stream() Real LLM Hel · lo · ! writes cassette.json raw chunk sequence EVERY RUN AFTER same chunks, same order, no delay Your test .stream() reads same chunks

What this proves: a replayed stream emits the exact same chunk-for-chunk sequence — tool calls included — asserted chunk-by-chunk, not just on the aggregated text — OllamaStreamingEndToEndTests.

The same test, zero code changes

Interception happens at Spring AI's advisor layer, attached through ChatClientBuilderCustomizer. Nothing under test — and nothing in production — knows the cache exists.

application-test.ymlthe entire integration
spring:
  ai:
    test:
      vcr:
        enabled: true
        mode: RECORD_OR_REPLAY   # REPLAY_ONLY in CI
OrderStatusTest.javaunchanged
@SpringBootTest
class OrderStatusTest {

    @Autowired
    private ChatClient.Builder chatClientBuilder;

    @Test
    void answersAboutTheOrder() {
        String answer = chatClientBuilder.build()
            .prompt()
            .user("Status of order ORD-4471?")
            .call()
            .content();

        assertThat(answer).contains("shipped");
    }
}

First run records against a real model. Every run after replays from disk — same test, same assertion, no edits.

Everything it does

Each capability is its own package, with its own fixture family and its own cache directory — nothing is a special case of anything else.

⏺ Record & Replay Capture a real model's answer once; replay it forever, offline, byte-for-byte. ✎ Stubbing Hand-author a response — inline or from a file — for timeouts, refusals and edge cases you can't record. ✓ Assertions Fluent AssertJ checks on tool calls, finish reasons and JSON fields — no model call to assert. ≈ Semantic Assertions Compare answers by meaning via embeddings — deterministic, because the embedding replays too. ⛨ Tool Isolation Replay a @Tool call's result without re-running its side effects. Safe by default. ⋮⋮ Embedding Replay EmbeddingModel calls cached independently of chat — a replayed vector is exact, not approximate. ⚖ Evaluator Testing Spring AI's own evaluators, deterministic in CI or live on demand — the choice it doesn't offer. ⇉ Streaming Flux responses replayed chunk-for-chunk, tool calls included. No single-chunk fake.

Why this exists

Testing an LLM-backed application is slow, expensive and unrepeatable by default.

Slow

Testcontainers + Ollama re-runs full inference every mvn test. One cold call: ~47 s.

Expensive

Against a hosted provider, every test on every branch is billable tokens — times every developer, times every CI job.

Unrepeatable

The same prompt can answer differently tomorrow. A test asserting on model output is flaky by construction.

Untestable in CI

No GPU, no model container — and a provider API key in a pipeline is a security problem, not a test strategy.

How it compares

The difference is the layer. This works at Spring AI's own abstractions; HTTP mocks work at the wire.

spring-ai-test-tools WireMock / MockWebServer Mockito
Level it works at Spring AI abstractions Raw HTTP Java objects
Getting a response Recorded from a real model Hand-authored provider JSON Hand-built ChatResponse
Provider coupling None Total None, but rebuilt by hand
Switching providers Same fixture replays Rewrite every stub Rewrite every mock
Streaming Chunk-for-chunk Hand-craft SSE frames Hand-build a Flux
Tool calling Replayed + isolated Model the loop yourself Hand-build tool calls
Structured output Schema in the cache key Invisible at HTTP level Schema never exercised
Catches real integration bugs Yes Partly — right bytes, wrong layer No
Setup One property Server, ports, matchers Per-scenario builder code

Where the alternatives are genuinely better

WireMock and MockWebServer are the right tool when the HTTP layer itself is what you're testing — retry and backoff policy, timeouts, connection pooling, a proxy, a 429 with Retry-After, or a body that breaks mid-stream. This library deliberately sits above that layer and cannot see any of it. Mockito remains right for everything that isn't a model call, and for a pure unit test wanting zero I/O — which is exactly why Stubbing exists rather than pretending record/replay covers it. And nothing here replaces a real integration test before you ship; it replaces running one on every commit.

How it's put together

One dependency, one property — and four independent fixture families that evolve separately.

io.github.rifatcakir.springai.testtools
├── recorder/
│   ├── advisor/       DeterministicVcrAdvisor — CallAdvisor + StreamAdvisor
│   ├── key/           VcrCacheKeyGenerator — hand-assembled SHA-256 canonical form
│   ├── track/         VcrTrack — the .call() fixture format
│   ├── stream/        VcrStreamTrack — raw chunk sequence
│   ├── embedding/     VcrEmbeddingModel — no advisor chain exists here
│   ├── tool/          VcrToolCallingManager — tool-call isolation
│   ├── junit/         @Vcr, @VcrTool — per-test escape hatches
│   └── autoconfigure/ Spring Boot wiring, off unless enabled
├── assertions/        VcrAssertions — fluent, deterministic checks
└── stub/              VcrStubs — hand-authored ChatModel/EmbeddingModel
0.8ms
Median replayed call
was 2.9 – 4.1 s live
3,500×
Faster than a warm
real model call
$0
Tokens per run
on every replay, forever
0
Network requests
asserted by a counter

Measured, not estimated: 200 timed iterations of a real chatClient…call().content() against a committed fixture (median 0.819 ms), and OllamaToolIsolationEndToEndTests against Testcontainers-managed llama3.2:1b — cold turn 46.65 s, warm turn 0.40 s, both turns replaying in ~30 ms. The hero terminal's other scenarios are measured the same way, against this project's own committed fixtures under REPLAY_ONLY: embedding 1.02 ms, evaluator 1.34 ms, streaming 0.80 ms median (100 iterations each). CPU inference on Windows 11; hosted providers were not measured. Full method & compatibility →
* These figures were measured locally (real-model timings via Testcontainers/Ollama on CPU; replay timings are local fixture reads). Results vary by hardware.

Make your Spring AI tests boring

Fast, free, offline and identical every single run.

Get started Worked examples
Not an official Spring project. This is an independent, community-maintained project. It is not affiliated with, endorsed by, or an official project of Broadcom, VMware, Spring, or Spring AI. “Spring” and “Spring AI” are trademarks of their respective owners; this library simply integrates with their public APIs.