Why an agent takes longer
to start replying

In a long conversation, an AI agent receives the system instruction, earlier messages and tool results again. The context grows after every turn. If the infrastructure processes the entire history again, users wait longer for the next response to begin and compute resources repeat work that has already been done.

On 19 October 2026, AWS described a way to reduce this repeated computation in vLLM deployments with separate prefill and decode workers. The vLLM documentation describes the mechanism and its settings. It is useful for understanding the causes of agent latency, even when the architecture does not fit a particular workload.

Where work is repeated

With disaggregated serving, one worker performs prefill: it processes the input context and builds the KV cache. A second worker receives that cache and generates the response. Separating these tasks lets operators allocate resources for long inputs and response generation independently.

In a one-way arrangement, KV state moves from prefill to decode. After generation, decode holds state that includes the new model response. On the next turn, prefill may not receive that state back and may process familiar history again. A complete cache miss means recomputing the system instruction, requests, tool results and model responses.

For the user, this appears as increased time to first token, or TTFT: the delay between sending a request and receiving its first output token. Total response time is a separate measurement.

What bidirectional KV transfer changes

In the documented implementation, decode temporarily retains KV blocks after completing a response. A proxy associates them with conversation_id and passes their metadata to prefill on the next turn. Prefill fetches the saved state and computes only the new tokens.

The first turn follows the normal path because no earlier cache exists. After the response, the proxy stores the metadata. Subsequent turns follow this sequence:

1. The client continues with the same conversation_id.
2. The proxy finds the previous turn’s state.
3. Prefill fetches the retained KV blocks from decode.
4. Prefill processes the new part of the context.
5. Decode generates a response and updates the state for the following turn.

Transferring a cache has a cost

Remote transfer is not always faster than local recomputation. For a short prefix, network and coordination costs can exceed the work saved. vLLM therefore uses kv_recompute_threshold: below the threshold, prefill recomputes locally; at or above it, prefill fetches remote KV blocks. In the cited revision, the default is 64 tokens, so remote retrieval applies at 64 tokens or more.

The decoder_kv_blocks_ttl parameter determines how long decode retains blocks. Its documented default is 480 seconds. If the next turn arrives later, the blocks have been released and the system returns to normal prefill. Increasing the TTL may enable more reuse, but also occupies accelerator memory for longer.

These defaults are not universal deployment settings. Choose the threshold and TTL using conversation lengths, pauses between turns, interconnect speed, model size and competition for memory.

What the published benchmark showed

AWS tested one prefill worker and one decode worker on EC2 P5en instances with EFA. Prefix caching on prefill was deliberately disabled to simulate local cache eviction or routing to another prefill worker. For Qwen3-32B with a 24,000-token context, AWS reported a TTFT reduction of up to 1.6×. For GLM-5-FP8 with a 20,000-token context, it reported up to 3.3×.

These are third-party results from a particular configuration and test method. They are not Novera results or a prediction for an arbitrary enterprise agent.

The useful observation is that repeated prefill becomes more expensive as conversation history grows. Reusing remote state helps when fetching the cache costs less than recomputing a long prefix.

When to test this architecture

Consider a bounded experiment when an agent runs multi-turn sessions, regularly receives long tool outputs and uses separate prefill and decode workers. Another signal is TTFT growing with conversation length while profiling shows repeated prefill of context that was already processed.

For short requests, infrequent follow-up turns or simple infrastructure, bidirectional transfer may merely add state and network coordination. First measure:

• TTFT by context length;
• the share of tokens processed more than once;
• the pause between turns in a session;
• KV transfer cost compared with local prefill.

Compare both configurations on the same conversations. Alongside average latency, measure P95/P99, memory use, cache hit rate and behaviour after the TTL expires. A separate test should verify fallback to normal prefill when state is lost or stale.

Choose an architectural optimisation after locating the source of latency. KV transfer will not fix a slow tool, a queue or an external API.

If TTFT grows across an agent’s turns, start with one real workflow: context length, repeated prefill, tool latency and the pause between turns. Read about AI development or share the workflow and its constraints for a focused discussion.