How prompt-cache sections flow through LLM

PR #24063: message templates, code-defined prompts, OpenAI rendering, and cache accounting

Background

PR #24063 makes Jacob's message-style prompt templates executable and uses their structure to place explicit OpenAI prompt-cache boundaries. The core change is that a breakpoint is metadata attached to an ordered prompt section. It is not a marker hidden inside prompt text.

The pieces involved

What existed before

Top-level prompt templates supported field, code_key, and code_auto body sources. All three eventually became one string. Jacob's branch added ordered messages with roles and origins, then temporarily rejected new message templates until LLM could consume them. The older caching PR used a separate binary CacheableInput made of tagged segments. It proved the provider path, but no production caller built that input.

The new design reuses the message model as the seam. Prompt authors choose where the reusable prefix ends, and LLM carries that choice without parsing text.

Code-defined prompt behavior

Intuition

Think of cache_boundary as punctuation outside the text: “the reusable prefix ends after this section.” The text can come from an admin field or a compiled code-defined prompt. Its origin does not change the cache contract.

A small example

messages = [
  %{role: :system,
    origin: :code_key,
    code_prompt_key: "aia_chat/default",
    cache_boundary: true},
  %{role: :user,
    origin: :field,
    content: "Task: {{task}}",
    cache_boundary: false}
]

Variable replacement changes section text but leaves the boundary flag alone. Agent conversation messages are appended after the template messages, so the reusable prefix remains at the front and request-specific turns remain in the suffix.

Toy input and wire output

Section{
  text: "stable instructions",
  cache_boundary: true
}
Section{
  text: "dynamic task",
  cache_boundary: false
}

%{
  type: "input_text",
  text: "stable instructions",
  prompt_cache_breakpoint: %{mode: "explicit"}
}
%{type: "input_text", text: "dynamic task"}

Why code-defined prompts do not embed markers

Keeping cache metadata outside prompt files avoids provider syntax in business-owned text. It also avoids parsing sentinels and makes the fallback exact. The trade-off is granularity: one code-defined message is one section today. A boundary can follow the whole resolved prompt, not an arbitrary line inside it.

Code walkthrough

1. Store and validate message metadata

api/lib/jump/templates/template_message.ex:12-66 defines message roles, origins, source fields, and cache_boundary. It validates code keys and rejects boundaries after assistant messages.

field :origin, Ecto.Enum, values: [:field, :code_key, :code_auto]
field :cache_boundary, :boolean, default: false

if role == :assistant and cache_boundary do
  add_error(changeset, :cache_boundary, ...)
end

The admin switch in api/lib/jump_web/live/admin/template_live.ex:958-967 labels this choice “Cache prefix ends here.” The UI does not ask authors to edit prompt text.

2. Resolve code-defined text before compilation

api/lib/jump/prompts/prompt.ex:274-296 walks message templates. Field-origin messages keep their stored content. Code-key and code-auto messages call CodeDefinedPrompts.resolve_body/2, then put the resolved text back into the message.

defp resolve_template_message(%TemplateMessage{origin: :field} = message, _user),
  do: message

defp resolve_template_message(%TemplateMessage{} = message, user) do
  source = %{
    body_source: message.origin,
    code_prompt_key: message.code_prompt_key,
    purpose: message.purpose
  }

  %{message | content: CodeDefinedPrompts.resolve_body(source, user)}
end

3. Compile each message into structured input

api/lib/jump/prompts/prompt.ex:247-269 returns LLM.Input for message templates and text for every other template source. Each current template message becomes one Input.Message with one Input.Section.

defp template_message_to_input(%TemplateMessage{} = message) do
  section = Section.new!(message.content,
    cache_boundary: message.cache_boundary
  )

  InputMessage.new!(message.role, [section])
end

This is the exact answer to Jacob's question: code-defined text is not scanned for breakpoints. The enclosing message contributes the boundary metadata.

4. Format variables without losing structure

api/lib/jump/agent/llm/prompt_provider.ex:162-169 maps variable replacement across section text. LLM.Input.map_text/2 changes only Section.text, so role and boundary metadata survive.

case Prompt.get_prompt_input(prompt) do
  %Input{} = input ->
    Input.map_text(input, &Jump.Prompts.replace_variables(&1, variable_sets))

  text ->
    Jump.Prompts.replace_variables(text, variable_sets)
end

5. Append conversation and opt into caching

api/lib/jump/agent/llm/llm_execution.ex:297-305 preserves the old path for strings. Structured input appends the session conversation and adds prompt_cache: :optimize only when a boundary exists.

defp build_llm_input(%Input{} = input, llm_messages) do
  input = Input.append_messages(input, llm_messages)
  prompt_opts =
    if Input.cache_boundaries?(Input.semantic_messages(input)),
      do: [prompt_cache: :optimize],
      else: []

  {input, prompt_opts}
end

6. Keep policy provider-specific

api/lib/jump/integrations/llm/prompt_cache_decision.ex:83-121 detects structured boundaries. api/lib/jump/integrations/openai_v2/prompt_cache_breakpoints.ex:20-49 allows explicit rendering only for direct OpenAIV2 requests on the verified GPT-5.6 family.

7. Render the provider wire format at the last moment

api/lib/jump/integrations/llm/input.ex:101-129 owns plain and OpenAI section rendering. api/lib/jump/integrations/llm/execution.ex:210-241 chooses the render path after the provider decision. The end-to-end assertion in api/lib/jump/integrations/llm/prompt_cache_execution_test.exs:20-81 checks the exact HTTP body.

8. Account for cache writes

api/lib/jump/integrations/llm/usage_normalizer.ex:26-54 separates uncached input, cached reads, and GPT-5.6 cache writes. The follow-up commit also teaches api/lib/jump/open_inference/formatter.ex:645-661 to export nested cache_write_tokens.

uncached = input_tokens - cached_tokens - cache_write_tokens

%Usage{
  input_tokens: uncached,
  cached_input_tokens: cached_tokens,
  cache_creation_tokens: cache_write_tokens
}

Behavior summary

Validation and remaining risk

The changed-area suite passed 365 tests before the telemetry follow-up; 112 focused telemetry and caching tests passed afterward. Formatter diagnostics and diff checks are clean. The current PR uses mocked HTTP for its OpenAI wire assertion, so a live GPT-5.6 cache write/read remains the main unverified integration point.

Quiz