Just-in-Time Retrieval

Definition

A context strategy where agents "maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools" — loading context on demand rather than pre-loading everything up front.

Key points

• Just-in-time (verbatim): agents "maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools."
• Contrast — pre-loading / retrieval up front: load all relevant data into context before inference (traditional embeddings-based retrieval).
• Human analogy: "we generally don't memorize entire corpuses of information, but rather introduce external organization and indexing systems like file systems, inboxes, and bookmarks to retrieve relevant information on demand."
• Metadata itself is signal: folder structure, naming, and timestamps guide the agent's behavior.
• Trade-off (verbatim): "runtime exploration is slower than retrieving pre-computed data"; requires careful tool/heuristic engineering so the agent doesn't misuse context (e.g. wasteful searches).
• Hybrid strategies (some data up front + autonomous just-in-time exploration) can be used.

Why it matters for the exam

• Tests the design choice between pre-loading (RAG/embeddings) and runtime, reference-based loading. Expect scenarios with large corpuses or file systems where just-in-time keeps the attention budget small.

Common gotchas

• Just-in-time is not free: runtime exploration is slower than pre-computed retrieval and can waste context on bad searches — good tool design and heuristics are prerequisites.
• It's not all-or-nothing — hybrid (pre-load key data + explore the rest on demand) is often the right answer.

See also

• Context Engineering
• Structured Note-Taking
• Context Rot
• Server-side Compaction

Sources

• blog-context-engineering