Skip to content
Home › Prompt Engineering & Structured Output

Routing for cost and SLA

Prompt Engineering & Structured Output

Definition

Choosing between real-time and batch processing based on the SLA of the workload — never defaulting to real-time when the need is asynchronous.

Key points

  • Rule: Never default to real-time for asynchronous needs.
  • Decision tree: Incoming Documents → SLA Constraint →
    • Urgent Exceptions (< 30m SLA) → Real-time Messages API
    • Standard Monthly Reports → Batch API Queue
  • Table:
    • Urgent Exceptions → Real-time Messages API (High Cost, Instant Latency)
    • Standard Workflows → Message Batches API (50% Cost Savings)
    • Continuous Arrival (30h SLA) → submit batches every 6 hours containing documents from that window.
  • On the Architect's Reference Matrix this is the Latency × Data Extraction cell ("Batch Routing").

Why it matters for the exam

  • Cost/SLA tradeoff is a core Domain-1/4 scenario: pick batch for async standard workloads to capture the 50% savings; reserve real-time for the sub-30-minute SLA.

Common gotchas

  • Defaulting everything to real-time is the anti-pattern — it burns cost for no latency benefit.
  • Continuous arrival with a 30h SLA → 6-hour batch windows, not one giant end-of-period batch.

See also

Sources

Referenced by