The Context Deck

Answer: the eval set. Without one there is no way to tell whether any change is an improvement or a regression.

Answer: stable content first, per-request input last; then check cached-token telemetry.

Answer: vectors miss exact matches; run hybrid vector plus BM25 fused with RRF, then cross-encoder rerank.

Answer: the middle. Put critical instructions and the most relevant chunks at the edges; a 200k window does not eliminate reranking.

Answer: native structured-output APIs with constrained decoding, never please return JSON; validate and repair on failure.

Answer: a prompt. Name tools by intent, treat parameter descriptions as a contract, keep the toolset small.

Answer: iterations, total tokens, wall-clock, and cost, plus structured tool errors so the model can self-correct.

Answer: a model-generated are you sure is suppressible by prompt injection; keep irreversible-action privileges out of injected-content paths and put destructive confirmations in trusted UI.

Answer: metadata pre-filtering on tenant and ACLs before semantic search, so the vector index is search-within-results.

Answer: when calibrated against a human-labeled gold set with a held-out test split; pairwise beats pointwise.

Answer: floating-point non-associativity in batched matmul can produce occasional differences; design for statistical stability, not equality.

Answer: evaluate thresholds on real traffic and cache only stable factoid-style answers, never context-, time-, or user-state-dependent responses.