Design it step by step
Define the answer requirements
Invented query: “How do I export a project to PDF without internet?” Evidence must cover offline operation, format, and steps. A PDF overview is not an offline-export answer.
Retrieve a bounded shortlist first
Use keywords, embeddings, or both, with product and version filters. An illustrative ten-passage cap needs recall validation. Reranking cannot recover a document absent from its input.
Use one rubric for every pair
Example Score levels: 0=irrelevant, 1=format only, 2=export without offline coverage, 3=offline PDF steps. Save IDs and raw judgments. Ranking priority does not guarantee completeness.
Locate lines after selecting passages
Give source lines stable IDs. Choice can select a line or none; fetch neighboring context as needed. Copy original text by ID rather than rewriting commands or button labels.
Compare on fixed labeled queries
Measure needed-evidence retrieval, first useful result, and correct no-answer behavior. Moving invented candidate C from third to first improves that example only. Test missing answers, version mismatches, and cross-language queries.
Worked example · Invented by this site
| Observation | Judgment | Application action |
|---|---|---|
| A: PDF overview | Related topic, no steps | Do not use as an offline answer |
| B: cloud export | Requires internet | Preserve the conflicting condition |
| C: offline PDF steps | Covers the requirements | Fetch evidence lines and limits |
A copyable design draft
Original examples. JSON illustrates request or input structure; Python calculates invented scores without an API call. Verify current interfaces and task policy before integrating.
# Offline ranking exercise: invented scores.
ratings = {"A": 1, "B": 2, "C": 3}
ranked = sorted(ratings, key=lambda item: (-ratings[item], item))
print(ranked) # ['C', 'B', 'A']
# Ranking changes order; it does not verify answer support.Common mistakes
- Treating reranking as corpus retrieval.
- Changing the rubric between candidates.
- Assuming a similar line contains the answer.
TRY / THINK / COMPARE
Think first, then compare
The correct page is missing from the top ten. Tune only the reranker threshold?
Show explanation
No. Inspect retrieval and filters first; reranking cannot restore missing input.
Before handing it over
- Queries, candidates, and labels are fixed.
- All candidates share a rubric.
- Line IDs resolve to original versions.
- Missing answers and conflicts have fallbacks.
Common questions
Does line search replace document context?
No. Read the surrounding scope and conditions.
Can the top-ranked passage answer automatically?
Check coverage, conflict, and missing facts separately from rank.
Sources and further reading
Inspired by official patterns and Datawhale practice topics. Explanations, examples, and exercises are independently written. These are teaching designs, not live API runs or benchmarks.