1. Candidate retrieval / 候选召回
BM25, DPR, SPLADE and TILDE-family models search a large collection and return a manageable candidate set.
Complete review map
From term matching to evidence-grounded generation
This guide follows one connected Information Retrieval (IR) pipeline: represent and index documents, retrieve candidates, rerank them, evaluate the ranking, learn from interaction data, and provide evidence to a generator. 中文主线：表示文本 → 召回候选 → 神经重排序 → 离线/在线评价 → 检索增强生成 → 证据归因。
文本 → 词项
词项 → 索引
索引 → 排名
排名 → 评价
稀疏 → 稠密语义
Learned sparse → 混合词汇与语义
Qrels → Clicks / 在线反馈
Candidates → BERT / LLM reranking
Evidence → RAG answer
Claims → Attribution
9Weeks connected / 周次
30+Key formulas / 公式
35+Retrieval, ranking & RAG methods
7 + ProjectsPracticals 与作业
Mental model
One system, three connected layers
把每周知识放回同一个 pipeline：retrieval 决定“看哪些文档”，ranking 决定“先看哪些”，evaluation/feedback 决定“如何判断并继续学习”。
Corpus
原始文档集合
原始文档集合
→
Analyzer
分词、停用词、词干
分词、停用词、词干
→
Inverted index
term → postings
term → postings
→
Ranker
BM25 / dense model
BM25 / dense model
Query
用户信息需求
用户信息需求
→
Top-k run
每个 query 的排名
每个 query 的排名
→
Qrels
人工相关性判断
人工相关性判断
→
Evaluation
MAP / nDCG / test
MAP / nDCG / test
四个经常混淆的对象：
corpus 是所有文档；query 是用户输入；run 是系统产生的排名；qrels 是标准相关性答案。评价是在 run 与 qrels 之间进行。→
2. Ranking / 重排序
A cross-encoder or learned ranker uses richer features to reorder candidates. Pointwise, pairwise and listwise describe different learning targets.
→
3. Evidence / 评价证据
Offline qrels support nDCG/MAP. Online clicks require bias correction, interleaving or counterfactual estimators.
Data relationship: corpus supplies documents; a query triggers retrieval; a run stores the produced order; qrels provide explicit labels; click logs provide abundant but biased implicit feedback. 前四者支撑离线实验，点击日志把课程带到 Week 7 的在线学习。
Week 1
索引、文本统计与预处理
目标：理解搜索引擎如何把大量文本变成可以快速查找的数据结构。
IR、数据库与 relevance
数据库查询通常要求精确匹配并返回确定答案；IR 面对自然语言和不确定的信息需求，因此返回按相关性排序的结果。Relevance 不是单纯“出现了相同词”，而是文档是否满足该用户在该情境下的信息需求。
主线：representation（怎样表示文本）+ matching（怎样比较）+ ranking（怎样排序）+ evaluation（怎样证明更好）。
Corpus、document 与 term
Corpus / collection 是全部文档；document 是一条独立记录；token 是分词后的一次出现；term 是索引使用的规范词项。一个 term 可以在同一文档中出现多次。
tf某词在一个文档中的次数 df包含该词的文档数 cf该词在整个集合中的总次数
Zipf's law
少数词极其频繁，大多数词很少出现。按频率排序后，rank 越高，frequency 越低，近似呈幂律。
frequency(r) ≈ C / rs
图上会出现长尾：最高频词通常是功能词；最低频词可能是专名、拼写变体或噪声。这解释了 stopwords 与词表压缩的动机。
文本处理
- Tokenization：把文本切成可索引单元。
- Case folding：通常统一为小写。
- Stopping：移除高频、区分度低的词，但可能损失短语含义。
- Stemming：用规则截断为词干，例如 Porter stemmer；不保证是真实单词。
- Lemmatization：利用词法知识还原词元，通常更精确但更昂贵。
Inverted index
正向存储是 document → terms；倒排索引反过来保存 term → postings list。查询词出现在哪些文档，可直接读取对应 postings，不需要逐篇扫描 corpus。
| 层级 | posting 保存什么 | 支持的能力 |
|---|---|---|
| Boolean | docID | 判断是否出现，做 AND / OR / NOT |
| Counts | docID + tf | TF、TF-IDF、BM25 排名 |
| Positions | docID + tf + positions | 短语查询、邻近查询、高亮 |
| Doc vectors | 每篇文档的 term 与统计 | 课程中自定义 scorer 读取 tf/df；A1 要求 -storeDocvectors |
易错：
df 不是某词出现总次数。某个词在一篇文档中出现 20 次，这一篇对 df 的贡献仍然只是 1，但对 cf 的贡献是 20。Week 1 practical 在训练什么
读取 JSONL corpus、统计文档数、检查文档结构、观察词频分布与 Zipf 长尾、建立带不同 stemming/stopword 配置的 Lucene index，并通过
index_reader.stats() 验证索引。Week 2
评价：怎样证明一个系统更好
离线评价需要 queries、qrels 和 runs。不同指标体现不同用户行为，不能只记公式。
Precision 与 Recall
Precision = |Relevant ∩ Retrieved| / |Retrieved|
Recall = |Relevant ∩ Retrieved| / |Relevant|
Precision 关心“返回的结果有多干净”；Recall 关心“所有相关文档找回了多少”。搜索引擎首页更偏 precision，法律检索或医学系统常更重 recall。
F-measure
Fβ = (1 + β²)PR / (β²P + R)
F1 = 2PR / (P + R)
F1 是 precision 与 recall 的调和平均；β>1 更重 recall，β<1 更重 precision。
AP 与 MAP
AP = (1 / |Rel|) Σk=1..K P@k × rel(k)
MAP = (1 / |Q|) Σq∈Q AP(q)
只有在第 k 位相关时，P@k 才会进入 AP。没有被检索到的相关文档仍留在分母 |Rel| 中，因此会降低 AP。
RR 与 MRR
RR(q) = 1 / rank(first relevant result)
MRR = meanq RR(q)
只关心第一个相关答案在哪里，适合问答或导航型查询；不会奖励第二、第三个相关结果。
DCG 与 nDCG
DCG@K = Σk=1..K (2rel(k) − 1) / log2(k + 1)
nDCG@K = DCG@K / IDCG@K
高相关等级获得指数增益，排名越靠后获得折扣。IDCG 是把同一 query 的结果按理想相关性排序后的 DCG，因此 nDCG 通常在 0–1。
RBP
RBP = (1 − p) Σi=1..d ri pi−1
p 是用户继续查看下一条的概率，期望查看深度是 1/(1-p)。p 越大，越重视更深的排名。
Paired t-test
同一批 queries 分别由系统 A 和 B 得到分数，先计算每个 query 的差值，再检验差值均值是否显著偏离 0。
dq = metricB,q − metricA,q
H₀: mean(d) = 0
H₀: mean(d) = 0
p < .05 常称 statistically significant；p < .01 可称 strongly significant。
怎样解释 p-value
p-value 是：假设 H₀ 成立时，观察到当前或更极端差异的概率。它不是“H₀ 为真的概率”，也不代表效果大小。
先确认键存在：
KeyError: ndcg_cut_5 通常意味着 evaluator 初始化时没有请求这个 measure，而不是 t-test 本身出错。Week 2 practical 的代码链
search(..., k=1000) 生成 run → pytrec_eval.parse_run 读取 → evaluator 对每个 query 计算指标 → aggregate 得到总体均值 → scipy.stats.ttest_rel 比较两个系统的逐 query 分数。这里的 k 是每个 query 保留的 top-k 文档数，不是 BM25 的 k1。Week 3
词项匹配与经典排序模型
从“是否包含查询词”逐步增加词频、稀有性、长度归一化和概率解释。
| 模型 | 核心想法 | 公式 / 作用 | 主要限制 |
|---|---|---|---|
| Boolean | 满足逻辑条件即匹配 | AND / OR / NOT postings | 通常不产生细致排名 |
| Coordination | 匹配更多不同 query terms 更好 | matched query terms 数 | 忽略同一词出现次数和稀有性 |
| TF | query term 在文档中出现更多更好 | Σ f(qi,D) | 常见词和长文档占便宜 |
| IDF | 少见词区分度更高 | log(N/(1+dfi)) | 不看当前文档中的 tf |
| TF-IDF | 局部频率 × 全局稀有性 | Σ tfi,D × idfi | tf 线性增长、长度处理粗糙 |
| VSM | 把 query/doc 当向量比较方向 | cosine(q,d) | 仍依赖词项重合 |
| BIM | 估计词在 relevant/non-relevant 中的区别 | 概率相关性权重 | 词项独立假设；需估计概率 |
| BM25 | IDF + tf saturation + length normalization | 经典强稀疏基线 | 无法天然解决同义词和语义匹配 |
课程版 TF、IDF、TF-IDF
TF(D,Q) = Σi=1..|Q| f(qi,D)
IDF(qi) = log(N / (1 + dfi))
TF-IDF(D,Q) = Σ tfi,D × idf(qi)
1 + df 是平滑，避免 df=0 时除零。IDF 只依赖 term 和整个 collection，不依赖当前 document。
Vector Space Model
cos(q,d) = (q · d) / (||q|| ||d||)
点积受向量长度影响，cosine 先归一化，比较“方向”。若向量已 L2-normalized，则 dot product 与 cosine 相同。
BM25：逐部分理解
score(D,Q) = Σi IDF(qi) × [(k₁+1)tfi] / [tfi + k₁(1 − b + b·|D|/avgdl)]
- IDF：区分 rare term 与 common term。
- TF saturation：tf 从 1 到 2 的增益大于从 100 到 101；避免重复堆词无限加分。
- Length normalization：长文档本来有更多机会包含任何词，因此按
|D|/avgdl调整。 - k1：控制 tf 多快饱和；越大越接近线性 tf。
- b：控制长度归一化；0 表示不考虑长度，1 表示完全按相对长度调整。
RSJ-IDF ≈ log((N − df + 0.5) / (df + 0.5))
讲义中的完整 BM25 还可包含 query term frequency 因子；A1 的模板采用课程指定的简化 IDF。作业时以 notebook/marking sheet 的公式为准，不要自行混用不同 BM25 版本。
Week 3 practical 的 scorer 输入为什么是三个参数
tf_vector 保存每个 query term 在当前 doc 的 tf；df_vector 保存这些 terms 各自的 df；doc_len 是当前文档长度。scorer 返回一个总分，search 函数再按分数排序并写出 run。三个 assert 在验证什么
assert score(tf=3) > score(tf=1) 检查更多词频是否提高得分；assert score(short_doc) > score(long_doc) 检查长度归一化；assert np.isfinite(...) 检查没有 NaN/∞。assert 条件为 False 时立即抛出 AssertionError，因此它是一种小型自动测试。Week 4
Transformer：从 token 到 contextual embedding
经典稀疏模型要求词项重合；Transformer 学习上下文相关表示，为语义匹配提供基础。
Tokenization 与 embedding
Tokenizer 把文本映射成 token IDs；nn.Embedding(V,D) 像一张可训练查找表，把每个 ID 映射为 D 维向量。D 是模型设计参数，例如 BERT-base 的 hidden size 是 768。
inputi = token_embedi + position_embedi + segment_embedi
形状常写成 N × D：N 是 tokens 数量，D 是每个 token 的表示维度。
Position 与 segment
纯 self-attention 不知道顺序，所以加入 position embedding。BERT 处理句对时，segment/token-type embedding 区分 A 段与 B 段；[SEP] 是边界 token。
WordPiece：未知或低频词可拆成子词，
## 表示该片段接续在前一个子词后面。Self-attention
Q = XWQ, K = XWK, V = XWV
Attention(Q,K,V) = softmax(QKT / √dk)V
- 每个 token 产生 query：我想找什么；key：我能提供什么；value：我的内容是什么。
QKᵀ产生 N×N 分数矩阵，表示每个 token 对所有 token 的关注程度。- 除以
√d_k防止维度较大时点积过大、softmax 过度饱和。 - softmax 把每行变成和为 1 的权重；再乘 V 得到融合上下文的表示。
Multi-head attention
多个 attention heads 在不同投影子空间中并行建模关系，例如一个 head 偏语法依赖，另一个偏实体关系。各 head 输出拼接后再投影回 D 维。
FFN、hidden 与 residual
hidden = GELU(W₁x + b₁)
out = W₂hidden + b₂
out = W₂hidden + b₂
FFN 对每个 token 独立地做非线性特征变换，通常 D→4D→D。这里 hidden 只是中间激活张量，不是手工设置的“参数”；真正训练的参数是 W、b。Residual 保留原信息，LayerNorm 稳定训练。
Encoder 与 BERT
Encoder 可以同时看到左右上下文，适合分类、理解、ranking 和 embedding。BERT 以 masked language modelling 为主要预训练目标。
LMLM = −Σi∈M log P(xi | x\M)
Decoder 与 GPT
Decoder 使用 causal mask，只能看当前 token 左侧，从而逐 token 生成。Encoder backbone 不能直接像 decoder 一样自然生成，需要额外 decoder 或改造目标与 attention mask。
P(y₁…yT) = Πt=1..T P(yt | y<t)
Week 4 practical 中 NoneType.shape 的原因
若
out = tf_block(x) 得到 None，说明 forward() 或其中调用的方法漏写了 return x/return out。Python 函数没有 return 时默认返回 None，因此 out.shape 会报错。Week 5
Dense Retrieval、训练目标与负样本
把 query 和 document 映射到向量空间，使“car repair”也能接近“automobile maintenance”。
Bi-encoder
score(q,d) = φ(ηq(q), ηd(d))
query 和 document 分开编码。文档向量可以离线预计算并建立 ANN index，在线只编码 query，因此适合第一阶段 retrieval。
Cross-encoder
把 [query; document] 一起输入 Transformer，允许所有 token 深度交互，通常更准确但无法预先编码文档，适合 reranking 少量候选。
Pooling
- CLS/first token：使用第一个特殊 token 的 contextual embedding。
- Mean pooling：对有效 tokens 向量求平均。
- EOS/last token：decoder 模型常用最后一个 token，因为它已看到之前全部上下文。
- All-token：保留每个 token 的向量，如 ColBERT。
Similarity
dot(q,d) = Σ qidi
cos(q,d) = q·d / (||q|| ||d||)
若向量已做 L2 normalize，dot 与 cosine 的排名一致。题目中的 “mean pooling similarity” 不严谨：mean pooling 是聚合表示的方法，不是 similarity function。
InfoNCE / contrastive loss
L = −log [ exp(sim(q,d⁺)/τ) / (exp(sim(q,d⁺)/τ) + Σj exp(sim(q,d⁻j)/τ)) ]
目标是提高正样本相似度，同时相对于一组负样本降低其相似度。温度 τ 越小，softmax 越尖锐。In-batch negatives 会把同一个 batch 中其他 query 的正文档当作当前 query 的负样本，提高效率。
Multi-label margin loss
L = mean(d⁺,d⁻) max(0, 1 − [s(q,d⁺) − s(q,d⁻)])
只要正样本得分至少比负样本高 margin=1，该 pair 的损失就是 0。
ColBERT late interaction
MaxSim(q,d) = Σi=1..m maxj=1..n Eq,i · Ed,j
对每个 query token 找最相似的 document token，再把这些最大值相加。它不是单 query vector 与单 document vector 的一次点积。
| 负样本类型 | 来源 | 优点 | 风险 |
|---|---|---|---|
| Random negatives | 集合中随机文档 | 便宜、稳定 | 常常太简单，梯度很弱 |
| In-batch negatives | 同 batch 其他正样本 | 几乎零额外编码成本 | 可能出现 false negatives |
| BM25 hard negatives | BM25 高排但不相关的文档 | 词汇相似、训练更有挑战 | 依赖 qrels 完整性 |
| ANN/self-mined | 当前 dense model 的近邻错误 | 紧跟模型当前弱点 | 计算和刷新索引昂贵 |
Smoke query / smoke test 是什么
Smoke test 是“先看系统能不能基本跑通”的快速检查，不负责证明质量。你用一个
smoke_query 分别运行 TF 与 BM25，只取 top-5，确认没有异常、返回结构正确、分数有限、两个模型可能给出不同排名。正式结论仍要用全部 test queries 和 qrels 评价。Week 6
Learned Sparse Retrieval / 学习型稀疏检索
Keep vocabulary-aligned sparse representations and inverted-index efficiency, but learn term weights and expansion with neural models. 核心不是把向量变密，而是让稀疏词项权重具有上下文与语义。
The missing middle: BM25 is sparse and fast but depends on lexical overlap; DPR is semantic but uses dense ANN search. Learned sparse retrievers keep dimensions tied to vocabulary terms while learning which terms to activate and how strongly to weight them. 它们位于 BM25 与 dense retrieval 之间，试图同时保留可解释的词项维度和神经模型的语义能力。
| Family | Query representation | Document representation | Matching and cost | Main idea / 中文抓手 |
|---|---|---|---|---|
| BM25 | observed query terms | term counts + corpus statistics | inverted index; very cheap online | Hand-crafted lexical weights / 人工公式 |
| DPR | one dense vector | one dense vector | dot product + ANN; query encoder online | Semantic neighbourhood / 向量语义近邻 |
| TILDEv2 | token IDs only; stopwords removed | contextual weights for document tokens, often after offline expansion | exact token-ID match; no query encoder | Learned exact matching / 学习词权重，在线查询很轻 |
| SPLADE | learned sparse vocabulary vector | learned sparse vocabulary vector | sparse dot product; encoder on both sides | Implicit expansion / 自动激活原文没有的相关词 |
| PromptReps | dense hidden state + sparse next-token logits | same two representations | hybrid normalized score; LLM inference | Zero-shot dense+sparse signals / 无需 retriever training |
Vocabulary mismatch and expansion
A relevant document may say “automobile” while the query says “car.” Document expansion adds likely query terms before indexing; doc2query uses a sequence-to-sequence model such as T5 to generate likely queries for each document.
Cost placement: expansion is performed offline, so the larger index buys better recall without adding a query-time generation step. 中文：把计算成本提前到建索引阶段。
TILDE → TILDEv2
TILDE models document-side term likelihoods over the vocabulary. TILDEv2 instead produces contextual weights for actual or expanded document tokens, then scores only exact matches with query token IDs.
score(q,d) = Σt∈q maxi: token(di)=t wi,d
查询词若在文档 token 中出现多次，practical 取该词的最大 contextual weight；未匹配词贡献 0。
SPLADE and SPLADEv2
An MLM head maps contextual token states into vocabulary-sized term weights. ReLU removes negative activations; pooling combines token evidence. The sparse query and document vectors are scored by a dot product.
score(q,d) = vq · vd, v ∈ R|V| and mostly zero
SPLADEv2 improves pooling/training, including max-pooling and distillation variants. Sparsity regularisation such as FLOPS/L1 controls how many vocabulary terms remain active.
Sparse does not mean non-neural
Sparse/dense describes the output representation, not whether BERT was used. SPLADE uses a Transformer but outputs a mostly-zero vocabulary vector; DPR also uses a Transformer but outputs a dense embedding.
常见误区：learned sparse 的“sparse”指多数维度为 0，并不表示它没有神经网络或不能表达语义扩展。
Week 6 practical: two-stage BM25 + TILDEv2
BM25 retrieve
Search the full corpus and keep top-k candidates. High recall and low cost matter here.
→
Read raw text
Use each hit's document ID and raw() content as the reranker's input.
→
TILDEv2 rerank
Score every query-document pair and sort candidates by the new score.
Why two stages? Running a richer model over the whole collection is expensive. BM25 narrows the search space; TILDEv2 spends extra computation only on plausible candidates. 第一阶段负责“不要漏太多”，第二阶段负责“把更相关的推到前面”。
How the supplied tildev2_scoring function moves data
Document text → tokenizer tensors →
model.encode → document token IDs and learned weights. Query text → token IDs → remove stopword IDs. For each query ID, locate equal document IDs, take the maximum weight if it appears more than once, and sum across query terms. The returned scalar is a reranking score, not a probability.How to inspect a SPLADE vector
Convert the sparse tensor to dense only for inspection, count non-zero dimensions, then use
torch.topk and the tokenizer vocabulary to decode the strongest terms. Original terms reveal lexical evidence; activated related terms reveal learned expansion. Do not densify the whole collection in a real retrieval index.Week 7
Learning to Rank, Click Bias and Online Evaluation
Weeks 1–6 mostly ask how to produce a ranking. Week 7 asks how to learn and evaluate when feedback comes from users rather than complete qrels. 中文重点：点击很多，但点击不等于相关。
Where LTR sits in the retrieval pipeline
Retriever
BM25 / dense / learned sparse returns candidates and retrieval scores.
→
Feature vector
BM25 score, neural score, freshness, authority, query-document features.
→
LTR ranker
Learns a function that orders candidates to maximise a ranking objective.
A retrieval model can itself be learned, but Learning to Rank (LTR) usually means learning how to combine signals and order an already manageable candidate set. retrieval 解决规模，LTR 解决候选内部的顺序。
Offline LTR targets
- Pointwise: predict one document's relevance score or class independently.
- Pairwise: learn that document A should rank above B.
- Listwise: optimise the quality/order of the whole result list.
三者区别是训练目标的单位：单文档、文档对、整张列表。
Qrels versus click logs
Qrels are explicit, relatively clean labels but expensive, static, and limited. Clicks are abundant and current but implicit, noisy, and biased by what was shown and where.
No click ≠ non-relevant: the user may never have examined the document.
Three click biases
- Position bias: high ranks are examined more often.
- Selection bias: unshown documents cannot be clicked.
- Presentation bias: snippets or visual style affect clicks.
随机噪声可能随数据增多而抵消；系统性 bias 不会自动消失。
Examination model
P(ci=1 | yi) = P(ci=1 | oi=1,yi) · P(oi=1 | ranki)
A click requires both examination o and attraction/relevance evidence y. The propensity P(o=1|rank) captures how likely a position is viewed.
Naive clicks versus IPS
Δnaive(f) = Σi λ(ranki) · ci
ΔIPS(f) = Σi [λ(ranki) / P(oi=1)] · ci
The naive estimator overvalues results placed where users look more often. Inverse Propensity Scoring (IPS) divides each click by its examination probability, giving rare observations more weight and cancelling position bias in expectation when assumptions are correct.
中文直觉：第 10 位本来很少被看到，所以在那里发生的一次点击携带的信息量比第 1 位点击更大。IPS 用
1/propensity 放大这类样本。代价是 propensity 很小时方差会变大。Balanced interleaving
Maintain a scan pointer for each ranked list. Inspect the list with the smaller pointer, breaking ties with a randomly chosen starting ranker. Advance the pointer even for duplicates, but append only unseen documents. This balances inspected depths, not necessarily the number contributed by each ranker.
Advantage: both systems face nearly the same query, user and context. 比两组独立 A/B 流量更敏感。
DBGD
Dueling Bandit Gradient Descent samples a candidate parameter direction, compares the current and candidate rankers through interleaving, and moves toward the winner. Repeated duels estimate a useful gradient from user satisfaction.
Counterfactual OLTR
The deployed stochastic ranker logs displayed actions and their probabilities. The same interaction log is then reweighted with IPS to estimate the risk of many candidate rankers. A candidate replaces the current model when its estimated counterfactual risk is lower.
Logging policy
Sample rankings and record propensities.
→
Interaction log
Displayed documents, ranks, clicks/non-click losses.
→
IPS comparison
Estimate current and candidate risks from the same log.
Main advantage over regular interleaving: one logged randomized policy can evaluate multiple candidate rankers off-policy, instead of constructing a fresh interleaved list for every pairwise comparison. 更能复用历史数据，但依赖可靠 propensity、充分 exploration/support，并可能有高方差。
| Method | Data source | What it estimates | Strength | Main limitation |
|---|---|---|---|---|
| Offline qrel evaluation | queries + qrels + runs | MAP/nDCG/etc. | controlled and repeatable | labels expensive and may become stale |
| Balanced interleaving | live clicks on one mixed list | pairwise ranker preference | sensitive within-user comparison | fairness/correctness is not guaranteed in every case |
| DBGD | repeated interleaving outcomes | improving parameter direction | learns directly online | exploration may expose poorer rankings |
| Counterfactual + IPS | logged clicks + propensities | off-policy candidate value/risk | reuses logs; compares many candidates | propensity errors, support gaps, high variance |
How examination propensities can be estimated
A randomized top-n experiment can shuffle positions, collect clicks by rank, and normalize the resulting examination frequencies. This is useful because IPS needs the probability that an item at a position was examined. The logging policy must retain enough exploration so candidate actions have support.
Why OLTR can create a self-confirming loop
A random click on an irrelevant result can make the model promote similar items. Higher positions then attract more examination and clicks, which the learner may misread as stronger relevance. Bias correction and controlled exploration are needed to keep exposure from becoming mistaken evidence.
Week 8
Transformer-based Reranking: BERT and LLM Rankers
Week 8 moves from retrieving candidates to applying deeper query-document interaction. Encoder rerankers classify relevance; decoder-based Large Language Models (LLMs) can judge, compare or order passages through prompts. 核心关系：第一阶段保证候选覆盖，第二阶段用更昂贵的模型改善候选内部顺序。
Where neural reranking sits
Stage 1: retrieve
BM25 or another efficient retriever searches the full collection and returns top-k candidates.
→
Stage 2: interact
A cross-encoder jointly reads the query and each candidate, or an LLM receives them in a ranking prompt.
→
Final ranking
Sort by neural relevance scores, pairwise preferences or an output ordering.
Effectiveness-efficiency trade-off: deeper interaction usually improves ranking quality but increases latency. Reducing candidate count lowers cost, but a relevant document omitted by Stage 1 cannot be recovered by the reranker. reranker 只能重新排列已召回文档，不能救回没有进入候选集的文档。
monoBERT: pointwise cross-encoder
Bidirectional Encoder Representations from Transformers (BERT) receives [CLS] query [SEP] document [SEP]. The contextual [CLS] vector enters a classification layer that estimates whether the document is relevant.
s(q,d) = P(relevant = 1 | q,d)
Each candidate is scored independently, so monoBERT is pointwise. Training uses human positive judgements and commonly samples difficult negatives from BM25.
monoBERT training objective
L = -Σj∈Jpos log(sj) - Σj∈Jneg log(1-sj)
This is binary cross-entropy: reward high scores for relevant passages and low scores for negative passages. The score is useful for ordering candidates, but it should not automatically be interpreted as a perfectly calibrated real-world probability.
训练单位是一组 query-document pair；模型学的是相关/不相关分类信号。
BERT length limit: two aggregation families
| Family | What is divided | How evidence is combined | Main idea / 中文抓手 |
|---|---|---|---|
| BERT-FirstP | Document → passages | Use the first passage score | Cheap, but later evidence is ignored / 只看首段 |
| BERT-MaxP | Document → passages | Take the maximum passage score | One strong passage can represent the document / 取最强证据 |
| BERT-SumP | Document → passages | Sum passage scores | Accumulates evidence but may favour many passages / 累加证据 |
| Birch | Document → sentences | Interpolate first-stage score and weighted sentence scores | Combines retrieval and sentence evidence / 句子级聚合 |
| PARADE | Document → passage representations | Average, max, attention or Transformer aggregation | Combine representations before the final relevance decision / 先聚合表示再评分 |
Do not confuse: score aggregation combines already-produced passage scores; Passage Representation Aggregation for Document Reranking (PARADE) combines passage vectors and can model relationships among passages.
duoBERT: pairwise reranking
After monoBERT narrows the list, duoBERT jointly reads the query and two documents and estimates p(dᵢ > dⱼ | q). It therefore learns a pairwise preference rather than an independent relevance score.
s(dᵢ) = Σj≠i p(dᵢ > dⱼ | q)
Aggregating pairwise wins produces the final order. More pair comparisons may improve quality, but encoder cost grows quickly.
Multi-stage tuning
BM25 → monoBERT → duoBERT exposes several tuning knobs: candidate depth at each stage, model size and number of pair comparisons. These provide a flexible quality-latency trade-off, but also make the system more complex to tune and operate.
候选越少越快，但召回风险越大；候选越多、比较越细，成本越高。
LLM reranking families
| Family | Prompt input | Ranking signal | Cost and limitation |
|---|---|---|---|
| Pointwise | Query + one passage | Yes/No logits, generated label or relevance grade | Simple and parallelisable; absolute judgements may be poorly calibrated |
| Pairwise | Query + passage A + passage B | Preference A > B | Robust comparison, but many pairs require many inferences |
| Listwise | Query + a list of passages | Generate a permutation or ordered identifiers | Sees list context; generation and context length are costly |
| Setwise | Query + a small set of passages | Select the best passage using output logits | Fewer comparisons than pairwise and avoids full list generation |
Zero-shot in this lecture: once an instruction-tuned LLM is available, these ranking prompts do not require new contrastive retriever training. This does not mean the underlying LLM was never pretrained or instruction-tuned.
Why setwise can be efficient
Pairwise ranking compares two documents per call. Setwise ranking selects a winner from several documents at once, reducing the number of LLM inferences. Compared with listwise generation, setwise methods can use next-token logits instead of generating a complete permutation.
Prompt variation matters
Role instructions, output restrictions, formatting instructions and query/passage order can all change effectiveness. The slides report larger variation for pointwise and listwise prompts, with pairwise and setwise prompts generally more robust; behaviour also depends on model size, architecture and training.
Exam caution: a prompt published in a paper is not automatically the best prompt for every model or dataset.
Week 8 practical: two implementation pipelines
BM25 candidates
Use Pyserini to retrieve a small top-k and read each hit's raw passage text.
→
monoBERT scores
Tokenize each query-passage pair, obtain the relevant-class softmax score and sort descending.
→
Evaluate run
Write the reranked document IDs and compare the run with qrels.
The practical recommends beginning with only 10 or 20 candidates because monoBERT inference is expensive. Its second exercise uses TinyLlama: first inspect pointwise Yes probabilities, then design a pairwise prompt that chooses between two passages.
Data flow / 数据流：query → BM25 top-k → raw passage text → tokenizer → monoBERT or LLM score/preference → re-sort the same candidates → evaluation. Retrieval depth and evaluation cutoff are separate choices.
LLM4IR versus IR4LLM
Large Language Models for Information Retrieval (LLM4IR) uses LLMs to improve retrieval tasks, such as reranking passages. Information Retrieval for Large Language Models (IR4LLM) uses retrieval to supply external evidence to an LLM, as in retrieval-augmented generation. Week 8 practical focuses on LLM4IR reranking.
Why automatic prompt engineering is harder for ranking
A ranking demonstration must encode a query, several passage meanings, their relative relevance and an ordering; multiple passages may legitimately share a relevance level. This output structure is less unique and more complex than a single classification label.
Week 9
Information Retrieval in the Age of LLMs / LLM 时代的信息检索
Week 9 connects two directions: LLM4IR uses a Large Language Model to retrieve or rank; IR4LLM retrieves external evidence for generation. The lecture then asks when long context can replace retrieval, and why relevant evidence still may not produce a correct, grounded answer.
One map for the whole course / 全课程总图
Represent & retrieve
BM25, DPR, TILDEv2, SPLADE, CiC or a specialist retriever find candidate evidence.
→
Refine evidence
Rerank, rewrite, filter, summarise, fuse or compress before the prompt.
→
Generate & attribute
A frozen or adapted LLM answers, while citations and groundedness checks connect claims back to evidence.
关键区别：retrieval quality asks whether useful passages are ranked highly; answer quality asks whether the generator uses them correctly. A better nDCG score can help RAG, but it does not logically guarantee a better answer.
Prompting recap
Zero-shot gives instructions only; one-shot/few-shot add demonstrations; Chain-of-Thought (CoT) elicits intermediate reasoning tokens. Reasoning text is still generated output, not a transparent record of the model's internal computation.
示例改变的是上下文中的任务说明与模式，不等于重新训练模型参数。
LLMs as retrievers or rankers
An LLM can generate a dense/sparse representation, judge one passage, compare two passages, order a list, or select from a set. RepLLaMA and LLM2Vec produce dense representations; PromptReps and DiffRetriever explore alternative generated representations; pointwise/pairwise/listwise/setwise prompts produce ranking signals.
Memory hook: representation models decide where to search; rerankers decide how to reorder a candidate set.
Long-Context LLM (LC-LLM) versus RAG
| Dimension | LC-LLM / Corpus-in-Context | Retrieval-Augmented Generation |
|---|---|---|
| Input | Put a very large corpus directly in context | Retrieve a small top-k evidence set |
| Main benefit | One consolidated model call; fewer pipeline boundaries | Scales to larger, changing corpora; lower prompt size |
| Main cost | Attention cost and long-prompt latency/memory | Retriever/reranker maintenance and cascading errors |
| Failure mode | Position bias and lost-in-the-middle | Missing, distracting or conflicting retrieved evidence |
| When attractive | Corpus fits context and repeated prefixes can be cached | Corpus is large, dynamic, or needs explicit evidence IDs |
Corpus-in-Context (CiC, pronounced “seek”) assigns consistent document identifiers, places the corpus and corpus-grounded demonstrations in one prompt, and asks the LC-LLM to return document IDs. Prefix caching can encode a repeated corpus prefix once, but it does not remove positional bias.
Corpus-in-Context prompt design
- Use new, unambiguous document identifiers and consistent formatting.
- Ground every demonstration in the same corpus and request a fixed ID-list output.
- Attach a reasoning example when the task requires multi-step evidence selection.
- For multi-turn search, include previous queries and outputs consistently.
few-shot 示例数量增加可能改善效果，但 gold document 越靠后，检索质量可能下降。
Specialised retrieval interfaces
Screenshot retrievers encode rendered pages with a Vision-Language Model (VLM), preserving layout, figures and spatial signals lost by plain text extraction. DiffRetriever uses a diffusion language model to produce several representative tokens in parallel, then matches multiple query/document representations.
Do not infer “newer = always better”: compare representation, index, scoring cost, training data and task fit.
Basic RAG data flow / 基础 RAG 数据流
Query
information need
information need
→
Retriever
top-k passage IDs
top-k passage IDs
→
Prompt
query + evidence
query + evidence
→
LLM
answer + citations
answer + citations
RAG supplies knowledge that is newer, private, domain-specific or absent from pretrained weights. It differs from further language-model pretraining: the external collection remains separately updateable and the retrieved evidence can be inspected.
Project 2 connection: BM25, DPR and TILDEv2 produce top-10 evidence; the same LLM, prompt, settings and token budget must then isolate the effect of evidence condition.
Modular RAG: intervene before and after retrieval
| Stage | Methods | What they change | Risk |
|---|---|---|---|
| Pre-retrieval | routing, rewriting, expansion, HyDE | make the query easier for the index to match | rewrite may drift from the original need |
| Retrieval | BM25, dense, learned sparse, hybrid | choose the candidate evidence set | relevant evidence may never enter top-k |
| Post-retrieval | reranking, filtering, fusion | improve order and remove distractors | extra latency; filter can remove useful evidence |
| Prompt construction | summarisation, lexical or embedding compression | fit more useful evidence in a fixed budget | compression can delete qualifiers or provenance |
| Generation | read, cite, revise, Self-RAG | use evidence and decide whether to retrieve again | model knowledge may override supplied evidence |
Rewrite-Retrieve-Read
A rewriter converts an ambiguous or underspecified question into a search-friendly query, retrieval supplies evidence, and a reader answers. The lecture's trainable variant warms up on pseudo rewrites, keeps useful rewrites, then optimises the rewriter with a reward while constraining divergence from the initial policy.
它优化的是“送给 retriever 的 query”，不是直接修改 documents。
BlendFilter
BlendFilter combines retrieved knowledge with model-generated query/answer signals, then prompts an LLM to filter irrelevant passages before answer generation. It illustrates that retrieval is not the end of the pipeline: evidence selection can itself be an LLM task.
Prompt compression
Lexical compression removes low-information tokens using signals such as entropy or perplexity. Embedding compression maps a long context into learned compact memory tokens. Both trade fidelity for fewer input tokens, lower latency and less distraction.
Evidence conflict
A model may preserve an initially wrong answer even after receiving supporting correction, or flip a correct answer after contradictory evidence. Therefore “retrieved passage is relevant” and “generated answer follows evidence” are separate measurements.
这正是 Project 2 同时要求 reference-based answer score 与 evidence groundedness 的原因。
Noise, position and amount of evidence
| Observation | Interpretation | Design response |
|---|---|---|
| One high-scoring distractor can sharply reduce answer quality | semantic similarity is not the same as answer-supporting evidence | rerank/filter for utility, not topic overlap alone |
| More distractors progressively degrade performance | longer context creates more competing signals | control top-k and evidence token budget |
| Gold evidence in the middle may be used least | long-context attention has positional bias | test order; place or repeat critical evidence carefully |
| Random passages sometimes improve some models | LLM behaviour depends on model and context distribution | treat it as an empirical finding, not a universal recipe |
Attribution
Attribution is the ability to provide references or citations that support generated claims. Directly generated citations can be fabricated; retrieval-based attribution constrains citation candidates to real documents.
RARR
Retrofit Attribution using Research and Revision (RARR) researches evidence for an existing answer and revises unsupported content while preserving the original response as much as possible. It is a post-generation attribution-and-repair pipeline.
Why does better retrieval not guarantee better generation?
nDCG measures where judged-relevant passages appear. Generation additionally depends on passage truncation, order, prompt wording, answer format, conflicting model knowledge and whether the LLM actually attends to the evidence. The relationship should therefore be measured per query rather than assumed.
RAG, long context and fine-tuning: when are they different?
RAG changes external evidence at inference time; long-context prompting supplies much more text without explicit retrieval; fine-tuning changes model parameters. They can be combined, but they move cost, updateability and failure modes to different parts of the system.
Formula sheet
公式速查表
先认清符号，再问每个因子在奖励或惩罚什么。
| 名称 | 公式 | 最重要的解释 |
|---|---|---|
| P@k | relevant in top k / k | 只看截断位置 k 内的精确率 |
| Recall | retrieved relevant / all relevant | 需要知道总相关文档数 |
| F1 | 2PR/(P+R) | precision 与 recall 的调和平均 |
| AP | (1/|Rel|) Σ P@k × rel(k) | 每遇到一个 relevant 就记录当时 precision |
| MAP | mean(AP over queries) | 所有 query 同权 |
| RR | 1/rank(first relevant) | 只在乎第一个 relevant |
| nDCG@K | DCG@K / IDCG@K | 支持 graded relevance 与位置折扣 |
| RBP | (1-p)Σ r_i p^(i-1) | p 显式建模用户继续浏览概率 |
| 名称 | 公式 | 直觉 |
|---|---|---|
| IDF | log(N/(1+df)) | 越少文档出现，区分度越高 |
| TF-IDF | Σ tf × idf | 文档内重要 × 集合内稀有 |
| Cosine | q·d/(||q||||d||) | 比较方向，降低长度影响 |
| BIM weight | log((N-df)/df) | 来自词在相关/非相关文档中的概率优势 |
| BM25 | Σ IDF × saturated_tf(length-normalized) | 稀有词、饱和 tf、文档长度三者组合 |
| 名称 | 公式 | 直觉 |
|---|---|---|
| Softmax | exp(z_i)/Σexp(z_j) | 把任意分数转换为和为 1 的权重 |
| Attention | softmax(QKᵀ/√d_k)V | 按相关性加权汇总 value |
| InfoNCE | -log exp(s+)/[exp(s+)+Σexp(s-)] | 在候选集合中把正样本分类出来 |
| Margin | max(0, m-(s+-s-)) | 正样本必须领先负样本至少 m |
| MaxSim | Σ_i max_j E_qi·E_dj | 每个 query token 寻找最匹配 doc token |
| Autoregressive | Π_t P(y_t|y_<t) | 根据已生成前缀预测下一个 token |
| monoBERT score | P(relevant=1 | q,d) | 联合编码 query 与单篇 document 后做 pointwise 分类 |
| Binary cross-entropy | -Σ_pos log(s) - Σ_neg log(1-s) | 提高 relevant pair 的分数，压低 negative pair 的分数 |
| duoBERT aggregation | s(d_i)=Σ_{j≠i} p(d_i>d_j|q) | 把 document 对其他 candidates 的 pairwise 胜率相加 |
| RAG answer | y ~ P(y | q, retrieve(q)) | 答案同时依赖 query 与检索证据；retrieval 是可观察的中间变量 |
| Pearson correlation | r = cov(x,y)/(σxσy) | 衡量 retrieval gain 与 answer gain 的线性关系；零方差时未定义 |
| Name | Formula / procedure | Meaning / 中文直觉 |
|---|---|---|
| Full-information utility | Δ(f,D,y)=Σ λ(rank(d))·y(d) | Rank discount λ weights relevance y; DCG and P@k are instances of this pattern. |
| Naive click estimator | Σ λ(rank_i)·c_i | Directly treats clicks as labels, so exposure bias remains. |
| IPS estimator | Σ [λ(rank_i)/P(o_i=1)]·c_i | Inverse examination probability corrects position bias in expectation / 用反倾向权重校正。 |
| Off-policy weight | P(action|new policy) / P(action|logging policy) | Reweights logged actions as if they came from a candidate policy. |
| Counterfactual update | choose f if Risk_IPS(f) < Risk_IPS(f₀) | Deploy the candidate with lower estimated counterfactual risk. |
Model matrix
模型对照：表示、打分、损失与负样本
遇到选择题时，把模型拆成这四列判断，比背一句描述可靠。
| 模型 | Representation / pooling | Similarity / interaction | Training | 定位 |
|---|---|---|---|---|
| BM25 | 稀疏 term statistics | 逐 term 加权求和 | 通常无需训练；调 k1,b | 强 lexical baseline |
| TILDEv2 | query token IDs；document contextual token weights | exact match + per-term max weight | document-side neural training/expansion | fast learned sparse reranker |
| SPLADE(v2) | mostly-zero vocabulary vectors | sparse dot product | MLM head + distillation + sparsity regularization | learned sparse first-stage retrieval |
| PromptReps | LLM dense hidden state + sparse logits | normalized hybrid interpolation | zero-shot; no retriever training | LLM-based hybrid representation |
| DPR | CLS，query/doc 双编码 | dot product | InfoNCE/NLL；in-batch + hard negatives | dense first-stage retrieval |
| RepBERT | mean token embeddings | inner product | MultiLabelMarginLoss | BERT bi-encoder |
| ANCE | CLS / first token | dot product | NLL/InfoNCE-style；ANN self-mined negatives | 动态 hard-negative mining |
| Contriever | mean pooling | cosine | unsupervised contrastive；same-doc spans | 无监督 dense retriever |
| ColBERT | 保留所有 token embeddings | late interaction MaxSim | contrastive / pairwise CE | 效率与细粒度匹配折中 |
| E5-Mistral | Mistral decoder；EOS/last token | temperature-scaled cosine | InfoNCE；synthetic + supervised data | LLM embedding model |
| LLM2Vec | decoder 改双向 attention；pooling | cosine 常见 | MNTP + unsupervised SimCSE + optional supervised | 把 decoder 转为通用 encoder |
| RepLLaMA | Llama2；EOS last hidden | dot product | InfoNCE；supervised + LoRA | LLM retriever |
| Cross-encoder | query 与 doc 联合编码 | 分类/回归头输出 relevance | pointwise/pairwise/listwise | 高精度 reranker |
| monoBERT | [CLS] q [SEP] d [SEP] joint encoding | relevant-class softmax score | binary cross-entropy；human positives + BM25 negatives | pointwise BERT reranker |
| BERT-MaxP / FirstP / SumP | document passages | max / first / sum of passage scores | passage labels transferred from document | long-document score aggregation |
| PARADE | passage representations | average / max / attention / Transformer aggregation | document-level reranking | long-document representation aggregation |
| duoBERT | query + two candidate documents | pairwise probability then win aggregation | pairwise cross-entropy | late-stage BERT reranker |
| LLM pointwise | query + one passage prompt | label/logit or relevance grade | zero-shot prompting in Week 8 | simple LLM reranker |
| LLM pairwise | query + two passages | direct preference | zero-shot prompting | robust but comparison-heavy |
| LLM listwise | query + candidate list | generated permutation | zero-shot prompting | list context, generation cost |
| LLM setwise | query + small candidate set | best-item logits | zero-shot prompting | fewer calls than pairwise |
| Corpus-in-Context (CiC) | entire corpus + demonstrations in an LC-LLM prompt | generate relevant document IDs | few-shot prompting; no retriever training | extreme model-based retrieval; corpus must fit context |
| Screenshot retriever | page pixels/layout encoded by VLM | dense similarity / learned retrieval score | vision-language contrastive training | layout-rich and multimodal documents |
| DiffRetriever | multiple parallel representative tokens | multi-representation similarity | diffusion language-model representation learning | efficient multi-vector generative retriever |
| Basic RAG | query + top-k external passages | autoregressive answer likelihood | retriever and generator may be frozen or adapted | IR4LLM; evidence-grounded answering |
| Rewrite-Retrieve-Read | rewritten search query + retrieved passages | rewrite reward + downstream answer utility | pseudo-data warm-up and optional policy optimisation | repair ambiguous/underspecified queries |
| BlendFilter | retrieved evidence + model knowledge | LLM filtering then answer generation | prompted augmentation/filtering | remove distracting retrieved knowledge |
| RARR | draft answer + researched evidence | support check and revision | post-hoc research/revision | retrofit attribution to generated text |
Fast classification rule / 快速分类：先问输出是 lexical sparse、learned sparse、single dense vector 还是 token vectors；再问 matching 是 inverted-index lookup、dot/cosine、MaxSim 还是 joint cross-encoding；最后判断它用于 first-stage retrieval、reranking 还是 feedback learning。
Final-exam model family atlas / 模型家族记忆表
| Family | Searches full corpus? | Offline cost | Online cost | Best at | Characteristic failure |
|---|---|---|---|---|---|
| Lexical sparse: BM25 | Yes, inverted index | low indexing cost | very low | exact names, rare terms, transparent scores | vocabulary mismatch / 词不相同就难匹配 |
| Dense bi-encoder: DPR, ANCE, Contriever, E5, RepLLaMA | Yes, ANN vector index | encode every document | encode query + ANN | semantic paraphrases | false semantic neighbours; large dense index |
| Learned sparse: TILDEv2, SPLADE | Yes when indexed as sparse postings | neural document encoding | low to medium | lexical efficiency plus learned weighting/expansion | expansion noise; model/index coupling |
| Late interaction: ColBERT | Yes with specialised index | store token vectors | medium | fine-grained token matching | larger index and MaxSim computation |
| Cross-encoder: monoBERT, duoBERT | No, candidate reranker | model training | high per pair | deep query-document interaction | cannot recover documents absent from candidates |
| Prompted LLM ranker | Usually no, candidate reranker | little task-specific training | high; repeated generation/logits | zero-shot instructions and reasoning-rich comparison | prompt/order sensitivity and format failures |
| LC-LLM / CiC | Yes only if corpus fits context | construct/cache corpus prefix | very high without caching | consolidated small-corpus search | position bias, context cost, lost-in-the-middle |
| RAG system | Retriever does | index + prompt/evaluation design | retrieval + long generation | current, inspectable external knowledge | distractors, evidence conflict, unsupported answers |
Exam memory chain: BM25 counts and normalises words → DPR compares one vector per text → ColBERT keeps token vectors → SPLADE/TILDEv2 return to sparse term dimensions with learned weights → monoBERT/LLM rankers deeply compare candidates → RAG passes selected evidence to a generator.
Acronym and definition glossary
缩写、英文全称与核心定义
沿用 3208 学习页的阅读顺序：缩写 → 完整英文名 → 中文名 → 一句话定义。不是每个模型名都是首字母缩写；名称型模型会明确标注其作用。
| Short name | Full name | 中文 | Definition / 定义 |
|---|---|---|---|
| IR | Information Retrieval | 信息检索 | 从文档集合中找到并排序能够满足用户信息需求的内容。 |
| TF | Term Frequency | 词频 | 某个 term 在当前 document 中出现的次数。 |
| DF | Document Frequency | 文档频率 | 集合中包含某个 term 的 documents 数量。 |
| CF | Collection Frequency | 集合频率 | 某个 term 在整个 collection 中出现的总次数。 |
| IDF | Inverse Document Frequency | 逆文档频率 | 根据 term 在多少文档中出现来衡量其区分度。 |
| TF-IDF | Term Frequency-Inverse Document Frequency | 词频-逆文档频率 | 结合文档内频率与集合内稀有性的词项权重。 |
| VSM | Vector Space Model | 向量空间模型 | 把 query 和 document 表示为向量并比较其方向或相似度。 |
| BIM | Binary Independence Model | 二元独立模型 | 假设词项独立，用词在相关与非相关文档中的概率差异打分。 |
| BM25 | Best Matching 25 | 最佳匹配第 25 版 | 结合 IDF、term-frequency saturation 与 document-length normalisation 的经典词汇检索模型。 |
| RSJ | Robertson-Spärck Jones | 罗伯逊-斯帕克·琼斯权重 | BM25 常用的概率式 IDF 权重来源，用相关与非相关文档中的词项证据衡量区分度。 |
| F1 | F-score with β = 1 | F1 分数 | Precision 与 recall 的调和平均，在两者同等重要时使用。 |
| AP | Average Precision | 平均精确率 | 对一个 query，在每个相关结果出现位置记录 precision 并平均。 |
| MAP | Mean Average Precision | 平均精确率均值 | 对所有 queries 的 AP 取平均。 |
| RR / MRR | Reciprocal Rank / Mean Reciprocal Rank | 倒数排名 / 平均倒数排名 | 关注第一个相关结果的位置，再在 queries 间取平均。 |
| DCG | Discounted Cumulative Gain | 折损累计增益 | 让高相关文档得分更高，并按排名位置进行折扣。 |
| IDCG | Ideal Discounted Cumulative Gain | 理想折损累计增益 | 同一 query 在理想相关性顺序下可获得的 DCG。 |
| nDCG | Normalized Discounted Cumulative Gain | 归一化折损累计增益 | 以 DCG 除以 IDCG，使不同 query 的排名质量可比较。 |
| RBP | Rank-Biased Precision | 排名偏置精确率 | 用 persistence 参数模拟用户继续浏览下一结果的概率。 |
| BERT | Bidirectional Encoder Representations from Transformers | 双向 Transformer 编码表示 | 使用双向上下文的 encoder 模型，可联合编码 query-document pair。 |
| CLS / SEP / EOS | Classification / Separator / End of Sequence token | 分类 / 分隔 / 序列结束标记 | 分别用于汇总分类表示、分开输入片段，以及标记序列结束。 |
| MLM | Masked Language Modeling | 掩码语言建模 | 遮住部分 tokens，再根据上下文预测它们的预训练目标。 |
| FFN | Feed-Forward Network | 前馈神经网络 | Transformer block 中对每个 token 独立执行的非线性特征变换。 |
| GPT | Generative Pre-trained Transformer | 生成式预训练 Transformer | 以自回归 next-token prediction 为核心的 decoder 模型家族。 |
| LLM | Large Language Model | 大型语言模型 | 在大规模语料上训练、可通过自然语言 prompt 完成多种任务的模型。 |
| DPR | Dense Passage Retrieval | 稠密段落检索 | 分别编码 query 与 passage，并通过 dense-vector similarity 检索。 |
| ColBERT | Contextualized Late Interaction over BERT | BERT 上下文化后期交互模型 | 保留 query 与 document 的 token vectors，以 MaxSim 进行细粒度 late interaction。 |
| ANN | Approximate Nearest Neighbour | 近似最近邻 | 以少量精度损失换取大规模向量搜索速度。 |
| ANCE | Approximate Nearest Neighbor Negative Contrastive Estimation | 近似近邻负样本对比估计 | 利用当前 dense model 的 ANN 检索错误动态挖掘 hard negatives。 |
| NLL | Negative Log-Likelihood | 负对数似然 | 惩罚模型给正确目标分配较低概率的损失函数。 |
| CE | Cross-Entropy | 交叉熵 | 衡量目标分布与模型预测分布差异的分类损失。 |
| InfoNCE | Information Noise-Contrastive Estimation | 信息噪声对比估计 | 让正样本相对于一组 negatives 获得更高相似度的对比学习目标。 |
| TILDE | Term Independent Likelihood moDEl | 词项独立似然模型 | 从文档侧学习 vocabulary term likelihood，用于高效查询似然打分。 |
| SPLADE | Sparse Lexical and Expansion Model | 稀疏词汇扩展模型 | 学习 vocabulary-aligned sparse weights，并可激活原文未出现的相关词。 |
| ReLU | Rectified Linear Unit | 修正线性单元 | 使用 max(0,x) 去除负激活；SPLADE 用它产生非负词项权重。 |
| FLOPS | Floating-Point Operations | 浮点运算量 | SPLADE 中以预期计算开销为直觉的稀疏正则项，抑制过多词项被激活。 |
| T5 | Text-to-Text Transfer Transformer | 文本到文本迁移 Transformer | 将所有任务表述为文本到文本生成；doc2query 可用它离线生成文档扩展词。 |
| LTR | Learning to Rank | 学习排序 | 从 labels、preferences 或 lists 学习候选文档的排序函数。 |
| OLTR | Online Learning to Rank | 在线学习排序 | 通过实时用户交互持续评价和更新 ranker。 |
| IPS | Inverse Propensity Scoring | 逆倾向评分 | 用观察概率的倒数重新加权点击，以校正曝光或位置偏差。 |
| DBGD | Dueling Bandit Gradient Descent | 对决老虎机梯度下降 | 用 interleaving 比较当前与探索 ranker，并向获胜方向更新。 |
| COLTR | Counterfactual Online Learning to Rank | 反事实在线学习排序 | 使用记录了 propensities 的历史点击数据离策略评估候选 rankers。 |
| RM3 | Relevance Model 3 | 相关性模型 3 | 从初始检索结果估计反馈模型并扩展原 query。 |
| TREC | Text REtrieval Conference | 文本检索评测会议 | 提供标准 collections、topics 和 evaluation tracks 的信息检索评测体系。 |
| Qrels | Query Relevance Judgements | 查询相关性判断 | 记录 query-document relevance labels 的标准答案文件，用于评价 run。 |
| MS MARCO | Microsoft Machine Reading Comprehension | 微软机器阅读理解数据集 | 由真实搜索查询和人工答案/段落构成，常用于训练与评价 neural rankers。 |
| BEIR | Benchmarking Information Retrieval | 信息检索基准套件 | 跨多个领域测试 retriever 的 zero-shot generalisation。 |
| JSONL | JSON Lines | 逐行 JSON 格式 | 每一行保存一个独立 JSON object，适合流式读取文档集合。 |
| LoRA | Low-Rank Adaptation | 低秩适配 | 冻结大部分原模型参数，只训练低秩更新矩阵的高效微调方法。 |
| MNTP | Masked Next Token Prediction | 掩码下一词预测 | 通过遮住 token 并利用双向上下文训练 decoder 表示的目标。 |
| SimCSE | Simple Contrastive Learning of Sentence Embeddings | 句向量简单对比学习 | 以对比目标学习句子表示的方法。 |
| PARADE | Passage Representation Aggregation for Document Reranking | 面向文档重排序的段落表示聚合 | 先编码长文档的各 passages，再聚合其 representations 做文档级评分。 |
| LLM4IR | Large Language Models for Information Retrieval | 用于信息检索的大语言模型 | 使用 LLM 改善 retrieval、reranking 或其他 IR tasks。 |
| IR4LLM | Information Retrieval for Large Language Models | 用于大语言模型的信息检索 | 通过 retrieval 为 LLM 提供外部、可更新的证据。 |
| monoBERT | Model name, not an acronym | 单文档 BERT 重排器 | 对每个 query-document pair 独立输出 pointwise relevance score。 |
| duoBERT | Model name, not an acronym | 双文档 BERT 重排器 | 比较两个 candidates，输出对 query 的 pairwise preference。 |
| RAG | Retrieval-Augmented Generation | 检索增强生成 | 在生成前从外部 collection 检索证据，并把 query 与 evidence 一起交给生成模型。 |
| LC-LLM | Long-Context Large Language Model | 长上下文大语言模型 | 能够接收远长于传统模型输入窗口的 prompt，但成本与位置偏差仍需考虑。 |
| CiC | Corpus-in-Context | 语料库置于上下文 | 把整个可容纳 corpus 与示例放入 LC-LLM prompt，并生成相关 document IDs。 |
| CoT | Chain-of-Thought | 思维链提示 | 通过示例或指令诱导模型生成中间推理步骤；生成文字不等于可验证的内部推理。 |
| VLM | Vision-Language Model | 视觉语言模型 | 联合处理像素与文本；screenshot retrieval 用它保留页面布局和视觉内容。 |
| HyDE | Hypothetical Document Embeddings | 假想文档嵌入 | 先生成一个假想相关文档，再用其表示检索真实文档。 |
| RARR | Retrofit Attribution using Research and Revision | 通过研究与修订补加归因 | 为已有答案搜索支持证据并修正无支持内容。 |
| PPO | Proximal Policy Optimization | 近端策略优化 | 一种限制单次策略更新幅度的强化学习方法；可训练 query rewriter。 |
| Attribution | Evidence attribution | 证据归因 | 把生成答案中的 claims 与真实、支持它们的 references 或 citations 对齐。 |
Assignment 1
A1 完整实施流程
作业不是只写一个 BM25 函数，而是完成一个可验证、可调参、可比较的 IR 实验。
为什么分 train/test
Train qrels 用来选择 k1、b；test qrels 只用于最终一次无偏评价。若看着 test 结果反复改参数，就把 test 泄露成了 train，最终分数会过于乐观。
怎样讨论 tuned BM25
不要只写“分数提高”。解释 k1 改变 tf saturation、b 改变 length normalization，并结合 gain/loss queries 判断数据集是偏短文档、长文档，还是 query 中 rare terms 的作用更明显。
打开实验复习流程. 公开版仅保留方法讲解，不包含个人提交文件和实验输出。
# 一个清晰的 scorer 骨架（变量名按你的 notebook 调整）
def bm25(tf_vector, df_vector, doc_len, k1=1.2, b=0.75):
score = 0.0
for tf, df in zip(tf_vector, df_vector):
idf = math.log(N / (1 + df))
norm = 1 - b + b * (doc_len / average_doc_length)
tf_weight = ((k1 + 1) * tf) / (tf + k1 * norm)
score += idf * tf_weight
return score
Recall practice
自测题
先在脑中回答，再展开答案。能够解释“为什么”才算掌握。
1. tf、df、cf 分别是什么？
tf：term 在当前 document 的次数；df：包含 term 的 documents 数；cf：term 在整个 collection 中的总次数。
2. 为什么 BM25 不让 tf 线性增长？
同一词从 0 到 1 很重要，但从 100 到 101 提供的信息很少。TF saturation 防止重复堆词无限提高分数。
3. nDCG 为什么要除以 IDCG？
不同 query 的相关文档数量和等级不同；归一化后才能把它们放在近似 0–1 尺度上比较和平均。
4. p-value < .05 说明什么？
在 H₀ 成立时，观察到当前或更极端差异的概率低于 5%。它不是系统 B 更好的概率，也不直接表示提升幅度。
5. Self-attention 为什么除以 √d_k？
维度增大时点积方差随之增大；缩放可避免 softmax 过尖，保持梯度稳定。
6. Encoder 和 decoder 的关键区别？
Encoder 一般可双向看完整输入，擅长理解；decoder 使用 causal mask，只看左侧前缀，擅长自回归生成。
7. Dense retrieval 为什么能处理同义词？
模型通过训练把语义相近表达映射到邻近向量，不再要求 query 与 document 必须共享完全相同的表面词项。
8. ColBERT 分数怎么计算？
对每个 query token 找到所有 document tokens 中的最高相似度，再把各 query token 的最大值相加，即 MaxSim。
9. Hard negatives 为什么有用又危险？
它们与 query 很像但被标为不相关，能提供强梯度；但 qrels 不完整时，其中可能实际相关，形成 false negatives。
10. 为什么不能在 test set 调 BM25 参数？
这会让参数适配 test qrels，破坏 test 作为未见数据的角色，导致对 generalization 的估计偏乐观。
11. Why is SPLADE sparse even though it uses BERT?
Sparsity describes the output: a vocabulary-sized vector with mostly zero entries. BERT is the neural mechanism that learns which vocabulary dimensions to activate.
12. TILDEv2 and SPLADE differ most clearly where?
TILDEv2 keeps query processing lightweight with token IDs and learns document-token weights, while SPLADE encodes both query and document into learned sparse vocabulary vectors.
13. Why use BM25 before TILDEv2 reranking?
BM25 cheaply reduces the full corpus to a candidate set. TILDEv2 then spends richer document-side scoring only on those candidates.
14. Pointwise, pairwise and listwise LTR differ how?
The training unit is respectively one document, a document preference pair, or an entire ranked list.
15. Why can clicks not be used as relevance labels directly?
Clicks depend on examination, position, selection, presentation and noise. An unclicked item may simply never have been seen.
16. What is the main purpose of IPS?
To correct exposure or position bias by weighting an observed click with the inverse probability that the item was examined.
17. How does balanced interleaving compare two rankers?
It alternates unseen documents from both rankers into one displayed list and attributes clicks to infer a pairwise preference under nearly identical context.
18. Counterfactual OLTR versus regular interleaving?
Counterfactual OLTR can reuse one propensity-logged interaction stream to estimate many candidates off-policy; pairwise interleaving normally needs a newly mixed list for each direct comparison.
19. Why must monoBERT follow a first-stage retriever?
Joint query-document encoding is too expensive for every document. BM25 first supplies a manageable candidate set; monoBERT can only reorder that set.
20. Score aggregation and representation aggregation differ how?
MaxP/FirstP/SumP combine scalar passage scores. PARADE first combines learned passage vectors, allowing a richer document-level decision.
21. monoBERT and duoBERT differ how?
monoBERT independently estimates relevance for one document; duoBERT directly estimates which of two documents should rank higher.
22. Pointwise, pairwise, listwise and setwise LLM reranking differ how?
They respectively judge one document, compare two, generate an order for a list, or select the best item from a small set.
23. Why are setwise LLM rankers potentially cheaper?
One inference compares several passages using logits, requiring fewer comparisons than pairwise ranking and less generation than a full listwise permutation.
24. Why can prompt wording change ranking effectiveness?
Role text, output constraints, formatting and component order alter the model context. Sensitivity also depends on model size, architecture and training.
25. LLM4IR 与 IR4LLM 的方向有什么不同？
LLM4IR 用 LLM 改善 retrieval/ranking；IR4LLM 用 retrieval 给 LLM 提供外部证据。一个 LLM reranker 属于前者，RAG 属于后者。
26. Why might CiC replace a retriever for a small corpus?
If the corpus fits the context window, the LC-LLM can read it directly and generate document IDs, avoiding a separate retrieval model. Cost, caching and position bias remain.
27. 为什么 long context 不等于模型会使用所有证据？
注意力使用有位置偏差，关键证据位于中间时可能被忽略；更多 distractors 也会稀释或冲突关键证据。
28. RAG 与 fine-tuning 的知识更新方式有什么不同？
RAG 在 inference 时替换外部 evidence；fine-tuning 修改参数。RAG 更容易更新和检查来源，fine-tuning 不需要每次提供证据但知识不容易单独替换。
29. What can happen when retrieved evidence contradicts model knowledge?
The model may ignore the evidence, preserve a wrong prior answer, or flip a correct answer. Groundedness must therefore be evaluated separately from retrieval relevance.
30. 为什么 prompt compression 不是免费压缩？
它减少 tokens 与成本，但 lexical deletion 或 learned memory tokens 都可能丢掉限定条件、来源或少数关键证据。
31. Rewrite-Retrieve-Read 的三个对象分别是什么？
Rewriter 产生更适合搜索的 query；retriever 找 evidence；reader 根据原问题与 evidence 生成答案。Rewrite drift 是主要风险。
32. What is the difference between relevance and groundedness?
Relevance asks whether a passage helps the information need; groundedness asks whether answer claims are actually supported by supplied evidence.
33. 为什么 nDCG 提升不保证答案分数提升？
生成还受 passage truncation/order、prompt、模型先验和证据冲突影响。只能通过对齐 query 的差值与 correlation 实证检查。
34. What does attribution add beyond a fluent answer?
It identifies evidence supporting each claim, allowing verification. A fluent answer without valid support may still hallucinate.
35. Screenshot retrieval 解决了 text-only retrieval 的什么损失？
它保留布局、图像、表格和空间关系等纯文本抽取可能丢失的信号，但增加视觉编码成本。
36. RARR 的顺序是什么？
先有 draft answer，再 research evidence，最后 revise unsupported content 并补充 attribution；它是 post-generation repair，而不是 first-stage retrieval。
Source map
本手册整理范围
内容来自你提供的本地 INFS7410 课程目录；大体积 corpus、index、venv 只作为实验资源，不纳入复习正文。
- Week 1 lecture notes / slides / practical notebook
- Week 2 lecture notes / slides / practical notebook
- Week 3 lecture notes / slides / practical notebook
- Week 4 lecture notes / transformer worksheets / practical solution
- Week 5 lecture notes / slides
- Week 5 InfoNCE practical solution
- Week 5 MarginLoss practical solution
- Week 5 HardNegatives practical solution
- Week 6 practical: TILDEv2 and SPLADEv2
- Week 6 learned sparse lecture notes
- Week 7 full slide deck: offline/online LTR, IPS, interleaving and COLTR
- Week 8 full slide deck: BERT/LLM reranking, long-document aggregation and prompt variation
- Week 8 practical: BM25 + monoBERT pipeline and LLM pointwise/pairwise reranking
- Week 9 full slide deck: long-context retrieval, Corpus-in-Context, specialised retrievers, modular RAG, evidence conflict, noise and attribution
- Project 2 notebook, sampled BRIGHT Biology collection and marking sheet
- A1 project notebook
- A1 marking sheet
Coverage / 范围：This edition integrates Weeks 1–9. Week 6 connects lexical and dense retrieval through learned sparse representations; Week 7 connects rankings to click feedback; Week 8 introduces neural/LLM reranking; Week 9 connects retrieval to long-context search, evidence-based generation and attribution.
复习问题
通过 44 组知识点讲解与计算示例复习。公开版不提供原始 PDF、课件截图或课件全文。
没有找到匹配内容。试试更短的关键词。