1. Candidate retrieval / 候选召回
BM25, DPR, SPLADE and TILDE-family models search a large collection and return a manageable candidate set.
Information Retrieval · Weeks 1–13 · Practicals · Projects
This guide follows one connected Information Retrieval (IR) pipeline: represent and index documents, retrieve candidates, rerank them, evaluate the ranking, learn from interaction data, and provide evidence to a generator. 中文主线:表示文本 → 召回候选 → 神经重排序 → 离线/在线评价 → 检索增强生成 → 证据归因。
把讲义概念与对应练习放在一起:每周沿着任务、数据流、代码步骤和检查点学习。
资料核对:2026 年 10 月 20 日。补齐 Week 5、7、9、11、12 的课程材料,并更新 Week 8;其余周次使用下方标明的本地归档。本页是学习讲解,不代表重新运行了全部模型实验。
把每周知识放回同一个 pipeline:retrieval 决定“看哪些文档”,ranking 决定“先看哪些”,evaluation/feedback 决定“如何判断并继续学习”。
corpus 是所有文档;query 是用户输入;run 是系统产生的排名;qrels 是标准相关性答案。评价是在 run 与 qrels 之间进行。BM25, DPR, SPLADE and TILDE-family models search a large collection and return a manageable candidate set.
A cross-encoder or learned ranker uses richer features to reorder candidates. Pointwise, pairwise and listwise describe different learning targets.
Offline qrels support nDCG/MAP. Online clicks require bias correction, interleaving or counterfactual estimators.
目标:理解搜索引擎如何把大量文本变成可以快速查找的数据结构。
数据库查询通常要求精确匹配并返回确定答案;IR 面对自然语言和不确定的信息需求,因此返回按相关性排序的结果。Relevance 不是单纯“出现了相同词”,而是文档是否满足该用户在该情境下的信息需求。
Corpus / collection 是全部文档;document 是一条独立记录;token 是分词后的一次出现;term 是索引使用的规范词项。一个 term 可以在同一文档中出现多次。
tf某词在一个文档中的次数 df包含该词的文档数 cf该词在整个集合中的总次数
少数词极其频繁,大多数词很少出现。按频率排序后,rank 越高,frequency 越低,近似呈幂律。
图上会出现长尾:最高频词通常是功能词;最低频词可能是专名、拼写变体或噪声。这解释了 stopwords 与词表压缩的动机。
正向存储是 document → terms;倒排索引反过来保存 term → postings list。查询词出现在哪些文档,可直接读取对应 postings,不需要逐篇扫描 corpus。
| 层级 | posting 保存什么 | 支持的能力 |
|---|---|---|
| Boolean | docID | 判断是否出现,做 AND / OR / NOT |
| Counts | docID + tf | TF、TF-IDF、BM25 排名 |
| Positions | docID + tf + positions | 短语查询、邻近查询、高亮 |
| Doc vectors | 每篇文档的 term 与统计 | 课程中自定义 scorer 读取 tf/df;A1 要求 -storeDocvectors |
df 不是某词出现总次数。某个词在一篇文档中出现 20 次,这一篇对 df 的贡献仍然只是 1,但对 cf 的贡献是 20。index_reader.stats() 验证索引。Turn a JSON Lines (JSONL) collection into an inverted index, then use real term statistics to explain Zipf’s law and the effects of preprocessing.把逐行 JSON 文档变成倒排索引,再用真实词项统计理解 Zipf 定律以及预处理的影响。索引是后续检索的基础,不是评价结果。
Data flow: JSONL documents → analyzer → Lucene index → term statistics → frequency plots → reindex and compare
One JSONL line represents one document. Parse it, identify the ID and text fields, and count documents separately from tokens. A document ID connects later search hits to the same original text.每行是一篇文档。先确认字段,不要把行号当成 docid。文档数、所有 token 出现次数、不同 term 数是三个不同统计量。
The supplied configuration keeps stopwords and disables stemming. Tokenisation still occurs: “noProcessing” is a folder label, not a claim that raw strings are indexed without an analyzer. Inspect index_reader.stats() after indexing.noProcessing 不等于完全不分词。先记录 analyzer 设置,再检查索引中的文档数和词项数;查询端之后必须使用匹配的处理规则。
Sort terms by collection frequency (cf), then put frequency rank on the horizontal axis. The high-frequency head and long tail illustrate Zipf’s law. Compare the >50,000-frequency terms with the least frequent 20 terms; they expose different parts of the distribution.按全集合出现次数排序,不是按字母排序,也不是按 df 排序。极少出现的词可能是专名或噪声,不能看到低频就直接删除。
Build a second index using Porter stemming and stopword removal. Recompute the statistics instead of reusing the previous vocabulary. Fewer distinct forms may improve matching, but can also merge different meanings.同一 collection,改变预处理,再观察词表和分布。是否提高检索效果要等 Week 2 用 qrels 验证;词表变小本身不等于效果更好。
If a term occurs 7 times in one document and 3 times in another, its cf is 10 and df is 2. Its tf in the first document is 7. APIs sometimes call collection frequency “term frequency”; interpret the scope of the returned statistic.同一个词:文档内 tf=7;出现过该词的文档数 df=2;整个集合的总次数 cf=10。看清 API 的统计范围比只记变量名重要。
Both index paths, their analyzer settings, stable document IDs and the same collection. Week 2 adds queries and qrels to test effectiveness, rather than judging quality from vocabulary size.两个索引、各自预处理设置和稳定 docid。Week 2 比较的是检索结果,不是看哪个索引词更少。
Source map: prac-week1.ipynb: Exercises 1–2 and reindexing comparison. Independent study explanation; original files remain on the course platform.
Back to Week 1 lecture concepts离线评价需要 queries、qrels 和 runs。不同指标体现不同用户行为,不能只记公式。
Precision 关心“返回的结果有多干净”;Recall 关心“所有相关文档找回了多少”。搜索引擎首页更偏 precision,法律检索或医学系统常更重 recall。
F1 是 precision 与 recall 的调和平均;β>1 更重 recall,β<1 更重 precision。
只有在第 k 位相关时,P@k 才会进入 AP。没有被检索到的相关文档仍留在分母 |Rel| 中,因此会降低 AP。
只关心第一个相关答案在哪里,适合问答或导航型查询;不会奖励第二、第三个相关结果。
高相关等级获得指数增益,排名越靠后获得折扣。IDCG 是把同一 query 的结果按理想相关性排序后的 DCG,因此 nDCG 通常在 0–1。
p 是用户继续查看下一条的概率,期望查看深度是 1/(1-p)。p 越大,越重视更深的排名。
同一批 queries 分别由系统 A 和 B 得到分数,先计算每个 query 的差值,再检验差值均值是否显著偏离 0。
p < .05 常称 statistically significant;p < .01 可称 strongly significant。
p-value 是:假设 H₀ 成立时,观察到当前或更极端差异的概率。它不是“H₀ 为真的概率”,也不代表效果大小。
KeyError: ndcg_cut_5 通常意味着 evaluator 初始化时没有请求这个 measure,而不是 t-test 本身出错。search(..., k=1000) 生成 run → pytrec_eval.parse_run 读取 → evaluator 对每个 query 计算指标 → aggregate 得到总体均值 → scipy.stats.ttest_rel 比较两个系统的逐 query 分数。这里的 k 是每个 query 保留的 top-k 文档数,不是 BM25 的 k1。Compare two analyzer configurations and two retrieval depths using the same queries and judgments. Learn what a metric, a significance test and a plot each contribute.固定 queries 和人工相关性判断,比较两套索引与两个召回深度。指标、显著性检验、图表分别回答不同的问题。
Data flow: queries.tsv + index → ranked run → qrels join → per-query measures → paired differences → plots
Search both Week 1 indexes with their matching query analyzers. Retain qid, docid, rank and score. Retrieval depth k=10 or k=1000 is not the BM25 parameter k1 and is not automatically the evaluation cutoff.qid 连接 query 和 qrels,docid 连接结果和文档。召回 1000 条,仍可只计算前 10 条的 nDCG@10。
Precision and recall describe the returned set. Average Precision (AP) and normalised Discounted Cumulative Gain (nDCG) also reward ordering. Mean Average Precision (MAP) averages AP across queries. IDCG uses the query’s available judgments, including relevant documents not retrieved.调换相同集合的顺序不改变整体 precision/recall,但可能改变 AP、nDCG 和截断指标。不能只用返回结果里的相关文档建立理想排序。
Compute one metric value per qid per system. Align arrays by the same qid order before ttest_rel. A small p-value is evidence against the no-mean-difference hypothesis under the test assumptions, not the probability that the null hypothesis is true.配对单位是同一 query,不是两张随机顺序的分数表。p-value 不告诉你提升有多大,也不能保证换一批 query 仍然提升。
For each query, compute candidate minus baseline. Positive bars are gains; negative bars are losses. Use mean scores, a gain–loss plot and a distribution plot together, then inspect documents for a large gain and a large loss.平均值可能掩盖部分 query 的退步。先说明差值方向,再回到 query 和文档解释原因,不要仅写“柱子更高”。
The four conditions are unprocessed/processed × top-10/top-1000. Hold depth fixed when comparing analyzers. For a deterministic ranker, merely extending the same ranking from 10 to 1000 should not alter its first 10 results; changes to nDCG@10 then require investigation.四组实验可以组成 2×2 对照。比较预处理时固定深度;比较深度时固定模型。只扩展结果列表,不应凭空改变其前十名。
No. Report the effect size and query-level failures as well. Ensure requested measures exist in RelevanceEvaluator before looking up keys such as ndcg_cut_10.不是。还要说明提升幅度和退步的 queries;KeyError 可能只是没有请求对应评价指标。
Source map: prac-week2.ipynb: Exercises 0–4. Independent study explanation; original files remain on the course platform.
Back to Week 2 lecture concepts从“是否包含查询词”逐步增加词频、稀有性、长度归一化和概率解释。
| 模型 | 核心想法 | 公式 / 作用 | 主要限制 |
|---|---|---|---|
| Boolean | 满足逻辑条件即匹配 | AND / OR / NOT postings | 通常不产生细致排名 |
| Coordination | 匹配更多不同 query terms 更好 | matched query terms 数 | 忽略同一词出现次数和稀有性 |
| TF | query term 在文档中出现更多更好 | Σ f(qi,D) | 常见词和长文档占便宜 |
| IDF | 少见词区分度更高 | log(N/(1+dfi)) | 不看当前文档中的 tf |
| TF-IDF | 局部频率 × 全局稀有性 | Σ tfi,D × idfi | tf 线性增长、长度处理粗糙 |
| VSM | 把 query/doc 当向量比较方向 | cosine(q,d) | 仍依赖词项重合 |
| BIM | 估计词在 relevant/non-relevant 中的区别 | 概率相关性权重 | 词项独立假设;需估计概率 |
| BM25 | IDF + tf saturation + length normalization | 经典强稀疏基线 | 无法天然解决同义词和语义匹配 |
1 + df 是平滑,避免 df=0 时除零。IDF 只依赖 term 和整个 collection,不依赖当前 document。
点积受向量长度影响,cosine 先归一化,比较“方向”。若向量已 L2-normalized,则 dot product 与 cosine 相同。
|D|/avgdl 调整。讲义中的完整 BM25 还可包含 query term frequency 因子;A1 的模板采用课程指定的简化 IDF。作业时以 notebook/marking sheet 的公式为准,不要自行混用不同 BM25 版本。
scorer(tf_vector[term], df_vector[term], doc_len),传入的是一个词的 tf、这个词的 collection df,以及当前文档长度。前两项已经用下标取成数字,不是整个向量。你的 scorer 返回该词的贡献,外层 search helper 再对匹配的 query terms 累加,然后排序候选文档。assert score(tf=3) > score(tf=1) 检查更多词频是否提高得分;assert score(short_doc) > score(long_doc) 检查长度归一化;assert np.isfinite(...) 检查没有 NaN/∞。assert 条件为 False 时立即抛出 AssertionError,因此它是一种小型自动测试。Implement Term Frequency (TF), Inverse Document Frequency (IDF), TF-IDF and BM25 (Best Matching 25), then compare them through Week 2’s evaluation workflow.把公式拆成可检查的单词贡献,再累加为文档分数。这里的检索模型产生排名,评价指标才负责判断排名质量。
Data flow: index statistics + query terms → per-term scorer(tf, df, doc_len) → summed document score → reordered candidates → run
The practical reindexes with document vectors and positions. The document vector supplies term counts; summing its values gives the analyzed document length. Collection document count N and average document length avgdl must come from the same index.不要把字符数当 doc_len,也不要混用另一套 analyzer 的 avgdl。这里的长度是分析后词项出现次数之和。
The checked helper calls scorer(tf_vector[term], df_vector[term], doc_len). Despite the parameter names, each call receives scalar tf and df for ONE matched term. The search helper owns the outer summation over matching terms.老师变量名带 vector,但实际传入的是下标取出的数字。你写的是一个词的贡献,不能再对数字 zip,也不能重复累计整条 query。
TF returns tf; IDF returns log(N/(1+df)); TF-IDF multiplies them. BM25 adds tf saturation and a length-dependent denominator. Return a numeric value, and test the one-term case before processing every query.先算 IDF、长度归一化、词频饱和,再相乘。测试 b=0 时长度惩罚消失;正 IDF 的固定词项下,更多 tf 的收益应逐渐变小。
The supplied searcher first obtains k candidates, then your scorer reorders those IDs. This is a candidate-pool comparison, not exhaustive retrieval with each custom formula. Reordering an unchanged complete top-10 set cannot change its P@10 or R@10, although AP/nDCG can change.同一批十篇文档只换顺序,前十篇的相关数量不变。若换了候选集合或只取更小 cutoff,precision/recall 才可能变化。不要把受候选限制的实验说成全库检索。
For a matched term with tf=3, df=9, N=100, doc_len=avgdl, k1=1.2 and b=0.75: IDF=ln(10); the tf factor is 6.6/4.2; the contribution is about 3.61835. Only the outer helper adds other query terms.文档长度等于平均长度时,长度因子为 1。算出的约 3.61835 是一个词的贡献,不是所有 query terms 的最终总分。
The IDF convention, analyzer, query-term treatment, candidate set and parameter values may differ. Check these before assuming the difference is a coding error or a quality improvement.同名 BM25 可能采用不同 IDF 版本;先对齐公式与设置,再解释分数差异。
Source map: prac-week3.ipynb: Exercise 0, ranking functions 1.1–1.4, Exercise 2. Independent study explanation; original files remain on the course platform.
Back to Week 3 lecture concepts经典稀疏模型要求词项重合;Transformer 学习上下文相关表示,为语义匹配提供基础。
Tokenizer 把文本映射成 token IDs;nn.Embedding(V,D) 像一张可训练查找表,把每个 ID 映射为 D 维向量。D 是模型设计参数,例如 BERT-base 的 hidden size 是 768。
形状常写成 N × D:N 是 tokens 数量,D 是每个 token 的表示维度。
纯 self-attention 不知道顺序,所以加入 position embedding。BERT 处理句对时,segment/token-type embedding 区分 A 段与 B 段;[SEP] 是边界 token。
## 表示该片段接续在前一个子词后面。QKᵀ 产生 N×N 分数矩阵,表示每个 token 对所有 token 的关注程度。√d_k 防止维度较大时点积过大、softmax 过度饱和。多个 attention heads 在不同投影子空间中并行建模关系,例如一个 head 偏语法依赖,另一个偏实体关系。各 head 输出拼接后再投影回 D 维。
FFN 对每个 token 独立地做非线性特征变换,通常 D→4D→D。这里 hidden 只是中间激活张量,不是手工设置的“参数”;真正训练的参数是 W、b。Residual 保留原信息,LayerNorm 稳定训练。
Encoder 可以同时看到左右上下文,适合分类、理解、ranking 和 embedding。BERT 以 masked language modelling 为主要预训练目标。
Decoder 使用 causal mask,只能看当前 token 左侧,从而逐 token 生成。Encoder backbone 不能直接像 decoder 一样自然生成,需要额外 decoder 或改造目标与 attention mask。
out = tf_block(x) 得到 None,说明 forward() 或其中调用的方法漏写了 return x/return out。Python 函数没有 return 时默认返回 None,因此 out.shape 会报错。Build the simplified Transformer layer, inspect BERT (Bidirectional Encoder Representations from Transformers), and contrast it with GPT (Generative Pre-trained Transformer) next-token prediction.这一周先理解网络如何生成 contextual embeddings;能生成向量不代表已经训练成检索模型。保留了本地 starter 与 solution,版本标记不冒充最新验证。
Data flow: token IDs → embeddings [B,L,H] → attention and FFN → contextual states → pooling or vocabulary head
The small layer uses x shaped [N,D] without an explicit batch axis. Q=XWq, K=XWk and V=XWv keep [N,D]; QKᵀ becomes [N,N]. Softmax over keys makes each query-token row sum to 1. Multiplying by V returns [N,D].N 是 token 数、D 是隐藏维度;attention 矩阵描述 token 对 token,不是文档对文档。softmax 的维度选错,就会改变权重的含义。
The Feed-Forward Network (FFN) changes D→4D→D with a nonlinear activation between projections. Residual addition requires matching dimensions. Return the output from both SelfAttention and FFN; a missing return propagates None.扩展维度是中间计算,输出要回到 D 才能与输入相加。NoneType.shape 往往是函数没有返回 tensor,而非维度真的不存在。
Token IDs have shape [B,L]; last_hidden_state has [B,L,H]. CLS pooling or mask-aware mean pooling produces [B,H]. Mask padding in mean pooling. Compare the embedding module output before the Transformer with the contextual output after its layers.B 是 batch 数,L 是 token 长度,H 是 hidden size。token ID 只是词表编号,embedding 是向量;经过上下文层后,同一个 token 可以获得不同表示。
Masked Language Modeling (MLM) predicts a selected masked position from bidirectional context. GPT’s vocabulary logits at the final real input position predict the next token. Autoregressive generation appends a token and repeats.BERT 填空和 GPT 续写是不同任务。因果 attention 若行是 query、列是 key,则右上方未来位置被屏蔽,允许关注的区域通常在下三角。不要不看坐标就背“上三角”。
A batch of 2 texts padded to 6 tokens has IDs [2,6] and, for H=768, hidden states [2,6,768]. Mask-aware mean pooling gives [2,768], not [2,6] or [768]. These pooled vectors can become retrieval inputs in Week 5 after appropriate model training.池化只消掉 token 这一维,不会消掉 batch,也不会把 hidden size 变成词表大小。Week 5 再学习如何让这些向量适合相关性比较。
No. A vocabulary prediction head is trained for token prediction. Retrieval relevance requires an appropriate representation, scoring rule and training/evaluation setup.不是。预测词的能力与判断某文档是否满足 query 的能力,需要通过任务与训练联系起来。
Source map: prac-week4.ipynb and prac-week4-solution.ipynb: Exercises 1–3; archived starter labels itself 2025.1. Independent study explanation; original files remain on the course platform.
Back to Week 4 lecture concepts把 query 和 document 映射到向量空间,使“car repair”也能接近“automobile maintenance”。
query 和 document 分开编码。文档向量可以离线预计算并建立 ANN index,在线只编码 query,因此适合第一阶段 retrieval。
把 [query; document] 一起输入 Transformer,允许所有 token 深度交互,通常更准确但无法预先编码文档,适合 reranking 少量候选。
若向量已做 L2 normalize,dot 与 cosine 的排名一致。题目中的 “mean pooling similarity” 不严谨:mean pooling 是聚合表示的方法,不是 similarity function。
目标是提高正样本相似度,同时相对于一组负样本降低其相似度。温度 τ 越小,softmax 越尖锐。In-batch negatives 会把同一个 batch 中其他 query 的正文档当作当前 query 的负样本,提高效率。
只要正样本得分至少比负样本高 margin=1,该 pair 的损失就是 0。
对每个 query token 找最相似的 document token,再把这些最大值相加。它不是单 query vector 与单 document vector 的一次点积。
| 负样本类型 | 来源 | 优点 | 风险 |
|---|---|---|---|
| Random negatives | 集合中随机文档 | 便宜、稳定 | 常常太简单,梯度很弱 |
| In-batch negatives | 同 batch 其他正样本 | 几乎零额外编码成本 | 可能出现 false negatives |
| BM25 hard negatives | BM25 高排但不相关的文档 | 词汇相似、训练更有挑战 | 依赖 qrels 完整性 |
| ANN/self-mined | 当前 dense model 的近邻错误 | 紧跟模型当前弱点 | 计算和刷新索引昂贵 |
smoke_query 分别运行 TF 与 BM25,只取 top-5,确认没有异常、返回结构正确、分数有限、两个模型可能给出不同排名。正式结论仍要用全部 test queries 和 qrels 评价。Compare DPR (Dense Passage Retrieval), ColBERT (Contextualized Late Interaction over BERT) v2 and RepLLaMA. The practical is model inference and scoring, not a new full training run.本周正式 notebook 包括 DPR、ColBERTv2、RepLLaMA。之前只有 InfoNCE 等题解并不等于已经覆盖这份 practical。
Data flow: BM25 candidates → text encoding → single-vector dot product OR token-level MaxSim → descending ranking
The classroom dpr_encode uses ielabgroup/StandardBERT-DR and the first-token hidden state, then flattens one example into [H]. Encode the query and each candidate and take their dot product. The helper truncates at 64 tokens; that is a classroom configuration, not a theoretical DPR limit.课程演示使用共享的 StandardBERT-DR 编码器,不要把它与所有 DPR 双编码器 checkpoint 的接口混为一谈。先看实际模型、pooling 和截断长度。
Read raw text for each BM25 hit, keep docid attached to the vector/score, sort by the new score, and evaluate against the same qrels. Encoding a few candidates on demand teaches the pipeline; scalable dense first-stage retrieval instead pre-encodes the corpus and uses an index.practical 中先由 BM25 缩小范围;全库 dense retrieval 才需要对完整 corpus 建向量索引。不能把重排十篇称为全库 ANN 搜索。
For one pair, query vectors [m,h] times passage vectors [n,h] transposed give [m,n]. Take a maximum over valid passage tokens for EACH query token, then sum those maxima. Match the model’s masks and query augmentation when comparing with the supplied model score.先对每个 query token 找最匹配的文档 token,再求和。不是整个矩阵取一次最大值,也不是先把 token 向量平均。padding 和专用 query tokens 会影响是否复现得分。
The example loads a Llama backbone with a Low-Rank Adaptation (LoRA) adapter, uses query/passage prefixes and an end-of-sequence (EOS) token, then normalises the last-token vectors before a dot product. With unit-length vectors, this is cosine similarity.Decoder 在这里输出的是检索向量,不是回答文本。模型架构是 decoder 不代表此处一定在生成;任务角色要看输出与打分方式。
Toy MaxSim matrix [[0.9,0.2,0.1],[0.3,0.8,0.4]] gives row maxima [0.9,0.8] and score 1.7. This can exceed 1 because it is a sum of token similarities, not a probability. The extra token-level detail costs more storage than one vector per passage.两个 query tokens 分别找到 0.9 和 0.8 的最佳证据,总分 1.7。更多细粒度信息带来更多存储与匹配开销。
Information Noise-Contrastive Estimation (InfoNCE) and margin losses explain how useful embeddings are learned. In-batch and hard negatives determine what must be distinguished. They are training concepts; the current practical mostly loads already-trained models.训练目标解释向量为什么有区分力;practical 使用预训练好的权重来观察结果。false negative 指实际相关却被当作负样本,不是简单指分数较高。
Source map: prac-week5.ipynb: Exercises 1–3; local InfoNCE/MarginLoss/HardNegatives HTMLs are supplementary worked notes. Independent study explanation; original files remain on the course platform.
Back to Week 5 lecture conceptsKeep vocabulary-aligned sparse representations and inverted-index efficiency, but learn term weights and expansion with neural models. 核心不是把向量变密,而是让稀疏词项权重具有上下文与语义。
| Family | Query representation | Document representation | Matching and cost | Main idea / 中文抓手 |
|---|---|---|---|---|
| BM25 | observed query terms | term counts + corpus statistics | inverted index; very cheap online | Hand-crafted lexical weights / 人工公式 |
| DPR | one dense vector | one dense vector | dot product + ANN; query encoder online | Semantic neighbourhood / 向量语义近邻 |
| TILDEv2 | token IDs only; stopwords removed | contextual weights for document tokens, often after offline expansion | exact token-ID match; no query encoder | Learned exact matching / 学习词权重,在线查询很轻 |
| SPLADE | learned sparse vocabulary vector | learned sparse vocabulary vector | sparse dot product; encoder on both sides | Implicit expansion / 自动激活原文没有的相关词 |
| PromptReps | dense hidden state + sparse next-token logits | same two representations | hybrid normalized score; LLM inference | Zero-shot dense+sparse signals / 无需 retriever training |
A relevant document may say “automobile” while the query says “car.” Document expansion adds likely query terms before indexing; doc2query uses a sequence-to-sequence model such as T5 to generate likely queries for each document.
TILDE models document-side term likelihoods over the vocabulary. TILDEv2 instead produces contextual weights for actual or expanded document tokens, then scores only exact matches with query token IDs.
查询词若在文档 token 中出现多次,practical 取该词的最大 contextual weight;未匹配词贡献 0。
An MLM head maps contextual token states into vocabulary-sized term weights. ReLU removes negative activations; pooling combines token evidence. The sparse query and document vectors are scored by a dot product.
SPLADEv2 improves pooling/training, including max-pooling and distillation variants. Sparsity regularisation such as FLOPS/L1 controls how many vocabulary terms remain active.
Sparse/dense describes the output representation, not whether BERT was used. SPLADE uses a Transformer but outputs a mostly-zero vocabulary vector; DPR also uses a Transformer but outputs a dense embedding.
Search the full corpus and keep top-k candidates. High recall and low cost matter here.
Use each hit's document ID and raw() content as the reranker's input.
Score every query-document pair and sort candidates by the new score.
Why two stages? Running a richer model over the whole collection is expensive. BM25 narrows the search space; TILDEv2 spends extra computation only on plausible candidates. 第一阶段负责“不要漏太多”,第二阶段负责“把更相关的推到前面”。
tildev2_scoring function moves datamodel.encode → document token IDs and learned weights. Query text → token IDs → remove stopword IDs. For each query ID, locate equal document IDs, take the maximum weight if it appears more than once, and sum across query terms. The returned scalar is a reranking score, not a probability.torch.topk and the tokenizer vocabulary to decode the strongest terms. Original terms reveal lexical evidence; activated related terms reveal learned expansion. Do not densify the whole collection in a real retrieval index.Connect neural context to lexical matching: TILDEv2 (Term Independent Likelihood moDEl v2) weights document tokens; SPLADE (SParse Lexical AnD Expansion) produces vocabulary-aligned sparse representations.学习型稀疏模型不是“没有神经网络”,而是输出的维度有词项含义。比较模型时要同时看输入、非零维度和匹配规则。
Data flow: document → contextual token weights; query → token IDs → exact matches → max per matching term → sum
The exercise contrasts terms in a passage: equal occurrence counts need not mean equal learned weights. Inspect the token IDs and weights together. Repeated tokens can have different contextual weights before aggregation.TF 只数出现次数,contextual weight 会考虑上下文。同一个词出现两次,两个位置的权重可能不同。
Tokenise the query without special tokens, remove its stop IDs, and locate equal document token IDs. Use the maximum document weight for each matched query ID; missing matches contribute zero. The supplied loop iterates query tokens, so repeated query IDs are counted repeatedly unless explicitly deduplicated.匹配的是相同 token ID,不是两个 dense vectors 的语义相似度。文档重复出现取 max;查询重复是否累计,要看具体循环,不能想当然。
BM25 provides candidate IDs and raw text. TILDEv2 supplies a new score for the same candidates. Keep the candidate set fixed for a reranking comparison. noExp identifies the no-expansion variant: contextual weighting alone does not invent every missing matching term.noExp 不表示没有训练,也不表示停用词。没有 expansion 时,查询词未出现在文档 token 中通常就没有对应匹配贡献。
For inspection, convert the small sparse output with to_dense(), count nonzeros and decode the top-10 weighted vocabulary IDs. A nonzero term absent from the original text demonstrates expansion. A sparse dot product compares query and passage on shared vocabulary dimensions.多数维度为零但维度仍对应词表。只为小例子 densify,不能因此在真实全库检索时把所有稀疏向量都展开存储。
For one query term, document occurrence weights [0.2,0.8] contribute 0.8, not 1.0. A second matching term of weight 0.4 produces total 1.2. By contrast, SPLADE scoring multiplies query and document weights in shared vocabulary dimensions and sums them.TILDEv2 的逐词 max 权重与 SPLADE 的稀疏点积是不同规则;不要因为都叫 sparse 就套用同一评分函数。
The local modeling.py and checkpoint interface may differ. Inspect the returned type and batch nesting, then match the course helper. Do not install an unrelated package named modeling or remove a dimension merely to suppress the exception.先检查真实返回结构以及 modeling.py 的版本。expected 2, got 1 可能涉及 batch 包装,不一定表示模型只输出一种信息。
Source map: prac-week6.ipynb: Exercise 1 TILDEv2, Exercise 2 SPLADEv2. Independent study explanation; original files remain on the course platform.
Back to Week 6 lecture conceptsWeeks 1–6 mostly ask how to produce a ranking. Week 7 asks how to learn and evaluate when feedback comes from users rather than complete qrels. 中文重点:点击很多,但点击不等于相关。
BM25 / dense / learned sparse returns candidates and retrieval scores.
BM25 score, neural score, freshness, authority, query-document features.
Learns a function that orders candidates to maximise a ranking objective.
A retrieval model can itself be learned, but Learning to Rank (LTR) usually means learning how to combine signals and order an already manageable candidate set. retrieval 解决规模,LTR 解决候选内部的顺序。
三者区别是训练目标的单位:单文档、文档对、整张列表。
Qrels are explicit, relatively clean labels but expensive, static, and limited. Clicks are abundant and current but implicit, noisy, and biased by what was shown and where.
随机噪声可能随数据增多而抵消;系统性 bias 不会自动消失。
A click requires both examination o and attraction/relevance evidence y. The propensity P(o=1|rank) captures how likely a position is viewed.
The naive estimator overvalues results placed where users look more often. Inverse Propensity Scoring (IPS) divides each click by its examination probability, giving rare observations more weight and cancelling position bias in expectation when assumptions are correct.
1/propensity 放大这类样本。代价是 propensity 很小时方差会变大。Maintain a scan pointer for each ranked list. Inspect the list with the smaller pointer, breaking ties with a randomly chosen starting ranker. Advance the pointer even for duplicates, but append only unseen documents. This balances inspected depths, not necessarily the number contributed by each ranker.
Advantage: both systems face nearly the same query, user and context. 比两组独立 A/B 流量更敏感。
Dueling Bandit Gradient Descent samples a candidate parameter direction, compares the current and candidate rankers through interleaving, and moves toward the winner. Repeated duels estimate a useful gradient from user satisfaction.
The deployed stochastic ranker logs displayed actions and their probabilities. The same interaction log is then reweighted with IPS to estimate the risk of many candidate rankers. A candidate replaces the current model when its estimated counterfactual risk is lower.
Sample rankings and record propensities.
Displayed documents, ranks, clicks/non-click losses.
Estimate current and candidate risks from the same log.
Main advantage over regular interleaving: one logged randomized policy can evaluate multiple candidate rankers off-policy, instead of constructing a fresh interleaved list for every pairwise comparison. 更能复用历史数据,但依赖可靠 propensity、充分 exploration/support,并可能有高方差。
| Method | Data source | What it estimates | Strength | Main limitation |
|---|---|---|---|---|
| Offline qrel evaluation | queries + qrels + runs | MAP/nDCG/etc. | controlled and repeatable | labels expensive and may become stale |
| Balanced interleaving | live clicks on one mixed list | pairwise ranker preference | sensitive within-user comparison | fairness/correctness is not guaranteed in every case |
| DBGD | repeated interleaving outcomes | improving parameter direction | learns directly online | exploration may expose poorer rankings |
| Counterfactual + IPS | logged clicks + propensities | off-policy candidate value/risk | reuses logs; compares many candidates | propensity errors, support gaps, high variance |
Contrast offline Learning to Rank (LTR) with Online Learning to Rank (OLTR). Logistic regression learns from explicit labels; Duelling Bandit Gradient Descent (DBGD) uses comparisons inferred from clicks.同样是更新排序器,前者直接看到训练标签,后者只根据交互比较来决定更新方向。模拟器使用 qrels 不等于 ranker 直接读取答案。
Data flow: MQ2007 features → logistic relevance scores → query-wise ranking; candidate ranker → interleave → simulated clicks → preference → update
Each MQ2007 row is a query–document pair with 46 features. The pointwise exercise maps labels 1 and 2 to 1. Preserve qid groups for ranking and evaluation; confirm the final query group is saved after parsing.输入不是原始文档文字,而是已经计算好的 46 维特征。二元化会丢失 1 和 2 的相关等级差异,需要说明评价使用哪套标签。
Compute p=sigmoid(Xθ), gradient Xᵀ(p−y)/m and θ_new=θ−α·gradient. Use the same old weights for every gradient component. Ranking by θᵀx or its sigmoid is equivalent because sigmoid is monotone.梯度衡量预测和标签的差异如何分摊到每个特征权重。先算全部梯度再更新,避免前面刚更新的权重污染后面分量。
Keep a pointer into each original ranking. Inspect the shallower pointer; use one random starting choice to break ties. Advance the pointer even when a duplicate is skipped. This balances inspected depths, not strict alternation of successful insertions.重复文档只展示一次,但扫描位置仍要前进。不能用“每边轮流成功加入一篇”替代老师的 pointer 算法。
If clicks exist, take the deepest displayed clicked document. Its better rank in the two original lists determines a common prefix cutoff. Count clicked IDs in each prefix; the exploratory list wins only if its count is larger. No clicks or a tie do not establish a winner.clicked_doc 必须保持显示顺序。截断位置是同一个值,再分别计票;不能仅按“哪边贡献了被点文档”来判断 balanced interleaving。
DBGD samples direction u and updates toward it only when the challenger wins. Compare perfect, navigational and informational click models using fixed settings and multiple seeds. The “perfect” simulator still has a probabilistic click for relevance grade 1; its name does not mean every relevant item is always clicked.单次 nDCG 曲线不保证每一步上升。区分训练目标、在线点击结果和测试 nDCG;测试标签用于监测,不应决定训练时的胜负。
A=[a,b,c], B=[b,c,a], displayed clicks=[b,c]. The deepest click is c; its better original rank is 2. A’s top two receive one click and B’s top two receive two, so B wins. Inspect the helper’s random draw: np.random.randint(0,1) always returns 0 because the upper bound is exclusive; a two-outcome draw needs an upper bound of 2.这也是阅读老师提供的代码时应核验的边界:函数名和注释不保证实际随机。这里是学习提示,网站没有改动归档的老师原文件。
A small weight update can preserve every document’s ordering. Check weight differences and the training loss before diagnosing a frozen model. Conversely, an unimplemented update also leaves scores unchanged; inspect the computation, not only the plot.排序没变和参数没变不是一回事。先确认权重、损失和排名分别是否变化。
Source map: prac-week7.ipynb: pointwise Tasks 1–2 and online Tasks 1–2; scripts/ClickModel.py and InterLeaving.py. Independent study explanation; original files remain on the course platform.
Back to Week 7 lecture conceptsWeek 8 moves from retrieving candidates to applying deeper query-document interaction. Encoder rerankers classify relevance; decoder-based Large Language Models (LLMs) can judge, compare or order passages through prompts. 核心关系:第一阶段保证候选覆盖,第二阶段用更昂贵的模型改善候选内部顺序。
BM25 or another efficient retriever searches the full collection and returns top-k candidates.
A cross-encoder jointly reads the query and each candidate, or an LLM receives them in a ranking prompt.
Sort by neural relevance scores, pairwise preferences or an output ordering.
Effectiveness-efficiency trade-off: deeper interaction usually improves ranking quality but increases latency. Reducing candidate count lowers cost, but a relevant document omitted by Stage 1 cannot be recovered by the reranker. reranker 只能重新排列已召回文档,不能救回没有进入候选集的文档。
Bidirectional Encoder Representations from Transformers (BERT) receives [CLS] query [SEP] document [SEP]. The contextual [CLS] vector enters a classification layer that estimates whether the document is relevant.
Each candidate is scored independently, so monoBERT is pointwise. Training uses human positive judgements and commonly samples difficult negatives from BM25.
This is binary cross-entropy: reward high scores for relevant passages and low scores for negative passages. The score is useful for ordering candidates, but it should not automatically be interpreted as a perfectly calibrated real-world probability.
训练单位是一组 query-document pair;模型学的是相关/不相关分类信号。
| Family | What is divided | How evidence is combined | Main idea / 中文抓手 |
|---|---|---|---|
| BERT-FirstP | Document → passages | Use the first passage score | Cheap, but later evidence is ignored / 只看首段 |
| BERT-MaxP | Document → passages | Take the maximum passage score | One strong passage can represent the document / 取最强证据 |
| BERT-SumP | Document → passages | Sum passage scores | Accumulates evidence but may favour many passages / 累加证据 |
| Birch | Document → sentences | Interpolate first-stage score and weighted sentence scores | Combines retrieval and sentence evidence / 句子级聚合 |
| PARADE | Document → passage representations | Average, max, attention or Transformer aggregation | Combine representations before the final relevance decision / 先聚合表示再评分 |
After monoBERT narrows the list, duoBERT jointly reads the query and two documents and estimates p(dᵢ > dⱼ | q). It therefore learns a pairwise preference rather than an independent relevance score.
Aggregating pairwise wins produces the final order. More pair comparisons may improve quality, but encoder cost grows quickly.
BM25 → monoBERT → duoBERT exposes several tuning knobs: candidate depth at each stage, model size and number of pair comparisons. These provide a flexible quality-latency trade-off, but also make the system more complex to tune and operate.
候选越少越快,但召回风险越大;候选越多、比较越细,成本越高。
| Family | Prompt input | Ranking signal | Cost and limitation |
|---|---|---|---|
| Pointwise | Query + one passage | Yes/No logits, generated label or relevance grade | Simple and parallelisable; absolute judgements may be poorly calibrated |
| Pairwise | Query + passage A + passage B | Preference A > B | Robust comparison, but many pairs require many inferences |
| Listwise | Query + a list of passages | Generate a permutation or ordered identifiers | Sees list context; generation and context length are costly |
| Setwise | Query + a small set of passages | Select the best passage using output logits | Fewer comparisons than pairwise and avoids full list generation |
Zero-shot in this lecture: once an instruction-tuned LLM is available, these ranking prompts do not require new contrastive retriever training. This does not mean the underlying LLM was never pretrained or instruction-tuned.
Pairwise ranking compares two documents per call. Setwise ranking selects a winner from several documents at once, reducing the number of LLM inferences. Compared with listwise generation, setwise methods can use next-token logits instead of generating a complete permutation.
Role instructions, output restrictions, formatting instructions and query/passage order can all change effectiveness. The slides report larger variation for pointwise and listwise prompts, with pairwise and setwise prompts generally more robust; behaviour also depends on model size, architecture and training.
Use Pyserini to retrieve a small top-k and read each hit's raw passage text.
Tokenize each query-passage pair, obtain the relevant-class softmax score and sort descending.
Write the reranked document IDs and compare the run with qrels.
The practical recommends beginning with only 10 or 20 candidates because monoBERT inference is expensive. The updated LLM section compares TinyLlama, MiniCPM and Qwen3-Reranker before the pairwise exercise. 原始 Yes token 概率与 Yes/No 归一化分数含义不同;比较排序而非跨模型分数大小,详细步骤见本周 Practical。
Follow the same candidate pool through an encoder cross-encoder and decoder-based relevance scoring. Distinguish a score used for ordering from a calibrated probability of relevance.本地旧版缺少 MiniCPM 与 Qwen3 的后续比较;本次已按老师更新版补齐。重点不是比较不同模型分数谁更大,而是它们把哪些文档放到前面。
Data flow: BM25 top-k IDs → raw text → query–passage pair → relevance score → sorted run → same qrels
BERT jointly encodes the query and one passage; a classification head supplies a relevant-class score. Start with 10 or 20 candidates as instructed. Sort by monoBERT, not by the old BM25 score, before writing the run.Cross-encoder 允许 query 与 document tokens 在网络中交互,因此不能像 bi-encoder 那样预先得到一个与 query 无关的最终文档得分。
TinyLlama and MiniCPM use the probability of a selected Yes token after a fixed prompt. Each model has its own tokenizer and chat template. Check whether Yes or leading-space Yes is one token and matches the answer format; keep the choice fixed across passages.相同字符串在不同 tokenizer 中可以对应不同 ID。必须先确定候选 token,不能看每篇结果再挑有利的 Yes 写法。多 token 候选需要完整序列概率。
The next-token distribution must come from the final real prompt position. The updated example scores passages individually to avoid right-padding mistakes. Passing an attention mask alone does not make logits[:,−1] point to the last real token of every right-padded example.最后一个数组位置可能是 PAD。模型设备、tokenizer、attention mask 和预测位置都需要对齐,才能比较输入的真实分数。
Qwen3-Reranker uses its task-specific format and designated yes/no logits. The notebook’s interface describes a logit difference z_yes−z_no; sigmoid maps that difference to exp(z_yes)/(exp(z_yes)+exp(z_no)) without changing order. Check the installed interface’s output convention.Yes/No 两项归一化不是全词表里某个 Yes 的概率。训练任务与模板也不同;归一化本身不保证模型判断更正确。
Provide the question and two labeled passages and request a preference. Reverse A/B positions and map the result back to the same document IDs to examine position bias. For many candidates, define ties and aggregation; pairwise preferences may be cyclic.pointwise 是每篇独立评分;pairwise 是两篇同时比较。不要把 A/B 标签当固定 docid,也不要默认所有比较都满足传递性。
Toy raw probabilities: passage A has P(Yes)=0.02 and P(No)=0.001; B has 0.40 and 0.10. Raw Yes prefers B, but Yes/No normalisation gives A≈0.952 and B=0.800, reversing the order. A’s high normalised score hides that both selected tokens had little total probability.归一化分母因 passage 而异,所以排名可能反转。归一化值很高,不代表模型大概率真的会生成 Yes,也不能当人工 gold。
The model, prompt, evidence, answer-token choices and generation/scoring position. Change only raw-versus-normalised scoring. Compare ranked relevance against judgments rather than comparing raw values across models.控制同一模型、同一输入与同一 token,只改变评分规则,才能讨论归一化的影响。
Source map: prac-week8.ipynb: BM25 + monoBERT, TinyLlama, MiniCPM, Qwen3-Reranker, optional scoring comparison, pairwise exercise; refreshed 16 October 2026. Independent study explanation; original files remain on the course platform.
Back to Week 8 lecture conceptsWeek 9 connects two directions: LLM4IR uses a Large Language Model to retrieve or rank; IR4LLM retrieves external evidence for generation. The lecture then asks when long context can replace retrieval, and why relevant evidence still may not produce a correct, grounded answer.
BM25, DPR, TILDEv2, SPLADE, CiC or a specialist retriever find candidate evidence.
Rerank, rewrite, filter, summarise, fuse or compress before the prompt.
A frozen or adapted LLM answers, while citations and groundedness checks connect claims back to evidence.
关键区别:retrieval quality asks whether useful passages are ranked highly; answer quality asks whether the generator uses them correctly. A better nDCG score can help RAG, but it does not logically guarantee a better answer.
Zero-shot gives instructions only; one-shot/few-shot add demonstrations; Chain-of-Thought (CoT) elicits intermediate reasoning tokens. Reasoning text is still generated output, not a transparent record of the model's internal computation.
示例改变的是上下文中的任务说明与模式,不等于重新训练模型参数。
An LLM can generate a dense/sparse representation, judge one passage, compare two passages, order a list, or select from a set. RepLLaMA and LLM2Vec produce dense representations; PromptReps and DiffRetriever explore alternative generated representations; pointwise/pairwise/listwise/setwise prompts produce ranking signals.
| Dimension | LC-LLM / Corpus-in-Context | Retrieval-Augmented Generation |
|---|---|---|
| Input | Put a very large corpus directly in context | Retrieve a small top-k evidence set |
| Main benefit | One consolidated model call; fewer pipeline boundaries | Scales to larger, changing corpora; lower prompt size |
| Main cost | Attention cost and long-prompt latency/memory | Retriever/reranker maintenance and cascading errors |
| Failure mode | Position bias and lost-in-the-middle | Missing, distracting or conflicting retrieved evidence |
| When attractive | Corpus fits context and repeated prefixes can be cached | Corpus is large, dynamic, or needs explicit evidence IDs |
Corpus-in-Context (CiC, pronounced “seek”) assigns consistent document identifiers, places the corpus and corpus-grounded demonstrations in one prompt, and asks the LC-LLM to return document IDs. Prefix caching can encode a repeated corpus prefix once, but it does not remove positional bias.
few-shot 示例数量增加可能改善效果,但 gold document 越靠后,检索质量可能下降。
Screenshot retrievers encode rendered pages with a Vision-Language Model (VLM), preserving layout, figures and spatial signals lost by plain text extraction. DiffRetriever uses a diffusion language model to produce several representative tokens in parallel, then matches multiple query/document representations.
RAG supplies knowledge that is newer, private, domain-specific or absent from pretrained weights. It differs from further language-model pretraining: the external collection remains separately updateable and the retrieved evidence can be inspected.
| Stage | Methods | What they change | Risk |
|---|---|---|---|
| Pre-retrieval | routing, rewriting, expansion, HyDE | make the query easier for the index to match | rewrite may drift from the original need |
| Retrieval | BM25, dense, learned sparse, hybrid | choose the candidate evidence set | relevant evidence may never enter top-k |
| Post-retrieval | reranking, filtering, fusion | improve order and remove distractors | extra latency; filter can remove useful evidence |
| Prompt construction | summarisation, lexical or embedding compression | fit more useful evidence in a fixed budget | compression can delete qualifiers or provenance |
| Generation | read, cite, revise, Self-RAG | use evidence and decide whether to retrieve again | model knowledge may override supplied evidence |
A rewriter converts an ambiguous or underspecified question into a search-friendly query, retrieval supplies evidence, and a reader answers. The lecture's trainable variant warms up on pseudo rewrites, keeps useful rewrites, then optimises the rewriter with a reward while constraining divergence from the initial policy.
它优化的是“送给 retriever 的 query”,不是直接修改 documents。
BlendFilter combines retrieved knowledge with model-generated query/answer signals, then prompts an LLM to filter irrelevant passages before answer generation. It illustrates that retrieval is not the end of the pipeline: evidence selection can itself be an LLM task.
Lexical compression removes low-information tokens using signals such as entropy or perplexity. Embedding compression maps a long context into learned compact memory tokens. Both trade fidelity for fewer input tokens, lower latency and less distraction.
A model may preserve an initially wrong answer even after receiving supporting correction, or flip a correct answer after contradictory evidence. Therefore “retrieved passage is relevant” and “generated answer follows evidence” are separate measurements.
这正是 Project 2 同时要求 reference-based answer score 与 evidence groundedness 的原因。
| Observation | Interpretation | Design response |
|---|---|---|
| One high-scoring distractor can sharply reduce answer quality | semantic similarity is not the same as answer-supporting evidence | rerank/filter for utility, not topic overlap alone |
| More distractors progressively degrade performance | longer context creates more competing signals | control top-k and evidence token budget |
| Gold evidence in the middle may be used least | long-context attention has positional bias | test order; place or repeat critical evidence carefully |
| Random passages sometimes improve some models | LLM behaviour depends on model and context distribution | treat it as an empirical finding, not a universal recipe |
Attribution is the ability to provide references or citations that support generated claims. Directly generated citations can be fabricated; retrieval-based attribution constrains citation candidates to real documents.
Retrofit Attribution using Research and Revision (RARR) researches evidence for an existing answer and revises unsupported content while preserving the original response as much as possible. It is a post-generation attribution-and-repair pipeline.
Use Information Retrieval for Large Language Models (IR4LLM), reversing Week 8’s LLM4IR role. Retrieval-Augmented Generation (RAG) passes evidence to a generator; retrieval quality and answer quality remain separate measurements.Week 8 让 LLM 帮助排序,Week 9 让检索帮助 LLM 回答。LLM 接收的是 query 与证据文本,不是把 nDCG 分数当上下文。
Data flow: question + retrieved documents → labeled context → instruction/chat template → tokens → generated continuation → claim-by-claim checking
Join the provided documents with labels, include the question, and ask for an answer supported only by that evidence. Tokenise the model-specific chat template, generate, then decode only output_ids after the input length. Keep model and input tensors on the same device.函数的关键是把证据实际放进 prompt。只写“请根据资料回答”但没有传入资料并不是 RAG。切掉输入 tokens,避免把 prompt 当生成答案。
The notebook supplies two weather snippets as example evidence. It does not implement a live weather fetch. The example teaches how external context can change an answer; its quoted temperature is not today’s verified forecast.示例文本不等于实时数据。RAG 能使用更新资料,但资料是否真的更新、来自哪里,需要检索和来源检查来保证。
The four fictional documents cover prices, caffeine, milk and a reusable-cup discount. A correct supported answer must satisfy every constraint and cite the relevant documents. Check arithmetic as well as the selected drink.答案需要综合四段信息:低咖啡因、无乳制品和预算缺一不可。只找到相关菜单仍不代表推理和计算正确。
Compare zero-shot instructions, few-shot demonstrations and a prompt requesting an explicit evidence check while holding the documents and generation settings fixed. Keep demonstrations separate from the actual café answer so examples do not leak the solution.Zero-shot 没有示范,few-shot 提供其他示范;要求说明计算可以帮助检查答案,但生成的解释不自动等于可靠的内部过程记录。
First compare prompts with native thinking off. Then hold the zero-shot prompt fixed and toggle thinking for MiniCPM5/Qwen3. Record correctness, cited support, output tokens and whether the answer finishes. Thinking tokens consume the same bounded output budget.不要同时换 prompt、模型和 thinking,再把差异归因给一个因素。被 token 上限截断的解释不算完成答案,单个小例子也不能证明某模型普遍更好。
The supplied fictional evidence implies a decaf latte with oat milk: $4.80 + $0.70 − $0.50 = $5.00, and 5 mg caffeine ≤10 mg. D1 supports the base price, D2 the caffeine, D3 the milk substitution/surcharge, and D4 the discount. This is a hand-derived evidence check, not a reported model run.人工核验答案:燕麦奶低因拿铁,刚好 5 美元。四条文档分别支持不同事实;这不是声称模型已经跑出了正确结果。
The remaining evidence no longer establishes a dairy-free option or its surcharge. A grounded response should state the missing information instead of recalling the removed fact as if it were still supported.缺了 D3 就无法从现有证据确认无奶选项及加价。即使模型记得答案,也不能把记忆冒充这次检索到的依据。
Source map: prac-week9.ipynb: Exercise 1 rag_answer, Exercise 2 prompting strategies and thinking-mode comparison; downloaded 16 October 2026. Independent study explanation; original files remain on the course platform.
Back to Week 9 lecture conceptsWeek 9 studied how retrieved evidence enters a generator. Week 10 lets the system inspect an observation, decide whether evidence is missing, and choose another search or reading action before answering. The important change is a feedback loop, not merely adding more tools to a fixed RAG prompt.
| System | Decision sequence | What happens when evidence is insufficient? |
|---|---|---|
| Static Retrieval-Augmented Generation (RAG) | user query → retrieve once → put passages in prompt → generate | Usually answers from the initial set, even when it misses a needed link. |
| Search agent | goal → action → observation → updated state → next action or finish | Can reformulate a query, inspect a candidate, read more lines, or stop at a budget. |
Agent means a goal-directed system interacting with an environment through an observe–think–act–update loop. Its policy chooses the next action from the current state; a tool executes that action; an observation is the returned result. ReAct (Reasoning and Acting) interleaves reasoning and actions rather than writing one complete answer before looking again.
例如先确认“大学创立于哪座城市”,再查“哪条河穿过该城市”。第一条证据只能支持城市,不能自动支持河流;第二次搜索要补上缺失的一跳。
The state can contain the original question, past queries, visited document IDs, tool observations and remaining budget. The policy should stop when the answer is supported by the evidence, when another search has low expected value, or when a step/token limit is reached. A plausible answer is not itself proof that the required sources were read.
The baseline pattern searches an index, then opens a promising result. It is useful when a short page or passage probably contains the answer. It becomes wasteful when a hit is a huge report: visiting the whole page consumes context before the agent knows where to look.
先找候选,再打开阅读;容易理解,但长文档会带来大量无关文本。
| Method and full name | Action pattern | Best fit | Trade-off |
|---|---|---|---|
| Search–Visit | rank results → open page | one short page likely answers the question | whole-page visits may waste tokens |
| SIEVE, Search–Inspect–Fetch | search → inspect headings/structure → fetch relevant section | one long report with clear sections | depends on useful structure; extra inspection step |
| DCI, Direct Corpus Interaction | grep raw files → read nearby lines | manageable local files and a known exact term/name | lexical mismatch and broad scans across large corpora |
| RISE, Retrieving Interaction SpacE | BM25 narrows files → grep/line navigation verifies | large file collection, keyword known but file unknown | retriever can omit the right file; still requires reading |
Why RISE is not simply DCI: DCI can search raw files directly; RISE puts a ranked candidate-selection step in front of the file tools. Why SIEVE is not simply Search–Visit: the agent inspects document structure before fetching only the relevant part. These are interfaces and workflows, not interchangeable ranking models.
An agent can recover from a weak first search only if useful evidence remains reachable within its budget. BM25 ranks lexical matches; dense and learned sparse retrievers can address vocabulary mismatch; a reranker can reorder a fixed candidate pool. The lecture compares results across document granularities and agent settings, so no single model is universally best.
评价不能只看离线 nDCG:还要看是否找到支撑答案的证据、用了多少次搜索和多少 tokens,以及最终答案是否 grounded。
Traditional retrieval scores a document mainly from the user's original query. During a trajectory, the next useful document also depends on what the agent has already learned. A previously visited relevant passage may still match the query but add almost no new information. Marginal evidence gain asks what this document adds now.
| Approach | Query-side context | Training signal | Key caution |
|---|---|---|---|
| AgentIR, Reasoning-Aware Retrieval for Deep Research Agents | agent query plus its pre-search reasoning | QA with gold documents; costly reranking gives positive and negative passages for intermediate queries | Gold labels and an external reranker are costly; a snippet may omit the needed evidence. |
| LRAT, Learning to Retrieve from Agent Trajectories | queries produced during successful trajectories | judge visited passages for positives; no gold-document set required | A visit alone is not proof of usefulness; treating all others as negatives can be noisy. |
| ITER, Interaction-Aware Retrieval for Agentic Search | current query + original question + previous queries | trajectory-relative judgments distinguish new evidence from already visited relevant or irrelevant documents | Selection bias and noisy inferred labels still need care. |
Connection to Week 5: all three still need informative positives and negatives for retriever training. The difference is where supervision comes from and whether the query representation includes interaction history. Connection to Week 7: successful observed trajectories are biased by what the old policy exposed; unvisited documents are not automatically irrelevant.
grep(pattern) finds a first matching line per document; BM25 search(query) returns ranked candidate IDs.
A CrossEncoder reranks the same BM25 pool; read(id, start_line, num_lines) opens numbered lines.
The tool result enters the agent history; the model chooses another action or a cited answer within its budget.
The exercise asks you to complete read line slicing and search query/passage scoring. Its small dataset has relevance judgments for inspection, but the agent should not see those labels. A grep match is only a location hint; a BM25 or reranker score is only a ranking signal; neither is verified answer evidence until the passage is read.
Worked example: the colour-afterimage question requires an explanation of adaptation in the visual system. Compare the candidate that merely names afterimages with the passage that explains retinal/neural adaptation, then inspect which lines actually support the final claim. Optional Qwen3 thinking text is not a tool observation or an independently verified source.
Research trail / 论文线索:ReAct · SIEVE · DCI · RISE · AgentIR · LRAT · ITER. These public papers support further reading; course PDFs and code remain on the restricted course platform.
Trace a ReAct (Reasoning and Acting)-inspired agent: the policy selects an action, Python validates and executes it, and an observation updates the state. Complete small tools rather than treating the agent as one opaque model.这是之前只简略整理过的 practical。本次补齐工具输入输出、边界测试、预算与失败分析,并连接 Week 9 的静态 RAG。
Data flow: question → policy → search/grep/read → tool observation → updated messages → another action or finish
The policy is the model’s action choice; tools are Python functions; observations are their returned text/errors; state is the accumulated messages and allowed IDs; a trajectory is the sequence of actions and observations. Qrels stay outside the agent prompt.LLM 负责选择,Python 负责执行与约束。一次工具返回才是可检查的 observation;模型自述的 thinking 不是外部证据。
Convert the public 1-based start_line to a 0-based slice start. Return up to num_lines, prefix each selected line with its original line number, and provide the next starting line only when unread lines remain. The tool accepts at most 12 lines.切片起点是 start_line−1。返回行号要从原文位置继续,不能每一页都从 1 编号;最后一页的 next_line 应为 None。
Retrieve eight BM25 candidates; build (query, passage) pairs; use reranker.predict; sort the paired candidate/score objects descending; return the best k (allowed 1–5). Sorting scores separately from candidates breaks the ID-to-score association.8 是重排候选深度,k 是返回数量。重排不能找回 BM25 没有选中的文档。保留 docid、text、score 的对应关系。
In Part 1, grep scans the ten-document sample. Part 2 explicitly requires search→grep→read: the latest search results constrain which documents grep/read may inspect. A grep match locates text; reading context establishes what it actually says.第二部分的访问范围是老师为了展示流程而设定,不代表所有 agent 必须用同一顺序。检索排名高和真正支持答案仍是两回事。
The classroom run caps decisions at 8, output at 512 tokens per decision and input at 8,192 tokens. Failed calls also spend steps. Inspect retrieval misses, malformed calls, repeated loops, budget stops and unsupported final claims separately.预算用完是停止原因,不是成功证据。finish 的引用 ID 已读,也不代表每一句答案都受支持;还要逐句对照引用行。
For lines [alpha,beta,gamma,delta], read(start_line=2,num_lines=2) should return “2: beta” and “3: gamma”, with next_line=4. Reading from 4 returns “4: delta” and next_line=None. Starting beyond the end should produce an empty window with no next page, under the supplied validation contract.这三个边界例子能检查 off-by-one、末页和越过末尾的行为。它们验证工具,不需要先花时间运行大型模型。
No. It demonstrates the mechanism. Compare the answer’s supported claims and token/time costs, then use broader evaluation for general conclusions. Optional thinking-on/off runs must use equal budgets and fixed questions/tools.一次成功轨迹是机制示范,不是大规模 benchmark。保留失败轨迹,才能区分检索、工具和生成环节的问题。
Source map: prac-week10.ipynb: Exercises 1 read, 2 search, comparison and optional thinking; search_agent.py and agent_display.py. Independent study explanation; original files remain on the course platform.
Back to Week 10 lecture concepts搜索引擎首先要发现网页,再把网页内容与链接图结合,最后避免 top results 只重复同一种解释。Week 11 把“收集什么、哪些页面更有权威、结果列表覆盖哪些意图”连成一个系统。
从 seed URLs 出发,从 frontier 取出允许访问且尚未处理的 URL。
请求页面、解析正文与链接、规范化 URL、识别重复内容。
建立内容索引与链接图,再综合 relevance、authority 与 diversity。
URL frontier 是已发现但等待抓取的 URL 集合。调度器要决定下一条 URL、host 访问间隔、何时重抓以保持 freshness,以及如何避开重复页面与 crawler traps。抓取范围受 seeds、链接可达性、访问规则、时间和存储限制,因此 crawl 不是完整复制整个 Web。
合理 crawler 要限制同一 host 的请求速率、标识自身、遵守适用规则,并处理 redirect、失败与缓存。robots.txt 表达 crawler access preference,不是认证机制,也不能用于保护私有内容。
网页是 node,hyperlink 是有方向的 edge。In-link 指向当前页面,out-link 从当前页面离开。Link analysis 可用于 ranking、link-based clustering、classification features 与 crawl prioritisation,也能发现与 spam 相关的异常链接分布。
课件区分两个假设:链接可能把 authority 赋予目标页;anchor text 可能描述目标页。索引页面 D 时,可把指向 D 的 anchor text 按权重加入表示,来源页 authority 也可影响权重。这能缓解 vocabulary mismatch,但会受到 spam 或协同锚文本操纵。
N 是页面数,L(u) 是页面 u 的 out-degree,d 是沿链接继续的概率;剩余概率用于 teleport。高 PageRank 页面传来的 link mass 更大,但要在它的所有 out-links 间分配。Dangling page 没有 out-link,需要明确重新分配,否则概率会丢失。
Power iteration 从一个概率向量开始,反复分配 link 与 teleport mass,直到变化足够小。PageRank 衡量的是图结构中的 importance,不是当前 query 的 relevance;实际排序可把它与 lexical、semantic、freshness 等信号结合。
| Retrieval model | How PageRank can enter | Caution |
|---|---|---|
| Language model | 用 PageRank 等 feature 构造 document prior P(D),替代 uniform prior | 理论上方便,不保证 effectiveness 上升 |
| BM25 | 把文档看成 text + non-text features,用调参或学习的 utility 组合 | 基础模型允许加 feature,但不会自动给出权重 |
| Vector Space Model | 在外部 utility 中组合 cosine、PageRank、spam score、difficulty | query-independent feature 不自然存在于 query vector |
PageRank、spamminess 与 URL depth 都可作为 document priors。它们是 feature,不是 relevance label;是否有帮助必须用目标指标实证评价。
jaguar 可能指动物、汽车品牌或球队,这叫 query ambiguity。即使用户意图相同,多篇近重复页面也会浪费前排位置,这叫 redundancy。Diversity 的目标是保留 relevance 的同时覆盖合理 aspects,不是让每条结果为了“不同”而不同。
Maximal Marginal Relevance (MMR) 从剩余候选 R 中逐个选下一篇,S 是已选集合。较大 λ 更重 query relevance;较小 λ 更强地惩罚与已选结果相似。Document similarity 与 relevance score 的尺度必须可比较。
Probability Ranking Principle (PRP) 按估计相关概率降序排列。它的简单 next-document rule 假设概率估计准确/确定,并假设当前文档 relevance 可独立于已排序文档判断。Ambiguity 挑战前一个假设,redundancy 挑战后一个假设;diversification 改为评估候选在当前列表中的 marginal value。
| List objective | Question | Failure if used alone |
|---|---|---|
| Relevance | 文档对 query 有多有用? | 可能重复一个 interpretation |
| Coverage | 覆盖哪些合理 intents/subtopics? | 可能牺牲过多 relevance |
| Novelty | 这条结果增加什么新信息? | 仅仅不同不一定有用 |
Diversity 不只是一条 reranking 公式。Query-ambiguity detection 判断是否存在多意图;query-aspect mining 产生候选意图;vertical diversification 可以混合不同结果类型。评价可使用 subtopic recall/precision/MRR、α-nDCG、intent-aware nDCG 或 intent-aware ERR。要明确 intent judgments 与 metric convention;普通单标签 nDCG 不能完整衡量 coverage + novelty。
Public reading / 公开延伸阅读:Stanford IR book: web crawling · Stanford IR book: PageRank · RFC 9309: Robots Exclusion Protocol · MMR paper。Week 11 已发布课件用于确认课程具体范围;受限课程文件不在本站托管。
The course playlist has 10 videos, totalling 28 min 36 sec. Links and chapter titles were checked on 13 October 2026; the explanations below were checked against the 40-page Week 11 lecture PDF. They are study notes, not video transcripts. 视频入口与标题已核对;以下讲解对照本地 40 页讲义整理,不冒充视频逐字稿。
Open the complete Week 11 playlist · Course announcement (login required)
| Chapter | Duration | Learning checkpoint |
|---|---|---|
| 11.1 Overview | 2:22 | Place graph authority and list diversity beside content relevance. 先区分内容相关性、链接权威度和列表多样性。 |
| 11.2 Links | 3:15 | Distinguish an endorsement from descriptive anchor text. 同一个链接可以提供权威信号和锚文本,二者不是同一数据。 |
| 11.3 Link analysis in IR | 3:18 | Explain why links help ranking, clustering, classification and crawling. 链接不仅用于排名,也用于聚类、分类和抓取调度。 |
| 11.4 PageRank | 4:41 | Trace one random-surfer step and the role of teleportation. 跟踪一次随机游走,理解为什么需要随机跳转。 |
| 11.5 Understanding PageRank | 1:56 | Use source authority and out-degree, not just the count of incoming links. 来源权威度和出链数量共同决定每条链接传来的权重。 |
| 11.6 PageRank in retrieval | 2:06 | Separate a document prior from the query-dependent matching score. 把文档先验与当前 query 的匹配分数分开理解。 |
| 11.7 Diversification | 1:17 | Explain why a relevant duplicate may add little value to a list. 单篇相关不等于加入列表以后仍有同样大的价值。 |
| 11.8 PRP | 3:22 | Expand Probability Ranking Principle and state the lecture's assumptions. 说出概率排序原则的全称,并解释课件中的两个假设。 |
| 11.9 Ambiguity and redundancy | 0:48 | Locate ambiguity in the query and redundancy in the result list. 歧义主要在 query,冗余主要在结果列表。 |
| 11.10 Diversity framework and MMR | 5:31 | Compute the next result's relevance-minus-redundancy score. 计算下一篇文档带来的相关性收益与重复惩罚。 |
PDF route: pp. 3–8 links and anchors; pp. 9–19 random surfer and PageRank; pp. 20–24 retrieval priors; pp. 25–36 diversity and PRP; pp. 37–40 diversification and MMR. The crawling/robots background above is a supporting extension, not the central scope of this PDF. 回看老师讲义时按这条路径定位。网页抓取和 robots 是补充背景,本周讲义主线是链接分析与多样化排序。
| Method and full name | Inputs → output | What changes the score? | Not a substitute for |
|---|---|---|---|
| BM25 (Best Matching 25); DPR (Dense Passage Retrieval) | query + document representation → matching score | query words or meaning | authority or a diversity objective |
| monoBERT (BERT: Bidirectional Encoder Representations from Transformers) | one query–document pair → relevance score | joint contextual matching | comparison with all already selected documents |
| PageRank (a model name, not an acronym) | directed link graph + teleport rule → one authority value per page | links, graph boundaries and teleport distribution | query relevance or factual verification |
| PRP (Probability Ranking Principle) | estimated relevance probabilities → descending order | the relevance estimates | a trained encoder or a redundancy penalty |
| MMR (Maximal Marginal Relevance) | candidate relevance + document similarity + selected set → next item | query AND what is already selected | first-stage retrieval or ground-truth labels |
Think of BM25/DPR as matching functions, PageRank as a graph feature, PRP as an ordering principle, and MMR as a set-aware selection rule. They can occupy different parts of the same pipeline; they are not five interchangeable neural models. 先判断它属于模型、特征、原则还是选取规则。PageRank 高不代表这条 query 相关;MMR 分数低也不代表文档不相关,可能只是已选结果已覆盖了它。
Original worked example: three pages with links A → B, C, B → C, and C → no out-links. Start at r₀ = (1/3, 1/3, 1/3). Let d = 0.85 mean follow-link probability and use uniform teleportation and uniform dangling-mass redistribution. 这是为复习新增的小例子,不是老师提供的实验结果。A 的权重要分成两份;C 没有出链,所以它的权重均匀分给所有页面。
| Target | Teleport | Dangling C contribution | Incoming link contribution | After one iteration |
|---|---|---|---|---|
| A | 0.15/3 = 0.05 | 0.85 × (1/3)/3 = 0.094444… | 0 | 0.144444… |
| B | 0.05 | 0.094444… | 0.85 × (1/3)/2 = 0.141667… | 0.286111… |
| C | 0.05 | 0.094444… | 0.85 × [(1/3)/2 + 1/3] = 0.425 | 0.569444… |
Check: the new entries sum to 1. This is one iteration, not the final stationary distribution. Compute every new entry from the same old vector, then replace the entire vector. Repeat until a stated tolerance such as Σ |r_new − r_old| < ε is met. 不要算出 A 的新值后马上用它更新 B;一次迭代应统一使用旧向量。收敛后才能解释为长期访问概率。
Without the dangling term, the next vector sums to 0.15 + 0.85 × 2/3 = 0.716667…. The missing mass is 0.85/3. A dangling page has no outgoing edge; a closed group may have outgoing edges internally but no exit. Teleportation also gives a way to escape such closed groups. 漏掉 dangling mass 会丢概率;不能把“权重变少了”当成正常收敛。无出链节点与封闭链接团体相关但不是同一种结构。
Notation trap: lecture p. 15 uses d for link-following, while p. 17 writes 1−λ for the same role. MMR later uses λ for relevance weight. Read the definition, not just the letter. 同一个 λ 在不同公式中不是同一个参数。PageRank 的跳转概率和 MMR 的相关性权重没有直接等同关系。
Here the language model is the query-likelihood retrieval model, not necessarily a generative LLM. P(q|D) models the query from the document; P(D) favours documents before inspecting this query. A positive, normalised PageRank distribution can supply the prior. 这里的 Language Model 是传统查询似然检索模型,不要自动理解成 ChatGPT。先验表示在看 query 前对文档的偏好。
Toy comparison: A has P(q|A)=0.02, P(A)=0.60; B has P(q|B)=0.08, P(B)=0.10. Query likelihood alone favours B; products 0.012 > 0.008 favour A. These products are unnormalised ranking values, not calibrated answer-correctness probabilities. The changed ranking may help or hurt. 例子说明权威先验可以改变排序,不是证明改后一定更好。基础 BM25 也不会因为算出了 PageRank 就自动使用它;需要明确组合方式、量纲与验证方案。
Lecture p. 39 supplies relevance values A=0.8, B=0.6, C=0.5 and similarities sim(A,B)=0.9, sim(A,C)=0.1. A and B cover the same interpretation; C covers another plausible one. Start by selecting highest-relevance A. We add the explicit choice λ=0.5 to make the arithmetic reproducible. 老师图中给出这组分数;这里补上 λ=0.5 的计算假设。已经选择 A,接下来只比较 B 与 C 的新增价值。
| Candidate | Relevance reward | Redundancy penalty against A | MMR at λ=0.5 |
|---|---|---|---|
| B | 0.5 × 0.6 = 0.30 | 0.5 × 0.9 = 0.45 | −0.15 |
| C | 0.5 × 0.5 = 0.25 | 0.5 × 0.1 = 0.05 | 0.20 |
Relevance-only order: A, B, C. With this MMR setting: A, C, B. A negative MMR value is allowed because it is a selection utility, not a probability. No new document has been retrieved: the candidate set is unchanged. MMR 只是重排已有候选,不能找回第一阶段没有召回的文档。负数表示此刻重复惩罚超过相关性收益,不表示“负相关概率”。
A already selected / 已选 A · B = −0.150; C = 0.200; A → C → B
B wins when 0.6λ−0.9(1−λ) > 0.5λ−0.1(1−λ), giving λ > 8/9. Equality is a tie, resolved here by original candidate order. At λ=1, only relevance matters. At λ=0, only redundancy matters after the initial seed; choosing A first is an explicit convention for the otherwise empty selected set. 比较两个表达式即可找到阈值。纯多样性可能提升不够相关的结果,所以一般先获取相关候选,再选取合适的权重。
Once S contains several documents, each remaining candidate is penalised by its most similar selected document, not its average similarity and not the sum. Recompute after every selection because S has changed. The relevance term and the document–document term may use different models; their score scales must be compatible. MMR is greedy and does not promise the global optimum for every list-level objective. 只要有一篇已选结果和候选高度重复,就可能带来大的惩罚。不能先对每篇算一次固定 MMR 分数,再排序到底。
Metric references / 指标来源:Expected Reciprocal Rank (2009) · Intent-based diversification: metrics and algorithms (2011).
A single relevance label can rate two duplicate documents equally highly. Therefore two lists can have identical ordinary nDCG yet cover different numbers of intents. α-nDCG (alpha-normalised Discounted Cumulative Gain) discounts repeated coverage of the same judged subtopic. ERR (Expected Reciprocal Rank) models stopping at a satisfying result; plain ERR is not automatically an intent-diversity measure. Intent-aware variants such as ERR-IA aggregate per-intent utility. 相关性判断不包含意图标注时,不能仅靠普通 nDCG 宣称多样性提升。ERR 与 intent-aware ERR 要区分。课件 p.40 列出评价方向,这里补充了术语边界。
Connection to Weeks 9–10, not an extra assignment requirement: under a RAG context budget, three near-duplicate passages may leave another necessary fact uncovered. Diversity-aware evidence selection can be tested while fixing the query set, generator, prompt and budget. However, novelty does not guarantee support or truth; complementary evidence, necessary corroboration and conflicting claims must still be checked. 这是帮助理解的架构联系,不是要求给 Project 2 额外添加 MMR。实验应分别衡量检索相关性、意图覆盖、答案正确性和 groundedness,不能把这些分数混为一谈。
Practice the numerical examples in Revision Q/A / 到题库检查自己是否真的会算
Implement pagerank(pages, lam, eps) for the supplied five-page graph. Relate every matrix operation to the lecture’s random surfer. MMR (Maximal Marginal Relevance) remains a lecture topic, not an extra task in this notebook.这份 practical 只有简化 PageRank 实现。不要把 lecture 的 MMR 自动当成老师要求额外完成的代码任务。
Data flow: column-stochastic link matrix M + teleport λ → initial r → repeated probability updates → convergence → page scores
M[i,j] is the probability of moving FROM page j TO page i. Each column distributes one source page’s outgoing mass and sums to 1. The supplied matrix is already weighted; do not replace its 0.5 entries with 1.列是出发页面、行是到达页面。先画出一列的箭头,才能知道应该使用 M @ r,而不是 r @ M。
Here lam=0.15 is teleport probability; 1−lam=0.85 follows links. For N=5, each target receives lam/N=0.03 teleport mass. This is the same model as the lecture formula using d=0.85 for link following.lam 和 d 取值不同,是因为定义相反。代码 docstring 里的 damping 一词不够明确,要以实际公式中乘在哪一项为准。
Start with r=ones(N)/N and compute r_new=lam/N+(1−lam)*(M @ r). Use the entire old vector for one update; do not overwrite one page and immediately use its new value for the next page.M @ r 把每个来源页面的概率按出链分配。同步更新对应一次随机浏览者转移;逐项覆盖会变成另一套迭代过程。
One explicit rule is sum(abs(r_new−r))<eps. Assign r_new before returning and set a maximum iteration count as a safety guard. A different norm can produce a different iteration count, so document the rule.eps 是两次迭代差异的容忍度,不是 PageRank 分数门槛,也不是 p-value。达到最大轮数却未收敛,应报告未收敛。
Check finite nonnegative entries, total probability near 1 and a small fixed-point residual. The classroom matrix has no dangling column. For a new graph, zero-outdegree columns need redistribution, such as uniform 1/N, before applying this stochastic-matrix formulation.原例已经每列和为 1;扩展到任意网页图时,不能忽略全零列。随机跳转项本身不自动补回丢失的 dangling mass。
Starting from [0.2,0.2,0.2,0.2,0.2], the teacher’s matrix yields [0.200,0.115,0.115,0.285,0.285] after one step. With an L1 stopping threshold of 1e−6 it reaches approximately [0.254192,0.138032,0.138032,0.205990,0.263755] after 57 updates. The two equal middle scores follow from symmetric incoming links. These numbers were checked locally using the supplied matrix.结果顺序对应矩阵中的页面顺序,不能排序后再与原索引配对。第 5 页最高,第 1 页其次;概率和约为 1。这里报告的是该小矩阵的数值验证,不是网页质量评测。
PageRank is query-independent graph authority. BM25/DPR estimate query–document matching; MMR also considers already selected results. To test PageRank in retrieval, combine it with a query-dependent score under an explicit rule and evaluate the ranking with qrels.高 PageRank 不等于当前问题最相关,更不等于答案真实。它可以作为排序特征,但不是替代 BM25、nDCG 或人工 gold。
Source map: prac-week11.ipynb: Implement simple pagerank; official notebook modified 14 October, downloaded 16 October 2026. Independent study explanation; original files remain on the course platform.
Back to Week 11 lecture conceptsWeek 12 研究三件事:如何利用 relevance evidence 改进查询,如何扩展或精简 query,以及如何把多个不同比例尺的 ranked lists 合成一个结果。
| Feedback | Evidence / 证据 | Main risk / 风险 |
|---|---|---|
| Explicit | 用户明确标出 relevant 或 non-relevant documents | 质量高,但需要用户额外操作 |
| Implicit | 把 clicks 等行为解释为 relevance signal | examination 与 presentation bias |
| Pseudo-relevance feedback (PRF) | 假设初始排名顶部文档相关 | 首轮排名弱时会 query drift |
三者差别是 evidence certainty,不只是界面不同。“More like this” 是 explicit query-by-example;普通点击不是天然 relevance label;PRF 通常不询问用户,只取正反馈。
Original query 负责锚定方向;relevant centroid 把 query 拉向有用词项;non-relevant centroid 把它推离。α、β、γ 表示对三类证据的信任。某些分量可能变负,实际系统必须说明保留、截断还是另行处理。
可以从 thesaurus/controlled vocabulary、collection 或 query-log statistics、top-ranked documents 加词。MeSH 这类领域词表可能帮助 specialist search;通用同义词扩展可能引入错误词义。Query suggestion 提供 alternative query,不一定只是把当前 query 加长。
Dice coefficient 与 mutual information 用共现估计词项关系;mutual information 容易偏好 rare events。逐个扩展 tropical 和 fish 可能丢失短语含义;whole-query 或 n-gram context 更稳,但不可能为所有潜在 query 预先穷举。
TLM 允许 document term u 通过 translation probability 为 query term w 提供概率质量,再把 translated document model 放进 smoothed query-likelihood model。Translation probabilities 可由 mutual information 或 embedding cosine similarity 估计,但 self-translation 必须足够强,不能为了语义扩展丢掉 exact matching。
Distributional representations 让上下文相似的词在向量空间接近。把 cosine similarity 归一化可得到 translation distribution,也能覆盖 syntactic variants。Similarity 不等于 identity;不受控的近邻会加入“主题相关但任务不相关”的词。
Verbose query 可能包含偏题词。IDF-r 保留 IDF 最高的一定比例词项;learned reduction 可用 query-performance predictors 及其他 features 预测保留比例。Expansion 和 reduction 表面方向相反,目标相同:让 query 更准确表达 information need。
| Family | Uses / 使用信息 | Examples |
|---|---|---|
| Voting | 哪些系统提名该文档 | plurality, Copeland |
| Rank aggregation | 文档在每个列表的位置 | Borda, Reciprocal Rank Fusion |
| Score aggregation | 通常先归一化,再组合 scores | CombSUM, weighted linear fusion, CombMNZ |
Chorus effect 奖励多系统一致;skimming effect 从不同系统取得不同文档;dark-horse system 可能贡献独特好结果,也可能带来独特噪声。Fusion 常能超过最佳单一系统,但加入任意系统都提升并无保证。
RRF 只使用 rank,常见 v=60,越靠后的结果贡献越小。CombSUM 直接加分;linear fusion 为不同系统赋权;CombMNZ 再乘以支持该文档的列表数。某列表未返回该文档时,该列表不贡献。
BM25、language model 和 neural scores 的来源与范围不同。可按每个 list 使用 min–max、shifted-sum 或 z-score。理想方案应 shift invariant、scale invariant,并尽量不让一个 outlier 改变所有其他文档。Min–max 可直接处理负分,但对极端 min/max 敏感。
q₀=(1,0),relevant centroid 为 (0.8,0.6),non-relevant centroid 为 (0.2,0.9),且 α=1, β=0.75, γ=0.25,则 q′=(1.55,0.225)。这是下一轮检索的 query direction,不是 relevance judgement。v=60。A 在两列表位于第 1、4 名,得 1/61+1/64≈0.03202;B 位于第 2、1 名,得 1/62+1/61≈0.03252,因此 B 更高。计算前要说明 rank 是否从 1 开始。[-8,-5,-2],min–max 后是 [0,0.5,1]。给所有原始分数加 100 后结果不变,说明 shift invariance;若出现新的极端最大值,其他 normalized gaps 会一起变化。Source status / 来源状态:已于 2026-10-20 核对正式发布的 49 页 Week 12 lecture,以及正式 practical notebook 与六个配套文件。官方 Week 12 Live Q/A deck 当时尚未开放;本站没有把任何 Week 12 题目冒充官方题,相关题目均明确标为 self-authored practice。原课程文件仅保留私有本地。
Public reading / 公开延伸:Stanford IR book: relevance feedback and query expansion · National Library of Medicine: MeSH.
Implement pseudo-relevance feedback (PRF) expansion, IDF-r reduction, and three rank-fusion methods. Evaluate every change with the same MS MARCO queries and qrels instead of assuming that a longer query or a fused list must be better.实现 PRF 扩展、IDF-r 精简以及三种排序融合方法。所有变化都要在同一组 MS MARCO queries 与 qrels 上评价,不能默认 query 更长或融合后就一定更好。
Data flow: 43 queries + index → baseline run → expand or reduce query → MAP/nDCG@10/recall@1000; four runs → per-query normalisation → Borda/CombSUM/CombMNZ → paired comparison
Load the 43 supplied queries, retrieve up to 1,000 documents with the same Lucene analyzer, and write qid, Q0, docid, rank, score and run tag. The supplied qrels contain many judged pairs, including relevance label 0; a qrel row is a judgment, not automatically a relevant document.先固定 analyzer、retrieval depth 与评价指标。qrels 中出现 docid 只表示它被判断过,label=0 仍是不相关;不能用“qrels 行数”当作相关文档数。
Retrieve an initial top-50 list, treat only the top n as pseudo-relevant, obtain each document vector, and aggregate candidate term evidence such as TF-IDF. Select the top m expansion terms, append them to the analyzed original query, rerun retrieval, then compare MAP, nDCG@10 and recall@1000. Exclude accidental noise and explain any query drift.n 是反馈文档数,m 是新增词数,不能混用。首轮 top documents 若属于错误词义,高权重词会把 query 继续拉偏;因此要同时检查总体指标与具体失败 query。
Analyze the query using the same Lucene analyzer, obtain document frequency for each resulting term, compute the practical’s IDF convention, and retain the n highest-IDF terms. Decide explicitly how duplicates, ties and n larger than the term count behave. A ratio-based variant must convert the ratio to a documented integer count.先分词再按 IDF 排序,保留区分度高的词。不要把 collection frequency 当 document frequency;删词可能提高效率与聚焦,也可能删掉 intent 所需的上下文。
The four supplied runs use TF, IDF, TF-IDF and BM25 scales. For each query and each run separately, min–max normalise scores before CombSUM or CombMNZ. Handle a constant-score list explicitly because its maximum equals its minimum. Borda uses rank positions, so raw score scale is irrelevant to its vote.归一化单位是“一个 query 的一条 run”,不是把所有 query 混在一起。若 max=min,直接除会得到零除;需要先声明处理规则。Borda 只看 rank,不看原始分数大小。
With a one-based rank r in a list of length n, Borda contributes (n−r+1)/n; if enumerate starts at zero i, use (n−i)/n. CombSUM adds normalised scores over the union of documents. CombMNZ multiplies that sum by the number of runs in which the document has a positive score. Try combinations of the four runs, report MAP and nDCG, and use paired per-query tests for the chosen comparison.Borda 最容易出现 0/1-based off-by-one。CombSUM 奖励总证据,CombMNZ 还奖励多条 run 的共同支持。搜索“最佳组合”会增加多重比较风险,不能只报最高平均数而不说明选择过程。
For two three-document rankings A=[d1,d2,d3] and B=[d2,d3,d4], Borda awards [1,2/3,1/3] by position. Totals are d2=5/3, d1=1, d3=1 and d4=1/3. If normalised score runs instead give d1 scores .9 and .2, d2=.4 once and d3=.8 once, CombSUM gives 1.1, .4 and .8; CombMNZ gives 2.2, .4 and .8 because only d1 has positive support from two runs.这个例子把 rank vote 与 score fusion 分开。Borda 不需要分数归一化;CombSUM/CombMNZ 则必须先让不同模型的分数可比较。若 min–max 后恰为 0,按 practical 定义它不增加 CombMNZ 的正分出现次数。
Fusion can add correlated errors, reward documents repeated by several weak systems, or distort evidence through unsuitable normalisation. Compare against every component run on the same qids, inspect per-query differences, and distinguish a mean gain from statistically reliable improvement.融合不是自动集成增益。多个弱模型可能重复同一错误;归一化也可能压缩有用差距。必须与每条 component run 在相同 qids 上比较,并检查哪些 query 变差。
Source map: prac-week12.ipynb: Exercises 1–3 and the supplied MS MARCO qrels, queries and four run files; released and reviewed 20 October 2026. Independent study explanation; original files remain on the course platform.
Back to Week 12 lecture conceptsWeek 13 回到 inverted index,讨论一个工程问题:在超大规模语料上,如何又快又好地返回 top-k 结果。内容包括 compressed postings、quantized impact scores、dynamic pruning,以及 learned sparse retrieval 带来的效率代价,最后介绍期末口试的形式。
有些搜索确实需要精确词项匹配;BM25 在没见过的数据上仍是强 baseline;倒排索引随语料增长扩展性好。SPLADE、TILDEv2 等 learned sparse 模型也把输出存进倒排索引,所以索引效率直接决定它们是否实用。在 web 规模下,快速返回足够好的结果,通常比慢慢返回最好的结果更有价值。
一个 posting 由 document identifier 加 term frequency 或分数组成,同一列表中的 identifier 严格递增。只存相邻 identifier 的差值(delta coding)会得到更小、更易压缩的整数。代价是要取第 n 个 identifier 时,必须把前面的差值累加(prefix sum)。Block-based index 直接存每个块的首个 identifier,查找只需在一个块内累加,整块也可以不解压就跳过。
Variable Byte(VByte)编码每个字节存 7 个数据位,剩下一位标记后面是否还有字节,所以小整数占用更少字节。实际系统大多用 block-based codecs。编码越小,编码和解码通常越慢;该选哪种 codec,取决于更看重空间还是查询延迟,也就是在 space–time Pareto frontier 上选位置。
Boolean conjunction(AND)只返回包含所有查询词的文档,做法是推进最短的列表,并在其他列表中向前 seek。Disjunction(OR)接受包含任意查询词的文档。匹配规则并不决定排序:ranked conjunction 只给 AND 匹配到的文档打分,而 top-k disjunctive retrieval 相信 ranking function,对任何匹配文档都可以打分。
对 BM25 这类可加模型,与查询有关的只有“用到哪些词”,所以每个词—文档贡献 sd,t 可在建索引时算好。浮点分数很难压缩,因此用全局 min–max 把它映射成 b 位整数,替代 term frequency 存进 postings;文档得分变成 impacts 之和。注意 2b 本身需要 b+1 位,所以实现通常乘以 2b − 1,或把最大值截断。
穷举式 top-k 检索给每个 posting 打分,同时用 min-heap 保留最好的 k 个文档;结果 rank-safe,但要解压并计算全部内容。Dynamic pruning 在保持同样 rank-safe 结果的前提下少做工作:建索引时存下每个列表的最大 impact Ut,用 heap 里的最小分数作为阈值 θ,把候选文档所在列表的上界相加;如果上界不可能超过 θ,就跳过该文档不打分,跳过的块也不用解压。前提是 scoring function 可加、预先存好列表上界,并且列表支持高效随机访问。判断是否跳过本身也有开销。
WAND(“weak” 或 “weighted” AND)按各列表当前文档排序,找到第一个累积上界超过 θ 的 pivot;pivot 之前的文档不可能进入 top-k。MaxScore 按上界给词排序,把合起来也达不到 θ 的低上界词标为 non-essential:候选只从 essential lists 产生,non-essential lists 只用来为这些候选查分。
| Strategy | Traversal order / 遍历顺序 | State / 状态 |
|---|---|---|
| Document-at-a-time (DAAT) | 所有列表一起,一次处理一个文档 | 当前 top-k 的 heap |
| Term-at-a-time (TAAT) | 一次处理一整个列表 | 每个文档一个 accumulator |
| Score-at-a-time (SAAT) | postings 按 impact 分组,从高到低处理 | accumulators;可以提前停止 |
倒排索引高效靠的是稀疏性:一篇文档只涉及少数词,一个词只出现在少数文档里,一个查询只有少数词。Zipf’s law 让大多数词都很少见。Learned sparse 模型削弱了这三个假设:expansion 给文档和查询加了很多词;有些 token 被“挪作他用”,几乎出现在所有文档里,部分像 dense vector;长列表还可能得到很高的 learned impact,而 BM25 给长列表的 impact 很低。于是上界一直很高,剪枝去掉的候选变少,打分的 postings 变多,查询变慢。鼓励稀疏的 regularisation 是一种应对方法。
| Stage / 阶段 | What happens / 会发生什么 | How to prepare / 怎样准备 |
|---|---|---|
| 前 5 分钟 | 拿到两道提示问题,同时核对身份证件 | 为每题准备简短讲解:定义、机制、例子、局限 |
| 主要对话 | 先回答其中一题,考官再追问细节、问题和改进方法 | 练习解释方法为什么有效、会在哪里出问题,而不只是背定义 |
| 切换主题 | 考官可能转到课程其他主题,以确认水平或给你冲更高档的机会 | 把各周串起来:representation → retrieval → reranking → evaluation → RAG 与 agents |
口试 30 分钟,闭卷,需要带身份证件;不能使用任何笔记或设备,结束时不会当场给分,因为成绩要在教学团队内统一审核。请通过课程官方渠道预约时间。把它当作一次技术面试:边想边说,说明假设,必要时及时纠正自己。
[3, 7, 8, 15, 30, 31, 45, 52] 变成差值 [3, 4, 1, 7, 15, 1, 14, 7]。不分块时,取第 7 个 identifier 要在第一个之后累加六个差值。每块 4 个、直接存首 identifier 3 和 30 时,第 7 个是 30 + 1 + 14 = 45,只需两次加法。10000001 00000010;300 = 2·128 + 44 编码为 10000010 00101100;5 只需一个字节 00000101。普通 32 位整数总是占四个字节。smin=0、smax=20、b=8,BM25 贡献 7.5 映射为 256 × 7.5 / 20 = 96。最大分数会映射到 256,需要 9 位,所以实践中会乘以 255 或截断。Ua=4、Ub=7、Uc=3,当前 heap 阈值 θ=9。只出现在 a、c 中的文档上界为 7 < 9,不打分直接跳过;出现在 a、b 中的文档上界为 11 ≥ 9,必须精确打分。对 MaxScore,把上界排序为 3(c)、4(a)、7(b):c 和 a 合计只有 7 < 9,是 non-essential,只有列表 b 产生候选。Source status / 来源状态:已于 2026-10-27 核对 206 页的 Week 13 客座讲座(高效倒排索引)和 4 页期末考试说明。Week 13 没有 Live Q/A deck;本站所有 Week 13 题目均明确标为 self-authored practice。原课程文件仅保留私有本地。
Public reading / 公开延伸:Stanford IR book: index compression · Stanford IR book: efficient scoring and ranking.
These are dated paraphrases of staff replies, not new assessment rules. Follow the latest course notebook and marking sheet if later updates differ. 这里是带日期的教师答复摘要,不是新增作业要求。以后若有更新,以最新版 notebook 和 marking sheet 为准。
| Scope | Verified clarification | Practical consequence |
|---|---|---|
| Main Project 2 / Task 2 | The earlier “development set” wording refers to the sampled collection, queries and qrels; staff corrected the notebook wording. | Use the simplified main-task sampled-data protocol. Do not reinterpret the bonus training queries as the main task's development set. |
| Dense-retriever fine-tuning bonus | Train queries/qrels are reserved for the bonus. A separate staff reply says dev queries/qrels have now been attached under the bonus folder. | Obtain the official bonus files; train on train data and use dev data for checkpoint/hyperparameter selection according to the bonus instructions. |
Important distinction: the main-task terminology correction and the later bonus dev-file addition concern different scopes. They are not contradictory. The bonus question identifies dev_queries.tsv and dev_qrels.txt. Staff confirmed availability; this website update has not checked the contents or counts of those new files. 主任务把 sampled data 称为 development set 的笔误,和 bonus 新增真正的 dev 文件,是两件事。不要把它们合并成“所有任务都用同一个 dev”。本次只核对教师回复,未核验新数据文件内容或条数。
For a scientific report, state exactly which data selected a model and which data evaluated it. Reusing a sample for selection and comparison is a course simplification, not evidence of an independent held-out generalisation estimate. 即使老师允许简化,也应如实记录选择模型与评价所用的数据;不要把重复使用的 sampled set 写成独立测试集。
Staff clarification: main-task sampled data (login required) · Staff clarification: bonus dev files (login required)
A newer course question asks which modeling.py source to use. No staff answer was visible when checked. This page therefore does not approve a replacement implementation. Compare the class API and checkpoint compatibility before treating files from different versions as interchangeable. 目前看到的是同学的问题,不是老师已批准的方案。不能把“能否用 GitHub 版本”的提问当作已确认答案。
先认清符号,再问每个因子在奖励或惩罚什么。
| 名称 | 公式 | 最重要的解释 |
|---|---|---|
| P@k | relevant in top k / k | 只看截断位置 k 内的精确率 |
| Recall | retrieved relevant / all relevant | 需要知道总相关文档数 |
| F1 | 2PR/(P+R) | precision 与 recall 的调和平均 |
| AP | (1/|Rel|) Σ P@k × rel(k) | 每遇到一个 relevant 就记录当时 precision |
| MAP | mean(AP over queries) | 所有 query 同权 |
| RR | 1/rank(first relevant) | 只在乎第一个 relevant |
| nDCG@K | DCG@K / IDCG@K | 支持 graded relevance 与位置折扣 |
| RBP | (1-p)Σ r_i p^(i-1) | p 显式建模用户继续浏览概率 |
| 名称 | 公式 | 直觉 |
|---|---|---|
| IDF | log(N/(1+df)) | 越少文档出现,区分度越高 |
| TF-IDF | Σ tf × idf | 文档内重要 × 集合内稀有 |
| Cosine | q·d/(||q||||d||) | 比较方向,降低长度影响 |
| BIM weight | log((N-df)/df) | 来自词在相关/非相关文档中的概率优势 |
| BM25 | Σ IDF × saturated_tf(length-normalized) | 稀有词、饱和 tf、文档长度三者组合 |
| 名称 | 公式 | 直觉 |
|---|---|---|
| Softmax | exp(z_i)/Σexp(z_j) | 把任意分数转换为和为 1 的权重 |
| Attention | softmax(QKᵀ/√d_k)V | 按相关性加权汇总 value |
| InfoNCE | -log exp(s+)/[exp(s+)+Σexp(s-)] | 在候选集合中把正样本分类出来 |
| Margin | max(0, m-(s+-s-)) | 正样本必须领先负样本至少 m |
| MaxSim | Σ_i max_j E_qi·E_dj | 每个 query token 寻找最匹配 doc token |
| Autoregressive | Π_t P(y_t|y_<t) | 根据已生成前缀预测下一个 token |
| monoBERT score | P(relevant=1 | q,d) | 联合编码 query 与单篇 document 后做 pointwise 分类 |
| Binary cross-entropy | -Σ_pos log(s) - Σ_neg log(1-s) | 提高 relevant pair 的分数,压低 negative pair 的分数 |
| duoBERT aggregation | s(d_i)=Σ_{j≠i} p(d_i>d_j|q) | 把 document 对其他 candidates 的 pairwise 胜率相加 |
| RAG answer | y ~ P(y | q, retrieve(q)) | 答案同时依赖 query 与检索证据;retrieval 是可观察的中间变量 |
| Pearson correlation | r = cov(x,y)/(σxσy) | 衡量 retrieval gain 与 answer gain 的线性关系;零方差时未定义 |
| Name | Formula / procedure | Meaning / 中文直觉 |
|---|---|---|
| Full-information utility | Δ(f,D,y)=Σ λ(rank(d))·y(d) | Rank discount λ weights relevance y; DCG and P@k are instances of this pattern. |
| Naive click estimator | Σ λ(rank_i)·c_i | Directly treats clicks as labels, so exposure bias remains. |
| IPS estimator | Σ [λ(rank_i)/P(o_i=1)]·c_i | Inverse examination probability corrects position bias in expectation / 用反倾向权重校正。 |
| Off-policy weight | P(action|new policy) / P(action|logging policy) | Reweights logged actions as if they came from a candidate policy. |
| Counterfactual update | choose f if Risk_IPS(f) < Risk_IPS(f₀) | Deploy the candidate with lower estimated counterfactual risk. |
遇到选择题时,把模型拆成这四列判断,比背一句描述可靠。
| 模型 | Representation / pooling | Similarity / interaction | Training | 定位 |
|---|---|---|---|---|
| BM25 | 稀疏 term statistics | 逐 term 加权求和 | 通常无需训练;调 k1,b | 强 lexical baseline |
| TILDEv2 | query token IDs;document contextual token weights | exact match + per-term max weight | document-side neural training/expansion | fast learned sparse reranker |
| SPLADE(v2) | mostly-zero vocabulary vectors | sparse dot product | MLM head + distillation + sparsity regularization | learned sparse first-stage retrieval |
| PromptReps | LLM dense hidden state + sparse logits | normalized hybrid interpolation | zero-shot; no retriever training | LLM-based hybrid representation |
| DPR | CLS,query/doc 双编码 | dot product | InfoNCE/NLL;in-batch + hard negatives | dense first-stage retrieval |
| RepBERT | mean token embeddings | inner product | MultiLabelMarginLoss | BERT bi-encoder |
| ANCE | CLS / first token | dot product | NLL/InfoNCE-style;ANN self-mined negatives | 动态 hard-negative mining |
| Contriever | mean pooling | cosine | unsupervised contrastive;same-doc spans | 无监督 dense retriever |
| ColBERT | 保留所有 token embeddings | late interaction MaxSim | contrastive / pairwise CE | 效率与细粒度匹配折中 |
| E5-Mistral | Mistral decoder;EOS/last token | temperature-scaled cosine | InfoNCE;synthetic + supervised data | LLM embedding model |
| LLM2Vec | decoder 改双向 attention;pooling | cosine 常见 | MNTP + unsupervised SimCSE + optional supervised | 把 decoder 转为通用 encoder |
| RepLLaMA | Llama2;EOS last hidden | dot product | InfoNCE;supervised + LoRA | LLM retriever |
| Cross-encoder | query 与 doc 联合编码 | 分类/回归头输出 relevance | pointwise/pairwise/listwise | 高精度 reranker |
| monoBERT | [CLS] q [SEP] d [SEP] joint encoding | relevant-class softmax score | binary cross-entropy;human positives + BM25 negatives | pointwise BERT reranker |
| BERT-MaxP / FirstP / SumP | document passages | max / first / sum of passage scores | passage labels transferred from document | long-document score aggregation |
| PARADE | passage representations | average / max / attention / Transformer aggregation | document-level reranking | long-document representation aggregation |
| duoBERT | query + two candidate documents | pairwise probability then win aggregation | pairwise cross-entropy | late-stage BERT reranker |
| LLM pointwise | query + one passage prompt | label/logit or relevance grade | zero-shot prompting in Week 8 | simple LLM reranker |
| LLM pairwise | query + two passages | direct preference | zero-shot prompting | robust but comparison-heavy |
| LLM listwise | query + candidate list | generated permutation | zero-shot prompting | list context, generation cost |
| LLM setwise | query + small candidate set | best-item logits | zero-shot prompting | fewer calls than pairwise |
| Corpus-in-Context (CiC) | entire corpus + demonstrations in an LC-LLM prompt | generate relevant document IDs | few-shot prompting; no retriever training | extreme model-based retrieval; corpus must fit context |
| Screenshot retriever | page pixels/layout encoded by VLM | dense similarity / learned retrieval score | vision-language contrastive training | layout-rich and multimodal documents |
| DiffRetriever | multiple parallel representative tokens | multi-representation similarity | diffusion language-model representation learning | efficient multi-vector generative retriever |
| Basic RAG | query + top-k external passages | autoregressive answer likelihood | retriever and generator may be frozen or adapted | IR4LLM; evidence-grounded answering |
| Rewrite-Retrieve-Read | rewritten search query + retrieved passages | rewrite reward + downstream answer utility | pseudo-data warm-up and optional policy optimisation | repair ambiguous/underspecified queries |
| BlendFilter | retrieved evidence + model knowledge | LLM filtering then answer generation | prompted augmentation/filtering | remove distracting retrieved knowledge |
| RARR | draft answer + researched evidence | support check and revision | post-hoc research/revision | retrofit attribution to generated text |
| PageRank | directed graph; one value per page | stationary random-surfer probability | power iteration, no relevance-label training required | query-independent authority feature |
| PRP | estimated document relevance probabilities | descending probability order | a ranking principle, not a trained model | relevance-oriented ordering |
| MMR | candidate scores + document similarities + selected set | relevance reward minus maximum redundancy | greedy selection; tune the trade-off separately | diversity-aware candidate reranking |
| Family | Searches full corpus? | Offline cost | Online cost | Best at | Characteristic failure |
|---|---|---|---|---|---|
| Lexical sparse: BM25 | Yes, inverted index | low indexing cost | very low | exact names, rare terms, transparent scores | vocabulary mismatch / 词不相同就难匹配 |
| Dense bi-encoder: DPR, ANCE, Contriever, E5, RepLLaMA | Yes, ANN vector index | encode every document | encode query + ANN | semantic paraphrases | false semantic neighbours; large dense index |
| Learned sparse: TILDEv2, SPLADE | Yes when indexed as sparse postings | neural document encoding | low to medium | lexical efficiency plus learned weighting/expansion | expansion noise; model/index coupling |
| Late interaction: ColBERT | Yes with specialised index | store token vectors | medium | fine-grained token matching | larger index and MaxSim computation |
| Cross-encoder: monoBERT, duoBERT | No, candidate reranker | model training | high per pair | deep query-document interaction | cannot recover documents absent from candidates |
| Prompted LLM ranker | Usually no, candidate reranker | little task-specific training | high; repeated generation/logits | zero-shot instructions and reasoning-rich comparison | prompt/order sensitivity and format failures |
| LC-LLM / CiC | Yes only if corpus fits context | construct/cache corpus prefix | very high without caching | consolidated small-corpus search | position bias, context cost, lost-in-the-middle |
| RAG system | Retriever does | index + prompt/evaluation design | retrieval + long generation | current, inspectable external knowledge | distractors, evidence conflict, unsupported answers |
沿用 3208 学习页的阅读顺序:缩写 → 完整英文名 → 中文名 → 一句话定义。不是每个模型名都是首字母缩写;名称型模型会明确标注其作用。
| Short name | Full name | 中文 | Definition / 定义 |
|---|---|---|---|
| IR | Information Retrieval | 信息检索 | 从文档集合中找到并排序能够满足用户信息需求的内容。 |
| TF | Term Frequency | 词频 | 某个 term 在当前 document 中出现的次数。 |
| DF | Document Frequency | 文档频率 | 集合中包含某个 term 的 documents 数量。 |
| CF | Collection Frequency | 集合频率 | 某个 term 在整个 collection 中出现的总次数。 |
| IDF | Inverse Document Frequency | 逆文档频率 | 根据 term 在多少文档中出现来衡量其区分度。 |
| TF-IDF | Term Frequency-Inverse Document Frequency | 词频-逆文档频率 | 结合文档内频率与集合内稀有性的词项权重。 |
| VSM | Vector Space Model | 向量空间模型 | 把 query 和 document 表示为向量并比较其方向或相似度。 |
| BIM | Binary Independence Model | 二元独立模型 | 假设词项独立,用词在相关与非相关文档中的概率差异打分。 |
| BM25 | Best Matching 25 | 最佳匹配第 25 版 | 结合 IDF、term-frequency saturation 与 document-length normalisation 的经典词汇检索模型。 |
| RSJ | Robertson-Spärck Jones | 罗伯逊-斯帕克·琼斯权重 | BM25 常用的概率式 IDF 权重来源,用相关与非相关文档中的词项证据衡量区分度。 |
| F1 | F-score with β = 1 | F1 分数 | Precision 与 recall 的调和平均,在两者同等重要时使用。 |
| AP | Average Precision | 平均精确率 | 对一个 query,在每个相关结果出现位置记录 precision 并平均。 |
| MAP | Mean Average Precision | 平均精确率均值 | 对所有 queries 的 AP 取平均。 |
| RR / MRR | Reciprocal Rank / Mean Reciprocal Rank | 倒数排名 / 平均倒数排名 | 关注第一个相关结果的位置,再在 queries 间取平均。 |
| DCG | Discounted Cumulative Gain | 折损累计增益 | 让高相关文档得分更高,并按排名位置进行折扣。 |
| IDCG | Ideal Discounted Cumulative Gain | 理想折损累计增益 | 同一 query 在理想相关性顺序下可获得的 DCG。 |
| nDCG | Normalized Discounted Cumulative Gain | 归一化折损累计增益 | 以 DCG 除以 IDCG,使不同 query 的排名质量可比较。 |
| RBP | Rank-Biased Precision | 排名偏置精确率 | 用 persistence 参数模拟用户继续浏览下一结果的概率。 |
| PageRank | Model name, not an acronym | 链接权威度模型 | 用带 teleport 的 random surfer 稳态概率定义 query-independent link authority。 |
| PRP | Probability Ranking Principle | 概率排序原则 | 在相关概率估计准确且 relevance utility 独立等假设下,按 estimated relevance probability 递减排序。 |
| MMR | Maximal Marginal Relevance | 最大边际相关性 | 一种 greedy reranking rule,在 query relevance 与相对已选结果的 redundancy 之间取舍。 |
| α-nDCG | Alpha-normalised Discounted Cumulative Gain | 考虑子主题的归一化折损累计增益 | 对同一 intent 的重复覆盖进行折扣,需要子主题相关性判断。 |
| ERR / ERR-IA | Expected Reciprocal Rank / Intent-Aware Expected Reciprocal Rank | 期望倒数排名 / 意图感知期望倒数排名 | ERR 模拟用户满意后停止;ERR-IA 按意图权重汇总效用。普通 ERR 不自动衡量意图多样性。 |
| RF / PRF | Relevance Feedback / Pseudo-Relevance Feedback | 相关反馈 / 伪相关反馈 | 从 relevance evidence 更新检索;PRF 不询问用户,直接假设初始 top results 相关。 |
| Rocchio | Model name, not an acronym | Rocchio 查询更新 | 让 vector-space query 靠近 relevant centroid,并远离 non-relevant centroid。 |
| TLM / NTLM | Translation Language Model / Neural Translation Language Model | 翻译语言模型 / 神经翻译语言模型 | 通过可翻译成 query term 的 document terms 建模;NTLM 用 embeddings 估计词项关系。 |
| QPP | Query Performance Prediction | 查询性能预测 | 预测 query 的潜在检索表现,也可帮助决定 verbose query 应保留多少词。 |
| RRF / CombMNZ | Reciprocal Rank Fusion / Combination Multiply Non-Zero | 倒数排名融合 / 非零计数加权组合 | RRF 用 reciprocal rank discount;CombMNZ 用支持列表数放大 score sum。 |
| VByte | Variable Byte coding | 可变字节编码 | 按 7 位一组存整数,每个字节用一位标记后面是否还有字节。 |
| WAND | Weak (or Weighted) AND | 弱与 / 加权与剪枝 | 用列表上界找 pivot,跳过不可能进入 top-k 的文档。 |
| MaxScore | Algorithm name, not an acronym | MaxScore 剪枝算法 | 按上界把词分成 essential 与 non-essential 列表。 |
| DAAT / TAAT / SAAT | Document-, Term- and Score-at-a-time | 按文档 / 按词 / 按分数遍历 | 三种索引遍历顺序:逐文档、逐列表、按 impact 从高到低。 |
| LSR | Learned Sparse Retrieval | 学习式稀疏检索 | 学习词权重(常含扩展词),但仍通过倒排索引检索。 |
| BERT | Bidirectional Encoder Representations from Transformers | 双向 Transformer 编码表示 | 使用双向上下文的 encoder 模型,可联合编码 query-document pair。 |
| CLS / SEP / EOS | Classification / Separator / End of Sequence token | 分类 / 分隔 / 序列结束标记 | 分别用于汇总分类表示、分开输入片段,以及标记序列结束。 |
| MLM | Masked Language Modeling | 掩码语言建模 | 遮住部分 tokens,再根据上下文预测它们的预训练目标。 |
| FFN | Feed-Forward Network | 前馈神经网络 | Transformer block 中对每个 token 独立执行的非线性特征变换。 |
| GPT | Generative Pre-trained Transformer | 生成式预训练 Transformer | 以自回归 next-token prediction 为核心的 decoder 模型家族。 |
| LLM | Large Language Model | 大型语言模型 | 在大规模语料上训练、可通过自然语言 prompt 完成多种任务的模型。 |
| DPR | Dense Passage Retrieval | 稠密段落检索 | 分别编码 query 与 passage,并通过 dense-vector similarity 检索。 |
| ColBERT | Contextualized Late Interaction over BERT | BERT 上下文化后期交互模型 | 保留 query 与 document 的 token vectors,以 MaxSim 进行细粒度 late interaction。 |
| ANN | Approximate Nearest Neighbour | 近似最近邻 | 以少量精度损失换取大规模向量搜索速度。 |
| ANCE | Approximate Nearest Neighbor Negative Contrastive Estimation | 近似近邻负样本对比估计 | 利用当前 dense model 的 ANN 检索错误动态挖掘 hard negatives。 |
| NLL | Negative Log-Likelihood | 负对数似然 | 惩罚模型给正确目标分配较低概率的损失函数。 |
| CE | Cross-Entropy | 交叉熵 | 衡量目标分布与模型预测分布差异的分类损失。 |
| InfoNCE | Information Noise-Contrastive Estimation | 信息噪声对比估计 | 让正样本相对于一组 negatives 获得更高相似度的对比学习目标。 |
| TILDE | Term Independent Likelihood moDEl | 词项独立似然模型 | 从文档侧学习 vocabulary term likelihood,用于高效查询似然打分。 |
| SPLADE | Sparse Lexical and Expansion Model | 稀疏词汇扩展模型 | 学习 vocabulary-aligned sparse weights,并可激活原文未出现的相关词。 |
| ReLU | Rectified Linear Unit | 修正线性单元 | 使用 max(0,x) 去除负激活;SPLADE 用它产生非负词项权重。 |
| FLOPS | Floating-Point Operations | 浮点运算量 | SPLADE 中以预期计算开销为直觉的稀疏正则项,抑制过多词项被激活。 |
| T5 | Text-to-Text Transfer Transformer | 文本到文本迁移 Transformer | 将所有任务表述为文本到文本生成;doc2query 可用它离线生成文档扩展词。 |
| LTR | Learning to Rank | 学习排序 | 从 labels、preferences 或 lists 学习候选文档的排序函数。 |
| OLTR | Online Learning to Rank | 在线学习排序 | 通过实时用户交互持续评价和更新 ranker。 |
| IPS | Inverse Propensity Scoring | 逆倾向评分 | 用观察概率的倒数重新加权点击,以校正曝光或位置偏差。 |
| DBGD | Dueling Bandit Gradient Descent | 对决老虎机梯度下降 | 用 interleaving 比较当前与探索 ranker,并向获胜方向更新。 |
| COLTR | Counterfactual Online Learning to Rank | 反事实在线学习排序 | 使用记录了 propensities 的历史点击数据离策略评估候选 rankers。 |
| RM3 | Relevance Model 3 | 相关性模型 3 | 从初始检索结果估计反馈模型并扩展原 query。 |
| TREC | Text REtrieval Conference | 文本检索评测会议 | 提供标准 collections、topics 和 evaluation tracks 的信息检索评测体系。 |
| Qrels | Query Relevance Judgements | 查询相关性判断 | 记录 query-document relevance labels 的标准答案文件,用于评价 run。 |
| MS MARCO | Microsoft Machine Reading Comprehension | 微软机器阅读理解数据集 | 由真实搜索查询和人工答案/段落构成,常用于训练与评价 neural rankers。 |
| BEIR | Benchmarking Information Retrieval | 信息检索基准套件 | 跨多个领域测试 retriever 的 zero-shot generalisation。 |
| JSONL | JSON Lines | 逐行 JSON 格式 | 每一行保存一个独立 JSON object,适合流式读取文档集合。 |
| LoRA | Low-Rank Adaptation | 低秩适配 | 冻结大部分原模型参数,只训练低秩更新矩阵的高效微调方法。 |
| MNTP | Masked Next Token Prediction | 掩码下一词预测 | 通过遮住 token 并利用双向上下文训练 decoder 表示的目标。 |
| SimCSE | Simple Contrastive Learning of Sentence Embeddings | 句向量简单对比学习 | 以对比目标学习句子表示的方法。 |
| PARADE | Passage Representation Aggregation for Document Reranking | 面向文档重排序的段落表示聚合 | 先编码长文档的各 passages,再聚合其 representations 做文档级评分。 |
| LLM4IR | Large Language Models for Information Retrieval | 用于信息检索的大语言模型 | 使用 LLM 改善 retrieval、reranking 或其他 IR tasks。 |
| IR4LLM | Information Retrieval for Large Language Models | 用于大语言模型的信息检索 | 通过 retrieval 为 LLM 提供外部、可更新的证据。 |
| monoBERT | Model name, not an acronym | 单文档 BERT 重排器 | 对每个 query-document pair 独立输出 pointwise relevance score。 |
| duoBERT | Model name, not an acronym | 双文档 BERT 重排器 | 比较两个 candidates,输出对 query 的 pairwise preference。 |
| RAG | Retrieval-Augmented Generation | 检索增强生成 | 在生成前从外部 collection 检索证据,并把 query 与 evidence 一起交给生成模型。 |
| LC-LLM | Long-Context Large Language Model | 长上下文大语言模型 | 能够接收远长于传统模型输入窗口的 prompt,但成本与位置偏差仍需考虑。 |
| CiC | Corpus-in-Context | 语料库置于上下文 | 把整个可容纳 corpus 与示例放入 LC-LLM prompt,并生成相关 document IDs。 |
| CoT | Chain-of-Thought | 思维链提示 | 通过示例或指令诱导模型生成中间推理步骤;生成文字不等于可验证的内部推理。 |
| VLM | Vision-Language Model | 视觉语言模型 | 联合处理像素与文本;screenshot retrieval 用它保留页面布局和视觉内容。 |
| HyDE | Hypothetical Document Embeddings | 假想文档嵌入 | 先生成一个假想相关文档,再用其表示检索真实文档。 |
| RARR | Retrofit Attribution using Research and Revision | 通过研究与修订补加归因 | 为已有答案搜索支持证据并修正无支持内容。 |
| PPO | Proximal Policy Optimization | 近端策略优化 | 一种限制单次策略更新幅度的强化学习方法;可训练 query rewriter。 |
| Attribution | Evidence attribution | 证据归因 | 把生成答案中的 claims 与真实、支持它们的 references 或 citations 对齐。 |
作业不是只写一个 BM25 函数,而是完成一个可验证、可调参、可比较的 IR 实验。
Train qrels 用来选择 k1、b;test qrels 只用于最终一次无偏评价。若看着 test 结果反复改参数,就把 test 泄露成了 train,最终分数会过于乐观。
不要只写“分数提高”。解释 k1 改变 tf saturation、b 改变 length normalization,并结合 gain/loss queries 判断数据集是偏短文档、长文档,还是 query 中 rare terms 的作用更明显。
# 一个清晰的 scorer 骨架(变量名按你的 notebook 调整)
def bm25(tf_vector, df_vector, doc_len, k1=1.2, b=0.75):
score = 0.0
for tf, df in zip(tf_vector, df_vector):
idf = math.log(N / (1 + df))
norm = 1 - b + b * (doc_len / average_doc_length)
tf_weight = ((k1 + 1) * tf) / (tf + k1 * norm)
score += idf * tf_weight
return score
先在脑中回答,再展开答案。能够解释“为什么”才算掌握。
内容来自你提供的本地 INFS7410 课程目录;大体积 corpus、index、venv 只作为实验资源,不纳入复习正文。
按周查找 67 个复习主题:依据课程知识点,用重新组织的场景与计算示例练习,不是官方原题或官方答案册。点击主题后先尝试作答,再展开讲解;不公开课件的原始题目文字。