INFS7410 Revision Hub

Information Retrieval · Weeks 1–13 · Practicals · Projects

A1 EN
Complete review map

From term matching to evidence-grounded generation

This guide follows one connected Information Retrieval (IR) pipeline: represent and index documents, retrieve candidates, rerank them, evaluate the ranking, learn from interaction data, and provide evidence to a generator. 中文主线:表示文本 → 召回候选 → 神经重排序 → 离线/在线评价 → 检索增强生成 → 证据归因。

文本 → 词项 词项 → 索引 索引 → 排名 排名 → 评价 稀疏 → 稠密语义 Learned sparse → 混合词汇与语义 Qrels → Clicks / 在线反馈 Candidates → BERT / LLM reranking Evidence → RAG answer Claims → Attribution Observation → Next search / 证据反馈 Web graph → Authority / 权威度 Ranked list → Diverse intents / 多意图 Feedback → Query reformulation / 查询重构 Multiple runs → Fused ranking / 融合排名 Index → Fast top-k / 快速 top-k
13Weeks connected / 周次
30+Key formulas / 公式
35+Retrieval, ranking & RAG methods
Practicals代码练习与作业
Weekly practical map

Lecture + practical / 每周讲义与练习

把讲义概念与对应练习放在一起:每周沿着任务、数据流、代码步骤和检查点学习。

Week / LecturePracticalConcept bridge
Week 1Text analysis and indexing / 文本处理与索引Tokens, tf/df/cf, inverted indexes and Zipf
Week 2From runs to defensible comparisons / 从检索结果到可信的比较Qrels, ranking measures and paired tests
Week 3Implementing lexical scoring / 把词项公式变成打分代码TF-IDF, BM25 saturation and candidate limits
Week 4Follow the Transformer tensors / 跟踪 Transformer 中的数据形状Attention, tensor shapes, pooling and prediction
Week 5Single-vector, multi-vector and decoder retrieval / 三种稠密表示的实际比较Bi-encoders, late interaction and decoder embeddings
Week 6Inspect learned term weights / 理解学习得到的稀疏词项权重Contextual weighting, exact matching and expansion
Week 7Learn from labels, then from clicks / 先用标签学习,再用点击学习Pointwise loss, clicks, interleaving and DBGD
Week 8Neural and LLM reranking / 神经模型与 LLM 重排序Cross-encoding, answer-token scores and pairwise ranking
Week 9Build and test evidence-grounded RAG / 构建并检查有证据支持的 RAGRAG, attribution, missing evidence and controlled prompts
Week 10From a retrieval call to a search–read loop / 从一次检索到搜索与阅读循环Policy, tools, observations, state and budgets
Week 11PageRank as a probability update / 把 PageRank 理解为概率更新Random surfer, stochastic matrices and convergence
Week 12Reformulate queries and fuse rankings / 查询重构与排序融合实践PRF, IDF-r, score normalisation and rank fusion

资料核对:2026 年 10 月 20 日。补齐 Week 5、7、9、11、12 的课程材料,并更新 Week 8;其余周次使用下方标明的本地归档。本页是学习讲解,不代表重新运行了全部模型实验。

Mental model

One system, three connected layers

把每周知识放回同一个 pipeline:retrieval 决定“看哪些文档”,ranking 决定“先看哪些”,evaluation/feedback 决定“如何判断并继续学习”。

Corpus
原始文档集合
→
Analyzer
分词、停用词、词干
→
Inverted index
term → postings
→
Ranker
BM25 / dense model
Query
用户信息需求
→
Top-k run
每个 query 的排名
→
Qrels
人工相关性判断
→
Evaluation
MAP / nDCG / test
四个经常混淆的对象:corpus 是所有文档;query 是用户输入;run 是系统产生的排名;qrels 是标准相关性答案。评价是在 run 与 qrels 之间进行。

1. Candidate retrieval / 候选召回

BM25, DPR, SPLADE and TILDE-family models search a large collection and return a manageable candidate set.

→

2. Ranking / 重排序

A cross-encoder or learned ranker uses richer features to reorder candidates. Pointwise, pairwise and listwise describe different learning targets.

→

3. Evidence / 评价证据

Offline qrels support nDCG/MAP. Online clicks require bias correction, interleaving or counterfactual estimators.

Data relationship: corpus supplies documents; a query triggers retrieval; a run stores the produced order; qrels provide explicit labels; click logs provide abundant but biased implicit feedback. 前四者支撑离线实验,点击日志把课程带到 Week 7 的在线学习。
Week 1

索引、文本统计与预处理

目标:理解搜索引擎如何把大量文本变成可以快速查找的数据结构。

IR、数据库与 relevance

数据库查询通常要求精确匹配并返回确定答案;IR 面对自然语言和不确定的信息需求,因此返回按相关性排序的结果。Relevance 不是单纯“出现了相同词”,而是文档是否满足该用户在该情境下的信息需求。

主线:representation(怎样表示文本)+ matching(怎样比较)+ ranking(怎样排序)+ evaluation(怎样证明更好)。

Corpus、document 与 term

Corpus / collection 是全部文档;document 是一条独立记录;token 是分词后的一次出现;term 是索引使用的规范词项。一个 term 可以在同一文档中出现多次。

tf某词在一个文档中的次数 df包含该词的文档数 cf该词在整个集合中的总次数

Zipf's law

少数词极其频繁,大多数词很少出现。按频率排序后,rank 越高,frequency 越低,近似呈幂律。

frequency(r) ≈ C / rs

图上会出现长尾:最高频词通常是功能词;最低频词可能是专名、拼写变体或噪声。这解释了 stopwords 与词表压缩的动机。

文本处理

  • Tokenization:把文本切成可索引单元。
  • Case folding:通常统一为小写。
  • Stopping:移除高频、区分度低的词,但可能损失短语含义。
  • Stemming:用规则截断为词干,例如 Porter stemmer;不保证是真实单词。
  • Lemmatization:利用词法知识还原词元,通常更精确但更昂贵。

Inverted index

正向存储是 document → terms;倒排索引反过来保存 term → postings list。查询词出现在哪些文档,可直接读取对应 postings,不需要逐篇扫描 corpus。

层级posting 保存什么支持的能力
BooleandocID判断是否出现,做 AND / OR / NOT
CountsdocID + tfTF、TF-IDF、BM25 排名
PositionsdocID + tf + positions短语查询、邻近查询、高亮
Doc vectors每篇文档的 term 与统计课程中自定义 scorer 读取 tf/df;A1 要求 -storeDocvectors
易错:df 不是某词出现总次数。某个词在一篇文档中出现 20 次,这一篇对 df 的贡献仍然只是 1,但对 cf 的贡献是 20。
Week 1 practical 在训练什么
读取 JSONL corpus、统计文档数、检查文档结构、观察词频分布与 Zipf 长尾、建立带不同 stemming/stopword 配置的 Lucene index,并通过 index_reader.stats() 验证索引。
Week 1 · Practical

Text analysis and indexing / 文本处理与索引

Turn a JSON Lines (JSONL) collection into an inverted index, then use real term statistics to explain Zipf’s law and the effects of preprocessing.把逐行 JSON 文档变成倒排索引,再用真实词项统计理解 Zipf 定律以及预处理的影响。索引是后续检索的基础,不是评价结果。

Data flow: JSONL documents → analyzer → Lucene index → term statistics → frequency plots → reindex and compare

  1. Inspect the collection

    One JSONL line represents one document. Parse it, identify the ID and text fields, and count documents separately from tokens. A document ID connects later search hits to the same original text.每行是一篇文档。先确认字段,不要把行号当成 docid。文档数、所有 token 出现次数、不同 term 数是三个不同统计量。

  2. Build the unprocessed baseline

    The supplied configuration keeps stopwords and disables stemming. Tokenisation still occurs: “noProcessing” is a folder label, not a claim that raw strings are indexed without an analyzer. Inspect index_reader.stats() after indexing.noProcessing 不等于完全不分词。先记录 analyzer 设置,再检查索引中的文档数和词项数;查询端之后必须使用匹配的处理规则。

  3. Plot the frequency distribution

    Sort terms by collection frequency (cf), then put frequency rank on the horizontal axis. The high-frequency head and long tail illustrate Zipf’s law. Compare the >50,000-frequency terms with the least frequent 20 terms; they expose different parts of the distribution.按全集合出现次数排序,不是按字母排序,也不是按 df 排序。极少出现的词可能是专名或噪声,不能看到低频就直接删除。

  4. Change one representation choice

    Build a second index using Porter stemming and stopword removal. Recompute the statistics instead of reusing the previous vocabulary. Fewer distinct forms may improve matching, but can also merge different meanings.同一 collection,改变预处理,再观察词表和分布。是否提高检索效果要等 Week 2 用 qrels 验证;词表变小本身不等于效果更好。

Worked example / 拆开看一个例子

If a term occurs 7 times in one document and 3 times in another, its cf is 10 and df is 2. Its tf in the first document is 7. APIs sometimes call collection frequency “term frequency”; interpret the scope of the returned statistic.同一个词:文档内 tf=7;出现过该词的文档数 df=2;整个集合的总次数 cf=10。看清 API 的统计范围比只记变量名重要。

Check your understanding: What should be retained for Week 2?

Both index paths, their analyzer settings, stable document IDs and the same collection. Week 2 adds queries and qrels to test effectiveness, rather than judging quality from vocabulary size.两个索引、各自预处理设置和稳定 docid。Week 2 比较的是检索结果,不是看哪个索引词更少。

Source map: prac-week1.ipynb: Exercises 1–2 and reindexing comparison. Independent study explanation; original files remain on the course platform.

Back to Week 1 lecture concepts
Week 2

评价:怎样证明一个系统更好

离线评价需要 queries、qrels 和 runs。不同指标体现不同用户行为,不能只记公式。

Precision 与 Recall

Precision = |Relevant ∩ Retrieved| / |Retrieved|
Recall = |Relevant ∩ Retrieved| / |Relevant|

Precision 关心“返回的结果有多干净”;Recall 关心“所有相关文档找回了多少”。搜索引擎首页更偏 precision,法律检索或医学系统常更重 recall。

F-measure

Fβ = (1 + β²)PR / (β²P + R)
F1 = 2PR / (P + R)

F1 是 precision 与 recall 的调和平均;β>1 更重 recall,β<1 更重 precision。

AP 与 MAP

AP = (1 / |Rel|) Σk=1..K P@k × rel(k)
MAP = (1 / |Q|) Σq∈Q AP(q)

只有在第 k 位相关时,P@k 才会进入 AP。没有被检索到的相关文档仍留在分母 |Rel| 中,因此会降低 AP。

RR 与 MRR

RR(q) = 1 / rank(first relevant result)
MRR = meanq RR(q)

只关心第一个相关答案在哪里,适合问答或导航型查询;不会奖励第二、第三个相关结果。

DCG 与 nDCG

DCG@K = Σk=1..K (2rel(k) − 1) / log2(k + 1)
nDCG@K = DCG@K / IDCG@K

高相关等级获得指数增益,排名越靠后获得折扣。IDCG 是把同一 query 的结果按理想相关性排序后的 DCG,因此 nDCG 通常在 0–1。

RBP

RBP = (1 − p) Σi=1..d ri pi−1

p 是用户继续查看下一条的概率,期望查看深度是 1/(1-p)。p 越大,越重视更深的排名。

Paired t-test

同一批 queries 分别由系统 A 和 B 得到分数,先计算每个 query 的差值,再检验差值均值是否显著偏离 0。

dq = metricB,q − metricA,q
H₀: mean(d) = 0

p < .05 常称 statistically significant;p < .01 可称 strongly significant。

怎样解释 p-value

p-value 是:假设 H₀ 成立时,观察到当前或更极端差异的概率。它不是“H₀ 为真的概率”,也不代表效果大小。

先确认键存在:KeyError: ndcg_cut_5 通常意味着 evaluator 初始化时没有请求这个 measure,而不是 t-test 本身出错。
Week 2 practical 的代码链
search(..., k=1000) 生成 run → pytrec_eval.parse_run 读取 → evaluator 对每个 query 计算指标 → aggregate 得到总体均值 → scipy.stats.ttest_rel 比较两个系统的逐 query 分数。这里的 k 是每个 query 保留的 top-k 文档数,不是 BM25 的 k1。
Week 2 · Practical

From runs to defensible comparisons / 从检索结果到可信的比较

Compare two analyzer configurations and two retrieval depths using the same queries and judgments. Learn what a metric, a significance test and a plot each contribute.固定 queries 和人工相关性判断,比较两套索引与两个召回深度。指标、显著性检验、图表分别回答不同的问题。

Data flow: queries.tsv + index → ranked run → qrels join → per-query measures → paired differences → plots

  1. Retrieve and preserve identity

    Search both Week 1 indexes with their matching query analyzers. Retain qid, docid, rank and score. Retrieval depth k=10 or k=1000 is not the BM25 parameter k1 and is not automatically the evaluation cutoff.qid 连接 query 和 qrels,docid 连接结果和文档。召回 1000 条,仍可只计算前 10 条的 nDCG@10。

  2. Separate set and rank measures

    Precision and recall describe the returned set. Average Precision (AP) and normalised Discounted Cumulative Gain (nDCG) also reward ordering. Mean Average Precision (MAP) averages AP across queries. IDCG uses the query’s available judgments, including relevant documents not retrieved.调换相同集合的顺序不改变整体 precision/recall,但可能改变 AP、nDCG 和截断指标。不能只用返回结果里的相关文档建立理想排序。

  3. Align paired observations

    Compute one metric value per qid per system. Align arrays by the same qid order before ttest_rel. A small p-value is evidence against the no-mean-difference hypothesis under the test assumptions, not the probability that the null hypothesis is true.配对单位是同一 query,不是两张随机顺序的分数表。p-value 不告诉你提升有多大,也不能保证换一批 query 仍然提升。

  4. Explain the distribution

    For each query, compute candidate minus baseline. Positive bars are gains; negative bars are losses. Use mean scores, a gain–loss plot and a distribution plot together, then inspect documents for a large gain and a large loss.平均值可能掩盖部分 query 的退步。先说明差值方向,再回到 query 和文档解释原因,不要仅写“柱子更高”。

Worked example / 拆开看一个例子

The four conditions are unprocessed/processed × top-10/top-1000. Hold depth fixed when comparing analyzers. For a deterministic ranker, merely extending the same ranking from 10 to 1000 should not alter its first 10 results; changes to nDCG@10 then require investigation.四组实验可以组成 2×2 对照。比较预处理时固定深度;比较深度时固定模型。只扩展结果列表,不应凭空改变其前十名。

Check your understanding: Is a statistically significant gain necessarily a large or universal gain?

No. Report the effect size and query-level failures as well. Ensure requested measures exist in RelevanceEvaluator before looking up keys such as ndcg_cut_10.不是。还要说明提升幅度和退步的 queries;KeyError 可能只是没有请求对应评价指标。

Source map: prac-week2.ipynb: Exercises 0–4. Independent study explanation; original files remain on the course platform.

Back to Week 2 lecture concepts
Week 3

词项匹配与经典排序模型

从“是否包含查询词”逐步增加词频、稀有性、长度归一化和概率解释。

原始课程图示:前往课程平台查看(需要登录)

模型核心想法公式 / 作用主要限制
Boolean满足逻辑条件即匹配AND / OR / NOT postings通常不产生细致排名
Coordination匹配更多不同 query terms 更好matched query terms 数忽略同一词出现次数和稀有性
TFquery term 在文档中出现更多更好Σ f(qi,D)常见词和长文档占便宜
IDF少见词区分度更高log(N/(1+dfi))不看当前文档中的 tf
TF-IDF局部频率 × 全局稀有性Σ tfi,D × idfitf 线性增长、长度处理粗糙
VSM把 query/doc 当向量比较方向cosine(q,d)仍依赖词项重合
BIM估计词在 relevant/non-relevant 中的区别概率相关性权重词项独立假设;需估计概率
BM25IDF + tf saturation + length normalization经典强稀疏基线无法天然解决同义词和语义匹配

课程版 TF、IDF、TF-IDF

TF(D,Q) = Σi=1..|Q| f(qi,D)
IDF(qi) = log(N / (1 + dfi))
TF-IDF(D,Q) = Σ tfi,D × idf(qi)

1 + df 是平滑,避免 df=0 时除零。IDF 只依赖 term 和整个 collection,不依赖当前 document。

Vector Space Model

cos(q,d) = (q · d) / (||q|| ||d||)

点积受向量长度影响,cosine 先归一化,比较“方向”。若向量已 L2-normalized,则 dot product 与 cosine 相同。

BM25:逐部分理解

score(D,Q) = Σi IDF(qi) × [(k₁+1)tfi] / [tfi + k₁(1 − b + b·|D|/avgdl)]
  • IDF:区分 rare term 与 common term。
  • TF saturation:tf 从 1 到 2 的增益大于从 100 到 101;避免重复堆词无限加分。
  • Length normalization:长文档本来有更多机会包含任何词,因此按 |D|/avgdl 调整。
  • k1:控制 tf 多快饱和;越大越接近线性 tf。
  • b:控制长度归一化;0 表示不考虑长度,1 表示完全按相对长度调整。
RSJ-IDF ≈ log((N − df + 0.5) / (df + 0.5))

讲义中的完整 BM25 还可包含 query term frequency 因子;A1 的模板采用课程指定的简化 IDF。作业时以 notebook/marking sheet 的公式为准,不要自行混用不同 BM25 版本。

Week 3 practical 的 scorer 输入为什么是三个参数
老师的 helper 实际调用 scorer(tf_vector[term], df_vector[term], doc_len),传入的是一个词的 tf、这个词的 collection df,以及当前文档长度。前两项已经用下标取成数字,不是整个向量。你的 scorer 返回该词的贡献,外层 search helper 再对匹配的 query terms 累加,然后排序候选文档。
三个 assert 在验证什么
assert score(tf=3) > score(tf=1) 检查更多词频是否提高得分;assert score(short_doc) > score(long_doc) 检查长度归一化;assert np.isfinite(...) 检查没有 NaN/∞。assert 条件为 False 时立即抛出 AssertionError,因此它是一种小型自动测试。
Week 3 · Practical

Implementing lexical scoring / 把词项公式变成打分代码

Implement Term Frequency (TF), Inverse Document Frequency (IDF), TF-IDF and BM25 (Best Matching 25), then compare them through Week 2’s evaluation workflow.把公式拆成可检查的单词贡献,再累加为文档分数。这里的检索模型产生排名,评价指标才负责判断排名质量。

Data flow: index statistics + query terms → per-term scorer(tf, df, doc_len) → summed document score → reordered candidates → run

  1. Read the index statistics

    The practical reindexes with document vectors and positions. The document vector supplies term counts; summing its values gives the analyzed document length. Collection document count N and average document length avgdl must come from the same index.不要把字符数当 doc_len,也不要混用另一套 analyzer 的 avgdl。这里的长度是分析后词项出现次数之和。

  2. Trace the function call, not just its name

    The checked helper calls scorer(tf_vector[term], df_vector[term], doc_len). Despite the parameter names, each call receives scalar tf and df for ONE matched term. The search helper owns the outer summation over matching terms.老师变量名带 vector,但实际传入的是下标取出的数字。你写的是一个词的贡献,不能再对数字 zip,也不能重复累计整条 query。

  3. Translate the formula

    TF returns tf; IDF returns log(N/(1+df)); TF-IDF multiplies them. BM25 adds tf saturation and a length-dependent denominator. Return a numeric value, and test the one-term case before processing every query.先算 IDF、长度归一化、词频饱和,再相乘。测试 b=0 时长度惩罚消失;正 IDF 的固定词项下,更多 tf 的收益应逐渐变小。

  4. Know the candidate boundary

    The supplied searcher first obtains k candidates, then your scorer reorders those IDs. This is a candidate-pool comparison, not exhaustive retrieval with each custom formula. Reordering an unchanged complete top-10 set cannot change its P@10 or R@10, although AP/nDCG can change.同一批十篇文档只换顺序,前十篇的相关数量不变。若换了候选集合或只取更小 cutoff,precision/recall 才可能变化。不要把受候选限制的实验说成全库检索。

Worked example / 拆开看一个例子

For a matched term with tf=3, df=9, N=100, doc_len=avgdl, k1=1.2 and b=0.75: IDF=ln(10); the tf factor is 6.6/4.2; the contribution is about 3.61835. Only the outer helper adds other query terms.文档长度等于平均长度时,长度因子为 1。算出的约 3.61835 是一个词的贡献,不是所有 query terms 的最终总分。

Check your understanding: Why can your BM25 differ from built-in BM25?

The IDF convention, analyzer, query-term treatment, candidate set and parameter values may differ. Check these before assuming the difference is a coding error or a quality improvement.同名 BM25 可能采用不同 IDF 版本;先对齐公式与设置,再解释分数差异。

Source map: prac-week3.ipynb: Exercise 0, ranking functions 1.1–1.4, Exercise 2. Independent study explanation; original files remain on the course platform.

Back to Week 3 lecture concepts
Week 4

Transformer:从 token 到 contextual embedding

经典稀疏模型要求词项重合;Transformer 学习上下文相关表示,为语义匹配提供基础。

Tokenization 与 embedding

Tokenizer 把文本映射成 token IDs;nn.Embedding(V,D) 像一张可训练查找表,把每个 ID 映射为 D 维向量。D 是模型设计参数,例如 BERT-base 的 hidden size 是 768。

inputi = token_embedi + position_embedi + segment_embedi

形状常写成 N × D:N 是 tokens 数量,D 是每个 token 的表示维度。

Position 与 segment

纯 self-attention 不知道顺序,所以加入 position embedding。BERT 处理句对时,segment/token-type embedding 区分 A 段与 B 段;[SEP] 是边界 token。

WordPiece:未知或低频词可拆成子词,## 表示该片段接续在前一个子词后面。

Self-attention

Q = XWQ,   K = XWK,   V = XWV
Attention(Q,K,V) = softmax(QKT / √dk)V
  1. 每个 token 产生 query:我想找什么;key:我能提供什么;value:我的内容是什么。
  2. QKᵀ 产生 N×N 分数矩阵,表示每个 token 对所有 token 的关注程度。
  3. 除以 √d_k 防止维度较大时点积过大、softmax 过度饱和。
  4. softmax 把每行变成和为 1 的权重;再乘 V 得到融合上下文的表示。

Multi-head attention

多个 attention heads 在不同投影子空间中并行建模关系,例如一个 head 偏语法依赖,另一个偏实体关系。各 head 输出拼接后再投影回 D 维。

FFN、hidden 与 residual

hidden = GELU(W₁x + b₁)
out = W₂hidden + b₂

FFN 对每个 token 独立地做非线性特征变换,通常 D→4D→D。这里 hidden 只是中间激活张量,不是手工设置的“参数”;真正训练的参数是 W、b。Residual 保留原信息,LayerNorm 稳定训练。

Encoder 与 BERT

Encoder 可以同时看到左右上下文,适合分类、理解、ranking 和 embedding。BERT 以 masked language modelling 为主要预训练目标。

LMLM = −Σi∈M log P(xi | x\M)

Decoder 与 GPT

Decoder 使用 causal mask,只能看当前 token 左侧,从而逐 token 生成。Encoder backbone 不能直接像 decoder 一样自然生成,需要额外 decoder 或改造目标与 attention mask。

P(y₁…yT) = Πt=1..T P(yt | y<t)
Week 4 practical 中 NoneType.shape 的原因
若 out = tf_block(x) 得到 None,说明 forward() 或其中调用的方法漏写了 return x/return out。Python 函数没有 return 时默认返回 None,因此 out.shape 会报错。
Week 4 · Practical

Follow the Transformer tensors / 跟踪 Transformer 中的数据形状

Build the simplified Transformer layer, inspect BERT (Bidirectional Encoder Representations from Transformers), and contrast it with GPT (Generative Pre-trained Transformer) next-token prediction.这一周先理解网络如何生成 contextual embeddings;能生成向量不代表已经训练成检索模型。保留了本地 starter 与 solution,版本标记不冒充最新验证。

Data flow: token IDs → embeddings [B,L,H] → attention and FFN → contextual states → pooling or vocabulary head

  1. Project and attend

    The small layer uses x shaped [N,D] without an explicit batch axis. Q=XWq, K=XWk and V=XWv keep [N,D]; QKᵀ becomes [N,N]. Softmax over keys makes each query-token row sum to 1. Multiplying by V returns [N,D].N 是 token 数、D 是隐藏维度;attention 矩阵描述 token 对 token,不是文档对文档。softmax 的维度选错,就会改变权重的含义。

  2. Preserve shape through the block

    The Feed-Forward Network (FFN) changes D→4D→D with a nonlinear activation between projections. Residual addition requires matching dimensions. Return the output from both SelfAttention and FFN; a missing return propagates None.扩展维度是中间计算,输出要回到 D 才能与输入相加。NoneType.shape 往往是函数没有返回 tensor,而非维度真的不存在。

  3. Inspect BERT outputs

    Token IDs have shape [B,L]; last_hidden_state has [B,L,H]. CLS pooling or mask-aware mean pooling produces [B,H]. Mask padding in mean pooling. Compare the embedding module output before the Transformer with the contextual output after its layers.B 是 batch 数,L 是 token 长度,H 是 hidden size。token ID 只是词表编号,embedding 是向量;经过上下文层后,同一个 token 可以获得不同表示。

  4. Compare prediction heads

    Masked Language Modeling (MLM) predicts a selected masked position from bidirectional context. GPT’s vocabulary logits at the final real input position predict the next token. Autoregressive generation appends a token and repeats.BERT 填空和 GPT 续写是不同任务。因果 attention 若行是 query、列是 key,则右上方未来位置被屏蔽,允许关注的区域通常在下三角。不要不看坐标就背“上三角”。

Worked example / 拆开看一个例子

A batch of 2 texts padded to 6 tokens has IDs [2,6] and, for H=768, hidden states [2,6,768]. Mask-aware mean pooling gives [2,768], not [2,6] or [768]. These pooled vectors can become retrieval inputs in Week 5 after appropriate model training.池化只消掉 token 这一维,不会消掉 batch,也不会把 hidden size 变成词表大小。Week 5 再学习如何让这些向量适合相关性比较。

Check your understanding: Does a high next-token probability prove document relevance?

No. A vocabulary prediction head is trained for token prediction. Retrieval relevance requires an appropriate representation, scoring rule and training/evaluation setup.不是。预测词的能力与判断某文档是否满足 query 的能力,需要通过任务与训练联系起来。

Source map: prac-week4.ipynb and prac-week4-solution.ipynb: Exercises 1–3; archived starter labels itself 2025.1. Independent study explanation; original files remain on the course platform.

Back to Week 4 lecture concepts
Week 5

Dense Retrieval、训练目标与负样本

把 query 和 document 映射到向量空间,使“car repair”也能接近“automobile maintenance”。

原始课程图示:前往课程平台查看(需要登录)

Bi-encoder

score(q,d) = φ(ηq(q), ηd(d))

query 和 document 分开编码。文档向量可以离线预计算并建立 ANN index,在线只编码 query,因此适合第一阶段 retrieval。

Cross-encoder

把 [query; document] 一起输入 Transformer,允许所有 token 深度交互,通常更准确但无法预先编码文档,适合 reranking 少量候选。

Pooling

  • CLS/first token:使用第一个特殊 token 的 contextual embedding。
  • Mean pooling:对有效 tokens 向量求平均。
  • EOS/last token:decoder 模型常用最后一个 token,因为它已看到之前全部上下文。
  • All-token:保留每个 token 的向量,如 ColBERT。

Similarity

dot(q,d) = Σ qidi
cos(q,d) = q·d / (||q|| ||d||)

若向量已做 L2 normalize,dot 与 cosine 的排名一致。题目中的 “mean pooling similarity” 不严谨:mean pooling 是聚合表示的方法,不是 similarity function。

InfoNCE / contrastive loss

L = −log [ exp(sim(q,d⁺)/τ) / (exp(sim(q,d⁺)/τ) + Σj exp(sim(q,d⁻j)/τ)) ]

目标是提高正样本相似度,同时相对于一组负样本降低其相似度。温度 τ 越小,softmax 越尖锐。In-batch negatives 会把同一个 batch 中其他 query 的正文档当作当前 query 的负样本,提高效率。

Multi-label margin loss

L = mean(d⁺,d⁻) max(0, 1 − [s(q,d⁺) − s(q,d⁻)])

只要正样本得分至少比负样本高 margin=1,该 pair 的损失就是 0。

ColBERT late interaction

MaxSim(q,d) = Σi=1..m maxj=1..n Eq,i · Ed,j

对每个 query token 找最相似的 document token,再把这些最大值相加。它不是单 query vector 与单 document vector 的一次点积。

原始课程图示:前往课程平台查看(需要登录)

负样本类型来源优点风险
Random negatives集合中随机文档便宜、稳定常常太简单,梯度很弱
In-batch negatives同 batch 其他正样本几乎零额外编码成本可能出现 false negatives
BM25 hard negativesBM25 高排但不相关的文档词汇相似、训练更有挑战依赖 qrels 完整性
ANN/self-mined当前 dense model 的近邻错误紧跟模型当前弱点计算和刷新索引昂贵
Smoke query / smoke test 是什么
Smoke test 是“先看系统能不能基本跑通”的快速检查,不负责证明质量。你用一个 smoke_query 分别运行 TF 与 BM25,只取 top-5,确认没有异常、返回结构正确、分数有限、两个模型可能给出不同排名。正式结论仍要用全部 test queries 和 qrels 评价。
Week 5 · Practical

Single-vector, multi-vector and decoder retrieval / 三种稠密表示的实际比较

Compare DPR (Dense Passage Retrieval), ColBERT (Contextualized Late Interaction over BERT) v2 and RepLLaMA. The practical is model inference and scoring, not a new full training run.本周正式 notebook 包括 DPR、ColBERTv2、RepLLaMA。之前只有 InfoNCE 等题解并不等于已经覆盖这份 practical。

Data flow: BM25 candidates → text encoding → single-vector dot product OR token-level MaxSim → descending ranking

  1. Use the supplied dense helper

    The classroom dpr_encode uses ielabgroup/StandardBERT-DR and the first-token hidden state, then flattens one example into [H]. Encode the query and each candidate and take their dot product. The helper truncates at 64 tokens; that is a classroom configuration, not a theoretical DPR limit.课程演示使用共享的 StandardBERT-DR 编码器,不要把它与所有 DPR 双编码器 checkpoint 的接口混为一谈。先看实际模型、pooling 和截断长度。

  2. Build BM25 + dense reranking

    Read raw text for each BM25 hit, keep docid attached to the vector/score, sort by the new score, and evaluate against the same qrels. Encoding a few candidates on demand teaches the pipeline; scalable dense first-stage retrieval instead pre-encodes the corpus and uses an index.practical 中先由 BM25 缩小范围;全库 dense retrieval 才需要对完整 corpus 建向量索引。不能把重排十篇称为全库 ANN 搜索。

  3. Compute ColBERT MaxSim

    For one pair, query vectors [m,h] times passage vectors [n,h] transposed give [m,n]. Take a maximum over valid passage tokens for EACH query token, then sum those maxima. Match the model’s masks and query augmentation when comparing with the supplied model score.先对每个 query token 找最匹配的文档 token,再求和。不是整个矩阵取一次最大值,也不是先把 token 向量平均。padding 和专用 query tokens 会影响是否复现得分。

  4. Inspect RepLLaMA’s representation

    The example loads a Llama backbone with a Low-Rank Adaptation (LoRA) adapter, uses query/passage prefixes and an end-of-sequence (EOS) token, then normalises the last-token vectors before a dot product. With unit-length vectors, this is cosine similarity.Decoder 在这里输出的是检索向量,不是回答文本。模型架构是 decoder 不代表此处一定在生成;任务角色要看输出与打分方式。

Worked example / 拆开看一个例子

Toy MaxSim matrix [[0.9,0.2,0.1],[0.3,0.8,0.4]] gives row maxima [0.9,0.8] and score 1.7. This can exceed 1 because it is a sum of token similarities, not a probability. The extra token-level detail costs more storage than one vector per passage.两个 query tokens 分别找到 0.9 和 0.8 的最佳证据,总分 1.7。更多细粒度信息带来更多存储与匹配开销。

Check your understanding: How do the negative-sample lecture exercises connect?

Information Noise-Contrastive Estimation (InfoNCE) and margin losses explain how useful embeddings are learned. In-batch and hard negatives determine what must be distinguished. They are training concepts; the current practical mostly loads already-trained models.训练目标解释向量为什么有区分力;practical 使用预训练好的权重来观察结果。false negative 指实际相关却被当作负样本,不是简单指分数较高。

Source map: prac-week5.ipynb: Exercises 1–3; local InfoNCE/MarginLoss/HardNegatives HTMLs are supplementary worked notes. Independent study explanation; original files remain on the course platform.

Back to Week 5 lecture concepts
Week 6

Learned Sparse Retrieval / 学习型稀疏检索

Keep vocabulary-aligned sparse representations and inverted-index efficiency, but learn term weights and expansion with neural models. 核心不是把向量变密,而是让稀疏词项权重具有上下文与语义。

The missing middle: BM25 is sparse and fast but depends on lexical overlap; DPR is semantic but uses dense ANN search. Learned sparse retrievers keep dimensions tied to vocabulary terms while learning which terms to activate and how strongly to weight them. 它们位于 BM25 与 dense retrieval 之间,试图同时保留可解释的词项维度和神经模型的语义能力。
FamilyQuery representationDocument representationMatching and costMain idea / 中文抓手
BM25observed query termsterm counts + corpus statisticsinverted index; very cheap onlineHand-crafted lexical weights / 人工公式
DPRone dense vectorone dense vectordot product + ANN; query encoder onlineSemantic neighbourhood / 向量语义近邻
TILDEv2token IDs only; stopwords removedcontextual weights for document tokens, often after offline expansionexact token-ID match; no query encoderLearned exact matching / 学习词权重,在线查询很轻
SPLADElearned sparse vocabulary vectorlearned sparse vocabulary vectorsparse dot product; encoder on both sidesImplicit expansion / 自动激活原文没有的相关词
PromptRepsdense hidden state + sparse next-token logitssame two representationshybrid normalized score; LLM inferenceZero-shot dense+sparse signals / 无需 retriever training

Vocabulary mismatch and expansion

A relevant document may say “automobile” while the query says “car.” Document expansion adds likely query terms before indexing; doc2query uses a sequence-to-sequence model such as T5 to generate likely queries for each document.

Cost placement: expansion is performed offline, so the larger index buys better recall without adding a query-time generation step. 中文:把计算成本提前到建索引阶段。

TILDE → TILDEv2

TILDE models document-side term likelihoods over the vocabulary. TILDEv2 instead produces contextual weights for actual or expanded document tokens, then scores only exact matches with query token IDs.

score(q,d) = Σt∈q maxi: token(di)=t wi,d

查询词若在文档 token 中出现多次,practical 取该词的最大 contextual weight;未匹配词贡献 0。

SPLADE and SPLADEv2

An MLM head maps contextual token states into vocabulary-sized term weights. ReLU removes negative activations; pooling combines token evidence. The sparse query and document vectors are scored by a dot product.

score(q,d) = vq · vd,   v ∈ R|V| and mostly zero

SPLADEv2 improves pooling/training, including max-pooling and distillation variants. Sparsity regularisation such as FLOPS/L1 controls how many vocabulary terms remain active.

Sparse does not mean non-neural

Sparse/dense describes the output representation, not whether BERT was used. SPLADE uses a Transformer but outputs a mostly-zero vocabulary vector; DPR also uses a Transformer but outputs a dense embedding.

常见误区:learned sparse 的“sparse”指多数维度为 0,并不表示它没有神经网络或不能表达语义扩展。

Week 6 practical: two-stage BM25 + TILDEv2

BM25 retrieve

Search the full corpus and keep top-k candidates. High recall and low cost matter here.

→

Read raw text

Use each hit's document ID and raw() content as the reranker's input.

→

TILDEv2 rerank

Score every query-document pair and sort candidates by the new score.

Why two stages? Running a richer model over the whole collection is expensive. BM25 narrows the search space; TILDEv2 spends extra computation only on plausible candidates. 第一阶段负责“不要漏太多”,第二阶段负责“把更相关的推到前面”。

How the supplied tildev2_scoring function moves data
Document text → tokenizer tensors → model.encode → document token IDs and learned weights. Query text → token IDs → remove stopword IDs. For each query ID, locate equal document IDs, take the maximum weight if it appears more than once, and sum across query terms. The returned scalar is a reranking score, not a probability.
How to inspect a SPLADE vector
Convert the sparse tensor to dense only for inspection, count non-zero dimensions, then use torch.topk and the tokenizer vocabulary to decode the strongest terms. Original terms reveal lexical evidence; activated related terms reveal learned expansion. Do not densify the whole collection in a real retrieval index.
Week 6 · Practical

Inspect learned term weights / 理解学习得到的稀疏词项权重

Connect neural context to lexical matching: TILDEv2 (Term Independent Likelihood moDEl v2) weights document tokens; SPLADE (SParse Lexical AnD Expansion) produces vocabulary-aligned sparse representations.学习型稀疏模型不是“没有神经网络”,而是输出的维度有词项含义。比较模型时要同时看输入、非零维度和匹配规则。

Data flow: document → contextual token weights; query → token IDs → exact matches → max per matching term → sum

  1. Compare counts with contextual importance

    The exercise contrasts terms in a passage: equal occurrence counts need not mean equal learned weights. Inspect the token IDs and weights together. Repeated tokens can have different contextual weights before aggregation.TF 只数出现次数,contextual weight 会考虑上下文。同一个词出现两次,两个位置的权重可能不同。

  2. Trace tildev2_scoring

    Tokenise the query without special tokens, remove its stop IDs, and locate equal document token IDs. Use the maximum document weight for each matched query ID; missing matches contribute zero. The supplied loop iterates query tokens, so repeated query IDs are counted repeatedly unless explicitly deduplicated.匹配的是相同 token ID,不是两个 dense vectors 的语义相似度。文档重复出现取 max;查询重复是否累计,要看具体循环,不能想当然。

  3. Rerank BM25 candidates

    BM25 provides candidate IDs and raw text. TILDEv2 supplies a new score for the same candidates. Keep the candidate set fixed for a reranking comparison. noExp identifies the no-expansion variant: contextual weighting alone does not invent every missing matching term.noExp 不表示没有训练,也不表示停用词。没有 expansion 时,查询词未出现在文档 token 中通常就没有对应匹配贡献。

  4. Decode a SPLADE representation

    For inspection, convert the small sparse output with to_dense(), count nonzeros and decode the top-10 weighted vocabulary IDs. A nonzero term absent from the original text demonstrates expansion. A sparse dot product compares query and passage on shared vocabulary dimensions.多数维度为零但维度仍对应词表。只为小例子 densify,不能因此在真实全库检索时把所有稀疏向量都展开存储。

Worked example / 拆开看一个例子

For one query term, document occurrence weights [0.2,0.8] contribute 0.8, not 1.0. A second matching term of weight 0.4 produces total 1.2. By contrast, SPLADE scoring multiplies query and document weights in shared vocabulary dimensions and sums them.TILDEv2 的逐词 max 权重与 SPLADE 的稀疏点积是不同规则;不要因为都叫 sparse 就套用同一评分函数。

Check your understanding: Why might token_ids, token_weights = model.encode(...) fail?

The local modeling.py and checkpoint interface may differ. Inspect the returned type and batch nesting, then match the course helper. Do not install an unrelated package named modeling or remove a dimension merely to suppress the exception.先检查真实返回结构以及 modeling.py 的版本。expected 2, got 1 可能涉及 batch 包装,不一定表示模型只输出一种信息。

Source map: prac-week6.ipynb: Exercise 1 TILDEv2, Exercise 2 SPLADEv2. Independent study explanation; original files remain on the course platform.

Back to Week 6 lecture concepts
Week 7

Learning to Rank, Click Bias and Online Evaluation

Weeks 1–6 mostly ask how to produce a ranking. Week 7 asks how to learn and evaluate when feedback comes from users rather than complete qrels. 中文重点:点击很多,但点击不等于相关。

Where LTR sits in the retrieval pipeline

Retriever

BM25 / dense / learned sparse returns candidates and retrieval scores.

→

Feature vector

BM25 score, neural score, freshness, authority, query-document features.

→

LTR ranker

Learns a function that orders candidates to maximise a ranking objective.

A retrieval model can itself be learned, but Learning to Rank (LTR) usually means learning how to combine signals and order an already manageable candidate set. retrieval 解决规模,LTR 解决候选内部的顺序。

Offline LTR targets

  • Pointwise: predict one document's relevance score or class independently.
  • Pairwise: learn that document A should rank above B.
  • Listwise: optimise the quality/order of the whole result list.

三者区别是训练目标的单位:单文档、文档对、整张列表。

Qrels versus click logs

Qrels are explicit, relatively clean labels but expensive, static, and limited. Clicks are abundant and current but implicit, noisy, and biased by what was shown and where.

No click ≠ non-relevant: the user may never have examined the document.

Three click biases

  • Position bias: high ranks are examined more often.
  • Selection bias: unshown documents cannot be clicked.
  • Presentation bias: snippets or visual style affect clicks.

随机噪声可能随数据增多而抵消;系统性 bias 不会自动消失。

Examination model

P(ci=1 | yi) = P(ci=1 | oi=1,yi) · P(oi=1 | ranki)

A click requires both examination o and attraction/relevance evidence y. The propensity P(o=1|rank) captures how likely a position is viewed.

Naive clicks versus IPS

Δnaive(f) = Σi λ(ranki) · ci
ΔIPS(f) = Σi [λ(ranki) / P(oi=1)] · ci

The naive estimator overvalues results placed where users look more often. Inverse Propensity Scoring (IPS) divides each click by its examination probability, giving rare observations more weight and cancelling position bias in expectation when assumptions are correct.

中文直觉:第 10 位本来很少被看到,所以在那里发生的一次点击携带的信息量比第 1 位点击更大。IPS 用 1/propensity 放大这类样本。代价是 propensity 很小时方差会变大。

Balanced interleaving

Maintain a scan pointer for each ranked list. Inspect the list with the smaller pointer, breaking ties with a randomly chosen starting ranker. Advance the pointer even for duplicates, but append only unseen documents. This balances inspected depths, not necessarily the number contributed by each ranker.

Advantage: both systems face nearly the same query, user and context. 比两组独立 A/B 流量更敏感。

DBGD

Dueling Bandit Gradient Descent samples a candidate parameter direction, compares the current and candidate rankers through interleaving, and moves toward the winner. Repeated duels estimate a useful gradient from user satisfaction.

Counterfactual OLTR

The deployed stochastic ranker logs displayed actions and their probabilities. The same interaction log is then reweighted with IPS to estimate the risk of many candidate rankers. A candidate replaces the current model when its estimated counterfactual risk is lower.

Logging policy

Sample rankings and record propensities.

→

Interaction log

Displayed documents, ranks, clicks/non-click losses.

→

IPS comparison

Estimate current and candidate risks from the same log.

Main advantage over regular interleaving: one logged randomized policy can evaluate multiple candidate rankers off-policy, instead of constructing a fresh interleaved list for every pairwise comparison. 更能复用历史数据,但依赖可靠 propensity、充分 exploration/support,并可能有高方差。

MethodData sourceWhat it estimatesStrengthMain limitation
Offline qrel evaluationqueries + qrels + runsMAP/nDCG/etc.controlled and repeatablelabels expensive and may become stale
Balanced interleavinglive clicks on one mixed listpairwise ranker preferencesensitive within-user comparisonfairness/correctness is not guaranteed in every case
DBGDrepeated interleaving outcomesimproving parameter directionlearns directly onlineexploration may expose poorer rankings
Counterfactual + IPSlogged clicks + propensitiesoff-policy candidate value/riskreuses logs; compares many candidatespropensity errors, support gaps, high variance
How examination propensities can be estimated
A randomized top-n experiment can shuffle positions, collect clicks by rank, and normalize the resulting examination frequencies. This is useful because IPS needs the probability that an item at a position was examined. The logging policy must retain enough exploration so candidate actions have support.
Why OLTR can create a self-confirming loop
A random click on an irrelevant result can make the model promote similar items. Higher positions then attract more examination and clicks, which the learner may misread as stronger relevance. Bias correction and controlled exploration are needed to keep exposure from becoming mistaken evidence.
Week 7 · Practical

Learn from labels, then from clicks / 先用标签学习,再用点击学习

Contrast offline Learning to Rank (LTR) with Online Learning to Rank (OLTR). Logistic regression learns from explicit labels; Duelling Bandit Gradient Descent (DBGD) uses comparisons inferred from clicks.同样是更新排序器,前者直接看到训练标签,后者只根据交互比较来决定更新方向。模拟器使用 qrels 不等于 ranker 直接读取答案。

Data flow: MQ2007 features → logistic relevance scores → query-wise ranking; candidate ranker → interleave → simulated clicks → preference → update

  1. Read the feature dataset

    Each MQ2007 row is a query–document pair with 46 features. The pointwise exercise maps labels 1 and 2 to 1. Preserve qid groups for ranking and evaluation; confirm the final query group is saved after parsing.输入不是原始文档文字,而是已经计算好的 46 维特征。二元化会丢失 1 和 2 的相关等级差异,需要说明评价使用哪套标签。

  2. Implement the offline update

    Compute p=sigmoid(Xθ), gradient Xᵀ(p−y)/m and θ_new=θ−α·gradient. Use the same old weights for every gradient component. Ranking by θᵀx or its sigmoid is equivalent because sigmoid is monotone.梯度衡量预测和标签的差异如何分摊到每个特征权重。先算全部梯度再更新,避免前面刚更新的权重污染后面分量。

  3. Interleave by inspected depth

    Keep a pointer into each original ranking. Inspect the shallower pointer; use one random starting choice to break ties. Advance the pointer even when a duplicate is skipped. This balances inspected depths, not strict alternation of successful insertions.重复文档只展示一次,但扫描位置仍要前进。不能用“每边轮流成功加入一篇”替代老师的 pointer 算法。

  4. Infer the preference

    If clicks exist, take the deepest displayed clicked document. Its better rank in the two original lists determines a common prefix cutoff. Count clicked IDs in each prefix; the exploratory list wins only if its count is larger. No clicks or a tie do not establish a winner.clicked_doc 必须保持显示顺序。截断位置是同一个值,再分别计票;不能仅按“哪边贡献了被点文档”来判断 balanced interleaving。

  5. Read the learning curve critically

    DBGD samples direction u and updates toward it only when the challenger wins. Compare perfect, navigational and informational click models using fixed settings and multiple seeds. The “perfect” simulator still has a probabilistic click for relevance grade 1; its name does not mean every relevant item is always clicked.单次 nDCG 曲线不保证每一步上升。区分训练目标、在线点击结果和测试 nDCG;测试标签用于监测,不应决定训练时的胜负。

Worked example / 拆开看一个例子

A=[a,b,c], B=[b,c,a], displayed clicks=[b,c]. The deepest click is c; its better original rank is 2. A’s top two receive one click and B’s top two receive two, so B wins. Inspect the helper’s random draw: np.random.randint(0,1) always returns 0 because the upper bound is exclusive; a two-outcome draw needs an upper bound of 2.这也是阅读老师提供的代码时应核验的边界:函数名和注释不保证实际随机。这里是学习提示,网站没有改动归档的老师原文件。

Check your understanding: Why can nDCG remain unchanged while weights change?

A small weight update can preserve every document’s ordering. Check weight differences and the training loss before diagnosing a frozen model. Conversely, an unimplemented update also leaves scores unchanged; inspect the computation, not only the plot.排序没变和参数没变不是一回事。先确认权重、损失和排名分别是否变化。

Source map: prac-week7.ipynb: pointwise Tasks 1–2 and online Tasks 1–2; scripts/ClickModel.py and InterLeaving.py. Independent study explanation; original files remain on the course platform.

Back to Week 7 lecture concepts
Week 8

Transformer-based Reranking: BERT and LLM Rankers

Week 8 moves from retrieving candidates to applying deeper query-document interaction. Encoder rerankers classify relevance; decoder-based Large Language Models (LLMs) can judge, compare or order passages through prompts. 核心关系:第一阶段保证候选覆盖,第二阶段用更昂贵的模型改善候选内部顺序。

Where neural reranking sits

Stage 1: retrieve

BM25 or another efficient retriever searches the full collection and returns top-k candidates.

→

Stage 2: interact

A cross-encoder jointly reads the query and each candidate, or an LLM receives them in a ranking prompt.

→

Final ranking

Sort by neural relevance scores, pairwise preferences or an output ordering.

Effectiveness-efficiency trade-off: deeper interaction usually improves ranking quality but increases latency. Reducing candidate count lowers cost, but a relevant document omitted by Stage 1 cannot be recovered by the reranker. reranker 只能重新排列已召回文档,不能救回没有进入候选集的文档。

monoBERT: pointwise cross-encoder

Bidirectional Encoder Representations from Transformers (BERT) receives [CLS] query [SEP] document [SEP]. The contextual [CLS] vector enters a classification layer that estimates whether the document is relevant.

s(q,d) = P(relevant = 1 | q,d)

Each candidate is scored independently, so monoBERT is pointwise. Training uses human positive judgements and commonly samples difficult negatives from BM25.

monoBERT training objective

L = -Σj∈Jpos log(sj) - Σj∈Jneg log(1-sj)

This is binary cross-entropy: reward high scores for relevant passages and low scores for negative passages. The score is useful for ordering candidates, but it should not automatically be interpreted as a perfectly calibrated real-world probability.

训练单位是一组 query-document pair;模型学的是相关/不相关分类信号。

BERT length limit: two aggregation families

FamilyWhat is dividedHow evidence is combinedMain idea / 中文抓手
BERT-FirstPDocument → passagesUse the first passage scoreCheap, but later evidence is ignored / 只看首段
BERT-MaxPDocument → passagesTake the maximum passage scoreOne strong passage can represent the document / 取最强证据
BERT-SumPDocument → passagesSum passage scoresAccumulates evidence but may favour many passages / 累加证据
BirchDocument → sentencesInterpolate first-stage score and weighted sentence scoresCombines retrieval and sentence evidence / 句子级聚合
PARADEDocument → passage representationsAverage, max, attention or Transformer aggregationCombine representations before the final relevance decision / 先聚合表示再评分
Do not confuse: score aggregation combines already-produced passage scores; Passage Representation Aggregation for Document Reranking (PARADE) combines passage vectors and can model relationships among passages.

duoBERT: pairwise reranking

After monoBERT narrows the list, duoBERT jointly reads the query and two documents and estimates p(dᵢ > dⱼ | q). It therefore learns a pairwise preference rather than an independent relevance score.

s(dᵢ) = Σj≠i p(dᵢ > dⱼ | q)

Aggregating pairwise wins produces the final order. More pair comparisons may improve quality, but encoder cost grows quickly.

Multi-stage tuning

BM25 → monoBERT → duoBERT exposes several tuning knobs: candidate depth at each stage, model size and number of pair comparisons. These provide a flexible quality-latency trade-off, but also make the system more complex to tune and operate.

候选越少越快,但召回风险越大;候选越多、比较越细,成本越高。

LLM reranking families

FamilyPrompt inputRanking signalCost and limitation
PointwiseQuery + one passageYes/No logits, generated label or relevance gradeSimple and parallelisable; absolute judgements may be poorly calibrated
PairwiseQuery + passage A + passage BPreference A > BRobust comparison, but many pairs require many inferences
ListwiseQuery + a list of passagesGenerate a permutation or ordered identifiersSees list context; generation and context length are costly
SetwiseQuery + a small set of passagesSelect the best passage using output logitsFewer comparisons than pairwise and avoids full list generation

Zero-shot in this lecture: once an instruction-tuned LLM is available, these ranking prompts do not require new contrastive retriever training. This does not mean the underlying LLM was never pretrained or instruction-tuned.

Why setwise can be efficient

Pairwise ranking compares two documents per call. Setwise ranking selects a winner from several documents at once, reducing the number of LLM inferences. Compared with listwise generation, setwise methods can use next-token logits instead of generating a complete permutation.

Prompt variation matters

Role instructions, output restrictions, formatting instructions and query/passage order can all change effectiveness. The slides report larger variation for pointwise and listwise prompts, with pairwise and setwise prompts generally more robust; behaviour also depends on model size, architecture and training.

Exam caution: a prompt published in a paper is not automatically the best prompt for every model or dataset.

Week 8 practical: two implementation pipelines

BM25 candidates

Use Pyserini to retrieve a small top-k and read each hit's raw passage text.

→

monoBERT scores

Tokenize each query-passage pair, obtain the relevant-class softmax score and sort descending.

→

Evaluate run

Write the reranked document IDs and compare the run with qrels.

The practical recommends beginning with only 10 or 20 candidates because monoBERT inference is expensive. The updated LLM section compares TinyLlama, MiniCPM and Qwen3-Reranker before the pairwise exercise. 原始 Yes token 概率与 Yes/No 归一化分数含义不同;比较排序而非跨模型分数大小,详细步骤见本周 Practical。

Data flow / 数据流:query → BM25 top-k → raw passage text → tokenizer → monoBERT or LLM score/preference → re-sort the same candidates → evaluation. Retrieval depth and evaluation cutoff are separate choices.
LLM4IR versus IR4LLM
Large Language Models for Information Retrieval (LLM4IR) uses LLMs to improve retrieval tasks, such as reranking passages. Information Retrieval for Large Language Models (IR4LLM) uses retrieval to supply external evidence to an LLM, as in retrieval-augmented generation. Week 8 practical focuses on LLM4IR reranking.
Why automatic prompt engineering is harder for ranking
A ranking demonstration must encode a query, several passage meanings, their relative relevance and an ordering; multiple passages may legitimately share a relevance level. This output structure is less unique and more complex than a single classification label.
Week 8 · Practical

Neural and LLM reranking / 神经模型与 LLM 重排序

Follow the same candidate pool through an encoder cross-encoder and decoder-based relevance scoring. Distinguish a score used for ordering from a calibrated probability of relevance.本地旧版缺少 MiniCPM 与 Qwen3 的后续比较;本次已按老师更新版补齐。重点不是比较不同模型分数谁更大,而是它们把哪些文档放到前面。

Data flow: BM25 top-k IDs → raw text → query–passage pair → relevance score → sorted run → same qrels

  1. Build BM25 + monoBERT

    BERT jointly encodes the query and one passage; a classification head supplies a relevant-class score. Start with 10 or 20 candidates as instructed. Sort by monoBERT, not by the old BM25 score, before writing the run.Cross-encoder 允许 query 与 document tokens 在网络中交互,因此不能像 bi-encoder 那样预先得到一个与 query 无关的最终文档得分。

  2. Score a general-purpose chat model

    TinyLlama and MiniCPM use the probability of a selected Yes token after a fixed prompt. Each model has its own tokenizer and chat template. Check whether Yes or leading-space Yes is one token and matches the answer format; keep the choice fixed across passages.相同字符串在不同 tokenizer 中可以对应不同 ID。必须先确定候选 token,不能看每篇结果再挑有利的 Yes 写法。多 token 候选需要完整序列概率。

  3. Avoid scoring padding

    The next-token distribution must come from the final real prompt position. The updated example scores passages individually to avoid right-padding mistakes. Passing an attention mask alone does not make logits[:,−1] point to the last real token of every right-padded example.最后一个数组位置可能是 PAD。模型设备、tokenizer、attention mask 和预测位置都需要对齐,才能比较输入的真实分数。

  4. Compare the dedicated reranker

    Qwen3-Reranker uses its task-specific format and designated yes/no logits. The notebook’s interface describes a logit difference z_yes−z_no; sigmoid maps that difference to exp(z_yes)/(exp(z_yes)+exp(z_no)) without changing order. Check the installed interface’s output convention.Yes/No 两项归一化不是全词表里某个 Yes 的概率。训练任务与模板也不同;归一化本身不保证模型判断更正确。

  5. Extend to pairwise judgments

    Provide the question and two labeled passages and request a preference. Reverse A/B positions and map the result back to the same document IDs to examine position bias. For many candidates, define ties and aggregation; pairwise preferences may be cyclic.pointwise 是每篇独立评分;pairwise 是两篇同时比较。不要把 A/B 标签当固定 docid,也不要默认所有比较都满足传递性。

Worked example / 拆开看一个例子

Toy raw probabilities: passage A has P(Yes)=0.02 and P(No)=0.001; B has 0.40 and 0.10. Raw Yes prefers B, but Yes/No normalisation gives A≈0.952 and B=0.800, reversing the order. A’s high normalised score hides that both selected tokens had little total probability.归一化分母因 passage 而异,所以排名可能反转。归一化值很高,不代表模型大概率真的会生成 Yes,也不能当人工 gold。

Check your understanding: What must stay fixed to test the effect of normalisation?

The model, prompt, evidence, answer-token choices and generation/scoring position. Change only raw-versus-normalised scoring. Compare ranked relevance against judgments rather than comparing raw values across models.控制同一模型、同一输入与同一 token,只改变评分规则,才能讨论归一化的影响。

Source map: prac-week8.ipynb: BM25 + monoBERT, TinyLlama, MiniCPM, Qwen3-Reranker, optional scoring comparison, pairwise exercise; refreshed 16 October 2026. Independent study explanation; original files remain on the course platform.

Back to Week 8 lecture concepts
Week 9

Information Retrieval in the Age of LLMs / LLM 时代的信息检索

Week 9 connects two directions: LLM4IR uses a Large Language Model to retrieve or rank; IR4LLM retrieves external evidence for generation. The lecture then asks when long context can replace retrieval, and why relevant evidence still may not produce a correct, grounded answer.

One map for the whole course / 全课程总图

Represent & retrieve

BM25, DPR, TILDEv2, SPLADE, CiC or a specialist retriever find candidate evidence.

→

Refine evidence

Rerank, rewrite, filter, summarise, fuse or compress before the prompt.

→

Generate & attribute

A frozen or adapted LLM answers, while citations and groundedness checks connect claims back to evidence.

关键区别:retrieval quality asks whether useful passages are ranked highly; answer quality asks whether the generator uses them correctly. A better nDCG score can help RAG, but it does not logically guarantee a better answer.

Prompting recap

Zero-shot gives instructions only; one-shot/few-shot add demonstrations; Chain-of-Thought (CoT) elicits intermediate reasoning tokens. Reasoning text is still generated output, not a transparent record of the model's internal computation.

示例改变的是上下文中的任务说明与模式,不等于重新训练模型参数。

LLMs as retrievers or rankers

An LLM can generate a dense/sparse representation, judge one passage, compare two passages, order a list, or select from a set. RepLLaMA and LLM2Vec produce dense representations; PromptReps and DiffRetriever explore alternative generated representations; pointwise/pairwise/listwise/setwise prompts produce ranking signals.

Memory hook: representation models decide where to search; rerankers decide how to reorder a candidate set.

Long-Context LLM (LC-LLM) versus RAG

DimensionLC-LLM / Corpus-in-ContextRetrieval-Augmented Generation
InputPut a very large corpus directly in contextRetrieve a small top-k evidence set
Main benefitOne consolidated model call; fewer pipeline boundariesScales to larger, changing corpora; lower prompt size
Main costAttention cost and long-prompt latency/memoryRetriever/reranker maintenance and cascading errors
Failure modePosition bias and lost-in-the-middleMissing, distracting or conflicting retrieved evidence
When attractiveCorpus fits context and repeated prefixes can be cachedCorpus is large, dynamic, or needs explicit evidence IDs

Corpus-in-Context (CiC, pronounced “seek”) assigns consistent document identifiers, places the corpus and corpus-grounded demonstrations in one prompt, and asks the LC-LLM to return document IDs. Prefix caching can encode a repeated corpus prefix once, but it does not remove positional bias.

Corpus-in-Context prompt design

  • Use new, unambiguous document identifiers and consistent formatting.
  • Ground every demonstration in the same corpus and request a fixed ID-list output.
  • Attach a reasoning example when the task requires multi-step evidence selection.
  • For multi-turn search, include previous queries and outputs consistently.

few-shot 示例数量增加可能改善效果,但 gold document 越靠后,检索质量可能下降。

Specialised retrieval interfaces

Screenshot retrievers encode rendered pages with a Vision-Language Model (VLM), preserving layout, figures and spatial signals lost by plain text extraction. DiffRetriever uses a diffusion language model to produce several representative tokens in parallel, then matches multiple query/document representations.

Do not infer “newer = always better”: compare representation, index, scoring cost, training data and task fit.

Basic RAG data flow / 基础 RAG 数据流

Query
information need
→
Retriever
top-k passage IDs
→
Prompt
query + evidence
→
LLM
answer + citations

RAG supplies knowledge that is newer, private, domain-specific or absent from pretrained weights. It differs from further language-model pretraining: the external collection remains separately updateable and the retrieved evidence can be inspected.

Project 2 connection: BM25, DPR and TILDEv2 produce top-10 evidence; the same LLM, prompt, settings and token budget must then isolate the effect of evidence condition.

Modular RAG: intervene before and after retrieval

StageMethodsWhat they changeRisk
Pre-retrievalrouting, rewriting, expansion, HyDEmake the query easier for the index to matchrewrite may drift from the original need
RetrievalBM25, dense, learned sparse, hybridchoose the candidate evidence setrelevant evidence may never enter top-k
Post-retrievalreranking, filtering, fusionimprove order and remove distractorsextra latency; filter can remove useful evidence
Prompt constructionsummarisation, lexical or embedding compressionfit more useful evidence in a fixed budgetcompression can delete qualifiers or provenance
Generationread, cite, revise, Self-RAGuse evidence and decide whether to retrieve againmodel knowledge may override supplied evidence

Rewrite-Retrieve-Read

A rewriter converts an ambiguous or underspecified question into a search-friendly query, retrieval supplies evidence, and a reader answers. The lecture's trainable variant warms up on pseudo rewrites, keeps useful rewrites, then optimises the rewriter with a reward while constraining divergence from the initial policy.

它优化的是“送给 retriever 的 query”,不是直接修改 documents。

BlendFilter

BlendFilter combines retrieved knowledge with model-generated query/answer signals, then prompts an LLM to filter irrelevant passages before answer generation. It illustrates that retrieval is not the end of the pipeline: evidence selection can itself be an LLM task.

Prompt compression

Lexical compression removes low-information tokens using signals such as entropy or perplexity. Embedding compression maps a long context into learned compact memory tokens. Both trade fidelity for fewer input tokens, lower latency and less distraction.

Evidence conflict

A model may preserve an initially wrong answer even after receiving supporting correction, or flip a correct answer after contradictory evidence. Therefore “retrieved passage is relevant” and “generated answer follows evidence” are separate measurements.

这正是 Project 2 同时要求 reference-based answer score 与 evidence groundedness 的原因。

Noise, position and amount of evidence

ObservationInterpretationDesign response
One high-scoring distractor can sharply reduce answer qualitysemantic similarity is not the same as answer-supporting evidencererank/filter for utility, not topic overlap alone
More distractors progressively degrade performancelonger context creates more competing signalscontrol top-k and evidence token budget
Gold evidence in the middle may be used leastlong-context attention has positional biastest order; place or repeat critical evidence carefully
Random passages sometimes improve some modelsLLM behaviour depends on model and context distributiontreat it as an empirical finding, not a universal recipe

Attribution

Attribution is the ability to provide references or citations that support generated claims. Directly generated citations can be fabricated; retrieval-based attribution constrains citation candidates to real documents.

RARR

Retrofit Attribution using Research and Revision (RARR) researches evidence for an existing answer and revises unsupported content while preserving the original response as much as possible. It is a post-generation attribution-and-repair pipeline.

Why does better retrieval not guarantee better generation?
nDCG measures where judged-relevant passages appear. Generation additionally depends on passage truncation, order, prompt wording, answer format, conflicting model knowledge and whether the LLM actually attends to the evidence. The relationship should therefore be measured per query rather than assumed.
RAG, long context and fine-tuning: when are they different?
RAG changes external evidence at inference time; long-context prompting supplies much more text without explicit retrieval; fine-tuning changes model parameters. They can be combined, but they move cost, updateability and failure modes to different parts of the system.
Week 9 · Practical

Build and test evidence-grounded RAG / 构建并检查有证据支持的 RAG

Use Information Retrieval for Large Language Models (IR4LLM), reversing Week 8’s LLM4IR role. Retrieval-Augmented Generation (RAG) passes evidence to a generator; retrieval quality and answer quality remain separate measurements.Week 8 让 LLM 帮助排序,Week 9 让检索帮助 LLM 回答。LLM 接收的是 query 与证据文本,不是把 nDCG 分数当上下文。

Data flow: question + retrieved documents → labeled context → instruction/chat template → tokens → generated continuation → claim-by-claim checking

  1. Implement rag_answer

    Join the provided documents with labels, include the question, and ask for an answer supported only by that evidence. Tokenise the model-specific chat template, generate, then decode only output_ids after the input length. Keep model and input tensors on the same device.函数的关键是把证据实际放进 prompt。只写“请根据资料回答”但没有传入资料并不是 RAG。切掉输入 tokens,避免把 prompt 当生成答案。

  2. Understand the weather demonstration

    The notebook supplies two weather snippets as example evidence. It does not implement a live weather fetch. The example teaches how external context can change an answer; its quoted temperature is not today’s verified forecast.示例文本不等于实时数据。RAG 能使用更新资料,但资料是否真的更新、来自哪里,需要检索和来源检查来保证。

  3. Combine the café evidence

    The four fictional documents cover prices, caffeine, milk and a reusable-cup discount. A correct supported answer must satisfy every constraint and cite the relevant documents. Check arithmetic as well as the selected drink.答案需要综合四段信息:低咖啡因、无乳制品和预算缺一不可。只找到相关菜单仍不代表推理和计算正确。

  4. Control the prompt experiment

    Compare zero-shot instructions, few-shot demonstrations and a prompt requesting an explicit evidence check while holding the documents and generation settings fixed. Keep demonstrations separate from the actual café answer so examples do not leak the solution.Zero-shot 没有示范,few-shot 提供其他示范;要求说明计算可以帮助检查答案,但生成的解释不自动等于可靠的内部过程记录。

  5. Separate prompt from native thinking

    First compare prompts with native thinking off. Then hold the zero-shot prompt fixed and toggle thinking for MiniCPM5/Qwen3. Record correctness, cited support, output tokens and whether the answer finishes. Thinking tokens consume the same bounded output budget.不要同时换 prompt、模型和 thinking,再把差异归因给一个因素。被 token 上限截断的解释不算完成答案,单个小例子也不能证明某模型普遍更好。

Worked example / 拆开看一个例子

The supplied fictional evidence implies a decaf latte with oat milk: $4.80 + $0.70 − $0.50 = $5.00, and 5 mg caffeine ≤10 mg. D1 supports the base price, D2 the caffeine, D3 the milk substitution/surcharge, and D4 the discount. This is a hand-derived evidence check, not a reported model run.人工核验答案:燕麦奶低因拿铁,刚好 5 美元。四条文档分别支持不同事实;这不是声称模型已经跑出了正确结果。

Check your understanding: What happens when the milk-options document is removed?

The remaining evidence no longer establishes a dairy-free option or its surcharge. A grounded response should state the missing information instead of recalling the removed fact as if it were still supported.缺了 D3 就无法从现有证据确认无奶选项及加价。即使模型记得答案,也不能把记忆冒充这次检索到的依据。

Source map: prac-week9.ipynb: Exercise 1 rag_answer, Exercise 2 prompting strategies and thinking-mode comparison; downloaded 16 October 2026. Independent study explanation; original files remain on the course platform.

Back to Week 9 lecture concepts
Week 10

Search Agents / 搜索智能体

Week 9 studied how retrieved evidence enters a generator. Week 10 lets the system inspect an observation, decide whether evidence is missing, and choose another search or reading action before answering. The important change is a feedback loop, not merely adding more tools to a fixed RAG prompt.

From static RAG to an agent / 从一次检索到循环搜索

SystemDecision sequenceWhat happens when evidence is insufficient?
Static Retrieval-Augmented Generation (RAG)user query → retrieve once → put passages in prompt → generateUsually answers from the initial set, even when it misses a needed link.
Search agentgoal → action → observation → updated state → next action or finishCan reformulate a query, inspect a candidate, read more lines, or stop at a budget.

Agent means a goal-directed system interacting with an environment through an observe–think–act–update loop. Its policy chooses the next action from the current state; a tool executes that action; an observation is the returned result. ReAct (Reasoning and Acting) interleaves reasoning and actions rather than writing one complete answer before looking again.

例如先确认“大学创立于哪座城市”,再查“哪条河穿过该城市”。第一条证据只能支持城市,不能自动支持河流;第二次搜索要补上缺失的一跳。

State, evidence and stopping

The state can contain the original question, past queries, visited document IDs, tool observations and remaining budget. The policy should stop when the answer is supported by the evidence, when another search has low expected value, or when a step/token limit is reached. A plausible answer is not itself proof that the required sources were read.

Failure modes: retrieval misses evidence; a tool call is malformed; the agent loops; the budget expires; or the final claim is not supported by the observed lines. Diagnose these separately.

Search–Visit

The baseline pattern searches an index, then opens a promising result. It is useful when a short page or passage probably contains the answer. It becomes wasteful when a hit is a huge report: visiting the whole page consumes context before the agent knows where to look.

先找候选,再打开阅读;容易理解,但长文档会带来大量无关文本。

Which interaction strategy fits the evidence?

Method and full nameAction patternBest fitTrade-off
Search–Visitrank results → open pageone short page likely answers the questionwhole-page visits may waste tokens
SIEVE, Search–Inspect–Fetchsearch → inspect headings/structure → fetch relevant sectionone long report with clear sectionsdepends on useful structure; extra inspection step
DCI, Direct Corpus Interactiongrep raw files → read nearby linesmanageable local files and a known exact term/namelexical mismatch and broad scans across large corpora
RISE, Retrieving Interaction SpacEBM25 narrows files → grep/line navigation verifieslarge file collection, keyword known but file unknownretriever can omit the right file; still requires reading

Why RISE is not simply DCI: DCI can search raw files directly; RISE puts a ranked candidate-selection step in front of the file tools. Why SIEVE is not simply Search–Visit: the agent inspects document structure before fetching only the relevant part. These are interfaces and workflows, not interchangeable ranking models.

Retrieval still matters inside an agent

An agent can recover from a weak first search only if useful evidence remains reachable within its budget. BM25 ranks lexical matches; dense and learned sparse retrievers can address vocabulary mismatch; a reranker can reorder a fixed candidate pool. The lecture compares results across document granularities and agent settings, so no single model is universally best.

评价不能只看离线 nDCG:还要看是否找到支撑答案的证据、用了多少次搜索和多少 tokens,以及最终答案是否 grounded。

Agent-aware retrieval objective

Traditional retrieval scores a document mainly from the user's original query. During a trajectory, the next useful document also depends on what the agent has already learned. A previously visited relevant passage may still match the query but add almost no new information. Marginal evidence gain asks what this document adds now.

Training retrievers from agent interactions

ApproachQuery-side contextTraining signalKey caution
AgentIR, Reasoning-Aware Retrieval for Deep Research Agentsagent query plus its pre-search reasoningQA with gold documents; costly reranking gives positive and negative passages for intermediate queriesGold labels and an external reranker are costly; a snippet may omit the needed evidence.
LRAT, Learning to Retrieve from Agent Trajectoriesqueries produced during successful trajectoriesjudge visited passages for positives; no gold-document set requiredA visit alone is not proof of usefulness; treating all others as negatives can be noisy.
ITER, Interaction-Aware Retrieval for Agentic Searchcurrent query + original question + previous queriestrajectory-relative judgments distinguish new evidence from already visited relevant or irrelevant documentsSelection bias and noisy inferred labels still need care.

Connection to Week 5: all three still need informative positives and negatives for retriever training. The difference is where supervision comes from and whether the query representation includes interaction history. Connection to Week 7: successful observed trajectories are biased by what the old policy exposed; unvisited documents are not automatically irrelevant.

Week 10 practical: follow the actual data flow

1. Candidate discovery

grep(pattern) finds a first matching line per document; BM25 search(query) returns ranked candidate IDs.

→

2. Select and read

A CrossEncoder reranks the same BM25 pool; read(id, start_line, num_lines) opens numbered lines.

→

3. Observe and finish

The tool result enters the agent history; the model chooses another action or a cited answer within its budget.

The exercise asks you to complete read line slicing and search query/passage scoring. Its small dataset has relevance judgments for inspection, but the agent should not see those labels. A grep match is only a location hint; a BM25 or reranker score is only a ranking signal; neither is verified answer evidence until the passage is read.

Worked example: the colour-afterimage question requires an explanation of adaptation in the visual system. Compare the candidate that merely names afterimages with the passage that explains retinal/neural adaptation, then inspect which lines actually support the final claim. Optional Qwen3 thinking text is not a tool observation or an independently verified source.

Lecture exercise: how should the agent continue after finding the city?
Search for an independent source identifying the river through that city, read it, and cite the two evidence links separately. Knowing the city is only the first hop; answering the river before the second observation would be an unsupported jump.
Lecture exercise: short page, long report, exact term or unknown file?
Short page: Search–Visit. Long structured report: SIEVE. Known exact term in a manageable local corpus: DCI. Large collection where the useful file is unknown: RISE narrows candidates with ranking before grep/read. Choose by corpus size, lexical certainty and document structure, not by model fashion.
Lecture exercise: where do an agent-aware retriever's positives and negatives come from?
With judged documents, use gold evidence and hard negatives as in standard retrieval training. Without gold documents, successful agent trajectories can provide candidate positive visits; judge whether each visit actually contributed evidence, distinguish redundant visited passages from new useful ones, and do not label every unvisited document negative.

Research trail / 论文线索:ReAct · SIEVE · DCI · RISE · AgentIR · LRAT · ITER. These public papers support further reading; course PDFs and code remain on the restricted course platform.

Week 10 · Practical

From a retrieval call to a search–read loop / 从一次检索到搜索与阅读循环

Trace a ReAct (Reasoning and Acting)-inspired agent: the policy selects an action, Python validates and executes it, and an observation updates the state. Complete small tools rather than treating the agent as one opaque model.这是之前只简略整理过的 practical。本次补齐工具输入输出、边界测试、预算与失败分析,并连接 Week 9 的静态 RAG。

Data flow: question → policy → search/grep/read → tool observation → updated messages → another action or finish

  1. Separate the roles

    The policy is the model’s action choice; tools are Python functions; observations are their returned text/errors; state is the accumulated messages and allowed IDs; a trajectory is the sequence of actions and observations. Qrels stay outside the agent prompt.LLM 负责选择,Python 负责执行与约束。一次工具返回才是可检查的 observation;模型自述的 thinking 不是外部证据。

  2. Complete read

    Convert the public 1-based start_line to a 0-based slice start. Return up to num_lines, prefix each selected line with its original line number, and provide the next starting line only when unread lines remain. The tool accepts at most 12 lines.切片起点是 start_line−1。返回行号要从原文位置继续,不能每一页都从 1 编号;最后一页的 next_line 应为 None。

  3. Complete search

    Retrieve eight BM25 candidates; build (query, passage) pairs; use reranker.predict; sort the paired candidate/score objects descending; return the best k (allowed 1–5). Sorting scores separately from candidates breaks the ID-to-score association.8 是重排候选深度,k 是返回数量。重排不能找回 BM25 没有选中的文档。保留 docid、text、score 的对应关系。

  4. Observe scoped evidence access

    In Part 1, grep scans the ten-document sample. Part 2 explicitly requires search→grep→read: the latest search results constrain which documents grep/read may inspect. A grep match locates text; reading context establishes what it actually says.第二部分的访问范围是老师为了展示流程而设定,不代表所有 agent 必须用同一顺序。检索排名高和真正支持答案仍是两回事。

  5. Evaluate failures and budgets

    The classroom run caps decisions at 8, output at 512 tokens per decision and input at 8,192 tokens. Failed calls also spend steps. Inspect retrieval misses, malformed calls, repeated loops, budget stops and unsupported final claims separately.预算用完是停止原因,不是成功证据。finish 的引用 ID 已读,也不代表每一句答案都受支持;还要逐句对照引用行。

Worked example / 拆开看一个例子

For lines [alpha,beta,gamma,delta], read(start_line=2,num_lines=2) should return “2: beta” and “3: gamma”, with next_line=4. Reading from 4 returns “4: delta” and next_line=None. Starting beyond the end should produce an empty window with no next page, under the supplied validation contract.这三个边界例子能检查 off-by-one、末页和越过末尾的行为。它们验证工具,不需要先花时间运行大型模型。

Check your understanding: Does a successful run on the ten-document sample establish general agent accuracy?

No. It demonstrates the mechanism. Compare the answer’s supported claims and token/time costs, then use broader evaluation for general conclusions. Optional thinking-on/off runs must use equal budgets and fixed questions/tools.一次成功轨迹是机制示范,不是大规模 benchmark。保留失败轨迹,才能区分检索、工具和生成环节的问题。

Source map: prac-week10.ipynb: Exercises 1 read, 2 search, comparison and optional thinking; search_agent.py and agent_display.py. Independent study explanation; original files remain on the course platform.

Back to Week 10 lecture concepts
Week 11

Web Crawling, Link Analysis and Diverse Ranking / 网页抓取、链接分析与多样化排序

搜索引擎首先要发现网页,再把网页内容与链接图结合,最后避免 top results 只重复同一种解释。Week 11 把“收集什么、哪些页面更有权威、结果列表覆盖哪些意图”连成一个系统。

The crawl–graph–rank pipeline / 抓取—建图—排序

Discover

从 seed URLs 出发,从 frontier 取出允许访问且尚未处理的 URL。

→

Fetch and parse

请求页面、解析正文与链接、规范化 URL、识别重复内容。

→

Index and rank

建立内容索引与链接图,再综合 relevance、authority 与 diversity。

URL frontier 是已发现但等待抓取的 URL 集合。调度器要决定下一条 URL、host 访问间隔、何时重抓以保持 freshness,以及如何避开重复页面与 crawler traps。抓取范围受 seeds、链接可达性、访问规则、时间和存储限制,因此 crawl 不是完整复制整个 Web。

Politeness, robots.txt and access

合理 crawler 要限制同一 host 的请求速率、标识自身、遵守适用规则,并处理 redirect、失败与缓存。robots.txt 表达 crawler access preference,不是认证机制,也不能用于保护私有内容。

区分:“crawler 不应抓取”是抓取规则;“用户无权读取”必须依靠真正的 access control。

The Web as a directed graph / 有向图

网页是 node,hyperlink 是有方向的 edge。In-link 指向当前页面,out-link 从当前页面离开。Link analysis 可用于 ranking、link-based clustering、classification features 与 crawl prioritisation,也能发现与 spam 相关的异常链接分布。

Authority and anchor text / 权威与锚文本

课件区分两个假设:链接可能把 authority 赋予目标页;anchor text 可能描述目标页。索引页面 D 时,可把指向 D 的 anchor text 按权重加入表示,来源页 authority 也可影响权重。这能缓解 vocabulary mismatch,但会受到 spam 或协同锚文本操纵。

PageRank: recursive authority / 递归权威度

PR(v) = (1 − d) / N + d Σu→v PR(u) / L(u)

N 是页面数,L(u) 是页面 u 的 out-degree,d 是沿链接继续的概率;剩余概率用于 teleport。高 PageRank 页面传来的 link mass 更大,但要在它的所有 out-links 间分配。Dangling page 没有 out-link,需要明确重新分配,否则概率会丢失。

Power iteration 从一个概率向量开始,反复分配 link 与 teleport mass,直到变化足够小。PageRank 衡量的是图结构中的 importance,不是当前 query 的 relevance;实际排序可把它与 lexical、semantic、freshness 等信号结合。

Integrating query-independent evidence / 融合查询无关证据

Retrieval modelHow PageRank can enterCaution
Language model用 PageRank 等 feature 构造 document prior P(D),替代 uniform prior理论上方便,不保证 effectiveness 上升
BM25把文档看成 text + non-text features,用调参或学习的 utility 组合基础模型允许加 feature,但不会自动给出权重
Vector Space Model在外部 utility 中组合 cosine、PageRank、spam score、difficultyquery-independent feature 不自然存在于 query vector

PageRank、spamminess 与 URL depth 都可作为 document priors。它们是 feature,不是 relevance label;是否有帮助必须用目标指标实证评价。

Why diversify results? / 为什么需要多样性

jaguar 可能指动物、汽车品牌或球队,这叫 query ambiguity。即使用户意图相同,多篇近重复页面也会浪费前排位置,这叫 redundancy。Diversity 的目标是保留 relevance 的同时覆盖合理 aspects,不是让每条结果为了“不同”而不同。

MMR: relevance minus redundancy

argmaxd∈R∖S λ Rel(d,q) − (1−λ) maxs∈S Sim(d,s)

Maximal Marginal Relevance (MMR) 从剩余候选 R 中逐个选下一篇,S 是已选集合。较大 λ 更重 query relevance;较小 λ 更强地惩罚与已选结果相似。Document similarity 与 relevance score 的尺度必须可比较。

PRP assumptions and interdependent relevance

Probability Ranking Principle (PRP) 按估计相关概率降序排列。它的简单 next-document rule 假设概率估计准确/确定,并假设当前文档 relevance 可独立于已排序文档判断。Ambiguity 挑战前一个假设,redundancy 挑战后一个假设;diversification 改为评估候选在当前列表中的 marginal value。

List objectiveQuestionFailure if used alone
Relevance文档对 query 有多有用?可能重复一个 interpretation
Coverage覆盖哪些合理 intents/subtopics?可能牺牲过多 relevance
Novelty这条结果增加什么新信息?仅仅不同不一定有用

Evaluating and supplying diversity / 评价与意图来源

Diversity 不只是一条 reranking 公式。Query-ambiguity detection 判断是否存在多意图;query-aspect mining 产生候选意图;vertical diversification 可以混合不同结果类型。评价可使用 subtopic recall/precision/MRR、α-nDCG、intent-aware nDCG 或 intent-aware ERR。要明确 intent judgments 与 metric convention;普通单标签 nDCG 不能完整衡量 coverage + novelty。

Worked example: PageRank 会偏好哪一页?
若 A 和 C 都链接到 B,而只有 A 链接到 C,B 得到两份 endorsement。但精确分数还取决于 A、C 自身的 PageRank、它们的 out-degree、teleport 概率与 dangling-page 处理。因此只数 in-links 不等于 PageRank。
Worked example: relevance 还是 diversity?
假设三篇高相关结果都讲 jaguar 动物,另一篇分数略低但讲 Jaguar 汽车。纯 relevance 可能选择三篇动物页面;MMR 可能提升汽车页面,因为它补充未覆盖解释。只有当该 aspect 对 query 合理时,这种提升才有价值。
Week 11 如何连接前面的周次?
Week 1 的 crawl/index pipeline 提供网页与边;Weeks 3–8 负责内容打分与候选重排;Week 11 增加 graph authority、query-independent priors 与 list-level marginal utility。Link score、query relevance 与 diversity 是不同信号,组合前后都要分别评估。

Public reading / 公开延伸阅读:Stanford IR book: web crawling · Stanford IR book: PageRank · RFC 9309: Robots Exclusion Protocol · MMR paper。Week 11 已发布课件用于确认课程具体范围;受限课程文件不在本站托管。

Lecture and video route / 讲义与视频导读

The course playlist has 10 videos, totalling 28 min 36 sec. Links and chapter titles were checked on 13 October 2026; the explanations below were checked against the 40-page Week 11 lecture PDF. They are study notes, not video transcripts. 视频入口与标题已核对;以下讲解对照本地 40 页讲义整理,不冒充视频逐字稿。

Open the complete Week 11 playlist · Course announcement (login required)

ChapterDurationLearning checkpoint
11.1 Overview2:22Place graph authority and list diversity beside content relevance.
先区分内容相关性、链接权威度和列表多样性。
11.2 Links3:15Distinguish an endorsement from descriptive anchor text.
同一个链接可以提供权威信号和锚文本,二者不是同一数据。
11.3 Link analysis in IR3:18Explain why links help ranking, clustering, classification and crawling.
链接不仅用于排名,也用于聚类、分类和抓取调度。
11.4 PageRank4:41Trace one random-surfer step and the role of teleportation.
跟踪一次随机游走,理解为什么需要随机跳转。
11.5 Understanding PageRank1:56Use source authority and out-degree, not just the count of incoming links.
来源权威度和出链数量共同决定每条链接传来的权重。
11.6 PageRank in retrieval2:06Separate a document prior from the query-dependent matching score.
把文档先验与当前 query 的匹配分数分开理解。
11.7 Diversification1:17Explain why a relevant duplicate may add little value to a list.
单篇相关不等于加入列表以后仍有同样大的价值。
11.8 PRP3:22Expand Probability Ranking Principle and state the lecture's assumptions.
说出概率排序原则的全称,并解释课件中的两个假设。
11.9 Ambiguity and redundancy0:48Locate ambiguity in the query and redundancy in the result list.
歧义主要在 query,冗余主要在结果列表。
11.10 Diversity framework and MMR5:31Compute the next result's relevance-minus-redundancy score.
计算下一篇文档带来的相关性收益与重复惩罚。

PDF route: pp. 3–8 links and anchors; pp. 9–19 random surfer and PageRank; pp. 20–24 retrieval priors; pp. 25–36 diversity and PRP; pp. 37–40 diversification and MMR. The crawling/robots background above is a supporting extension, not the central scope of this PDF. 回看老师讲义时按这条路径定位。网页抓取和 robots 是补充背景,本周讲义主线是链接分析与多样化排序。

Three signals, three different questions / 三种信号回答不同问题

Method and full nameInputs → outputWhat changes the score?Not a substitute for
BM25 (Best Matching 25); DPR (Dense Passage Retrieval)query + document representation → matching scorequery words or meaningauthority or a diversity objective
monoBERT (BERT: Bidirectional Encoder Representations from Transformers)one query–document pair → relevance scorejoint contextual matchingcomparison with all already selected documents
PageRank (a model name, not an acronym)directed link graph + teleport rule → one authority value per pagelinks, graph boundaries and teleport distributionquery relevance or factual verification
PRP (Probability Ranking Principle)estimated relevance probabilities → descending orderthe relevance estimatesa trained encoder or a redundancy penalty
MMR (Maximal Marginal Relevance)candidate relevance + document similarity + selected set → next itemquery AND what is already selectedfirst-stage retrieval or ground-truth labels

Think of BM25/DPR as matching functions, PageRank as a graph feature, PRP as an ordering principle, and MMR as a set-aware selection rule. They can occupy different parts of the same pipeline; they are not five interchangeable neural models. 先判断它属于模型、特征、原则还是选取规则。PageRank 高不代表这条 query 相关;MMR 分数低也不代表文档不相关,可能只是已选结果已覆盖了它。

PageRank: follow every number / 逐步算一次概率传播

Original worked example: three pages with links A → B, C, B → C, and C → no out-links. Start at r₀ = (1/3, 1/3, 1/3). Let d = 0.85 mean follow-link probability and use uniform teleportation and uniform dangling-mass redistribution. 这是为复习新增的小例子,不是老师提供的实验结果。A 的权重要分成两份;C 没有出链,所以它的权重均匀分给所有页面。

Dt = Σu: L(u)=0 rt(u)
rt+1(v) = (1−d)/N + d [Σu→v rt(u)/L(u) + Dt/N]
TargetTeleportDangling C contributionIncoming link contributionAfter one iteration
A0.15/3 = 0.050.85 × (1/3)/3 = 0.094444…00.144444…
B0.050.094444…0.85 × (1/3)/2 = 0.141667…0.286111…
C0.050.094444…0.85 × [(1/3)/2 + 1/3] = 0.4250.569444…

Check: the new entries sum to 1. This is one iteration, not the final stationary distribution. Compute every new entry from the same old vector, then replace the entire vector. Repeat until a stated tolerance such as Σ |r_new − r_old| < ε is met. 不要算出 A 的新值后马上用它更新 B;一次迭代应统一使用旧向量。收敛后才能解释为长期访问概率。

What breaks if we ignore C? / 漏掉无出链页面会怎样?

Without the dangling term, the next vector sums to 0.15 + 0.85 × 2/3 = 0.716667…. The missing mass is 0.85/3. A dangling page has no outgoing edge; a closed group may have outgoing edges internally but no exit. Teleportation also gives a way to escape such closed groups. 漏掉 dangling mass 会丢概率;不能把“权重变少了”当成正常收敛。无出链节点与封闭链接团体相关但不是同一种结构。

Notation trap: lecture p. 15 uses d for link-following, while p. 17 writes 1−λ for the same role. MMR later uses λ for relevance weight. Read the definition, not just the letter. 同一个 λ 在不同公式中不是同一个参数。PageRank 的跳转概率和 MMR 的相关性权重没有直接等同关系。

From a graph score to a retrieval prior / 图分数怎样进入检索

P(D | q) ∝ P(q | D) P(D)
log score(q,D) = log P(q | D) + log P(D)

Here the language model is the query-likelihood retrieval model, not necessarily a generative LLM. P(q|D) models the query from the document; P(D) favours documents before inspecting this query. A positive, normalised PageRank distribution can supply the prior. 这里的 Language Model 是传统查询似然检索模型,不要自动理解成 ChatGPT。先验表示在看 query 前对文档的偏好。

Toy comparison: A has P(q|A)=0.02, P(A)=0.60; B has P(q|B)=0.08, P(B)=0.10. Query likelihood alone favours B; products 0.012 > 0.008 favour A. These products are unnormalised ranking values, not calibrated answer-correctness probabilities. The changed ranking may help or hurt. 例子说明权威先验可以改变排序,不是证明改后一定更好。基础 BM25 也不会因为算出了 PageRank 就自动使用它;需要明确组合方式、量纲与验证方案。

MMR: work through the lecture example / 把讲义例子算完

Lecture p. 39 supplies relevance values A=0.8, B=0.6, C=0.5 and similarities sim(A,B)=0.9, sim(A,C)=0.1. A and B cover the same interpretation; C covers another plausible one. Start by selecting highest-relevance A. We add the explicit choice λ=0.5 to make the arithmetic reproducible. 老师图中给出这组分数;这里补上 λ=0.5 的计算假设。已经选择 A,接下来只比较 B 与 C 的新增价值。

CandidateRelevance rewardRedundancy penalty against AMMR at λ=0.5
B0.5 × 0.6 = 0.300.5 × 0.9 = 0.45−0.15
C0.5 × 0.5 = 0.250.5 × 0.1 = 0.050.20

Relevance-only order: A, B, C. With this MMR setting: A, C, B. A negative MMR value is allowed because it is a selection utility, not a probability. No new document has been retrieved: the candidate set is unchanged. MMR 只是重排已有候选,不能找回第一阶段没有召回的文档。负数表示此刻重复惩罚超过相关性收益,不表示“负相关概率”。

A already selected / 已选 A · B = −0.150; C = 0.200; A → C → B

Why does the order switch near λ=0.89? / 为什么约 0.89 时顺序会换?

B wins when 0.6λ−0.9(1−λ) > 0.5λ−0.1(1−λ), giving λ > 8/9. Equality is a tie, resolved here by original candidate order. At λ=1, only relevance matters. At λ=0, only redundancy matters after the initial seed; choosing A first is an explicit convention for the otherwise empty selected set. 比较两个表达式即可找到阈值。纯多样性可能提升不够相关的结果,所以一般先获取相关候选,再选取合适的权重。

Why maximum similarity, and why recompute? / 为什么取 max,还要重复计算?

Once S contains several documents, each remaining candidate is penalised by its most similar selected document, not its average similarity and not the sum. Recompute after every selection because S has changed. The relevance term and the document–document term may use different models; their score scales must be compatible. MMR is greedy and does not promise the global optimum for every list-level objective. 只要有一篇已选结果和候选高度重复,就可能带来大的惩罚。不能先对每篇算一次固定 MMR 分数,再排序到底。

Diversity, nDCG and evidence quality / 多样性、评价与证据质量

Metric references / 指标来源:Expected Reciprocal Rank (2009) · Intent-based diversification: metrics and algorithms (2011).

A single relevance label can rate two duplicate documents equally highly. Therefore two lists can have identical ordinary nDCG yet cover different numbers of intents. α-nDCG (alpha-normalised Discounted Cumulative Gain) discounts repeated coverage of the same judged subtopic. ERR (Expected Reciprocal Rank) models stopping at a satisfying result; plain ERR is not automatically an intent-diversity measure. Intent-aware variants such as ERR-IA aggregate per-intent utility. 相关性判断不包含意图标注时,不能仅靠普通 nDCG 宣称多样性提升。ERR 与 intent-aware ERR 要区分。课件 p.40 列出评价方向,这里补充了术语边界。

Connection to Weeks 9–10, not an extra assignment requirement: under a RAG context budget, three near-duplicate passages may leave another necessary fact uncovered. Diversity-aware evidence selection can be tested while fixing the query set, generator, prompt and budget. However, novelty does not guarantee support or truth; complementary evidence, necessary corroboration and conflicting claims must still be checked. 这是帮助理解的架构联系,不是要求给 Project 2 额外添加 MMR。实验应分别衡量检索相关性、意图覆盖、答案正确性和 groundedness,不能把这些分数混为一谈。

Practice the numerical examples in Revision Q/A / 到题库检查自己是否真的会算

Week 11 · Practical

PageRank as a probability update / 把 PageRank 理解为概率更新

Implement pagerank(pages, lam, eps) for the supplied five-page graph. Relate every matrix operation to the lecture’s random surfer. MMR (Maximal Marginal Relevance) remains a lecture topic, not an extra task in this notebook.这份 practical 只有简化 PageRank 实现。不要把 lecture 的 MMR 自动当成老师要求额外完成的代码任务。

Data flow: column-stochastic link matrix M + teleport λ → initial r → repeated probability updates → convergence → page scores

  1. Read matrix orientation

    M[i,j] is the probability of moving FROM page j TO page i. Each column distributes one source page’s outgoing mass and sums to 1. The supplied matrix is already weighted; do not replace its 0.5 entries with 1.列是出发页面、行是到达页面。先画出一列的箭头,才能知道应该使用 M @ r,而不是 r @ M。

  2. Translate the symbols

    Here lam=0.15 is teleport probability; 1−lam=0.85 follows links. For N=5, each target receives lam/N=0.03 teleport mass. This is the same model as the lecture formula using d=0.85 for link following.lam 和 d 取值不同,是因为定义相反。代码 docstring 里的 damping 一词不够明确,要以实际公式中乘在哪一项为准。

  3. Update all pages together

    Start with r=ones(N)/N and compute r_new=lam/N+(1−lam)*(M @ r). Use the entire old vector for one update; do not overwrite one page and immediately use its new value for the next page.M @ r 把每个来源页面的概率按出链分配。同步更新对应一次随机浏览者转移;逐项覆盖会变成另一套迭代过程。

  4. Define convergence and protection

    One explicit rule is sum(abs(r_new−r))<eps. Assign r_new before returning and set a maximum iteration count as a safety guard. A different norm can produce a different iteration count, so document the rule.eps 是两次迭代差异的容忍度,不是 PageRank 分数门槛,也不是 p-value。达到最大轮数却未收敛,应报告未收敛。

  5. Verify a probability vector

    Check finite nonnegative entries, total probability near 1 and a small fixed-point residual. The classroom matrix has no dangling column. For a new graph, zero-outdegree columns need redistribution, such as uniform 1/N, before applying this stochastic-matrix formulation.原例已经每列和为 1;扩展到任意网页图时,不能忽略全零列。随机跳转项本身不自动补回丢失的 dangling mass。

Worked example / 拆开看一个例子

Starting from [0.2,0.2,0.2,0.2,0.2], the teacher’s matrix yields [0.200,0.115,0.115,0.285,0.285] after one step. With an L1 stopping threshold of 1e−6 it reaches approximately [0.254192,0.138032,0.138032,0.205990,0.263755] after 57 updates. The two equal middle scores follow from symmetric incoming links. These numbers were checked locally using the supplied matrix.结果顺序对应矩阵中的页面顺序,不能排序后再与原索引配对。第 5 页最高,第 1 页其次;概率和约为 1。这里报告的是该小矩阵的数值验证,不是网页质量评测。

Check your understanding: Why does the function have no query argument?

PageRank is query-independent graph authority. BM25/DPR estimate query–document matching; MMR also considers already selected results. To test PageRank in retrieval, combine it with a query-dependent score under an explicit rule and evaluate the ranking with qrels.高 PageRank 不等于当前问题最相关,更不等于答案真实。它可以作为排序特征,但不是替代 BM25、nDCG 或人工 gold。

Source map: prac-week11.ipynb: Implement simple pagerank; official notebook modified 14 October, downloaded 16 October 2026. Independent study explanation; original files remain on the course platform.

Back to Week 11 lecture concepts
Week 12

Relevance Feedback, Query Reformulation and Rank Fusion / 相关反馈、查询重构与排序融合

Week 12 研究三件事:如何利用 relevance evidence 改进查询,如何扩展或精简 query,以及如何把多个不同比例尺的 ranked lists 合成一个结果。

Three kinds of relevance feedback / 三类相关反馈

FeedbackEvidence / 证据Main risk / 风险
Explicit用户明确标出 relevant 或 non-relevant documents质量高,但需要用户额外操作
Implicit把 clicks 等行为解释为 relevance signalexamination 与 presentation bias
Pseudo-relevance feedback (PRF)假设初始排名顶部文档相关首轮排名弱时会 query drift

三者差别是 evidence certainty,不只是界面不同。“More like this” 是 explicit query-by-example;普通点击不是天然 relevance label;PRF 通常不询问用户,只取正反馈。

Rocchio: move the query in vector space / 在向量空间移动查询

q′ = αq₀ + (β / |Dᵣ|) Σd∈Dᵣd − (γ / |Dₙᵣ|) Σd∈Dₙᵣd

Original query 负责锚定方向;relevant centroid 把 query 拉向有用词项;non-relevant centroid 把它推离。α、β、γ 表示对三类证据的信任。某些分量可能变负,实际系统必须说明保留、截断还是另行处理。

Query expansion families / 查询扩展来源

可以从 thesaurus/controlled vocabulary、collection 或 query-log statistics、top-ranked documents 加词。MeSH 这类领域词表可能帮助 specialist search;通用同义词扩展可能引入错误词义。Query suggestion 提供 alternative query,不一定只是把当前 query 加长。

Association needs context / 关联需要上下文

Dice coefficient 与 mutual information 用共现估计词项关系;mutual information 容易偏好 rare events。逐个扩展 tropical 和 fish 可能丢失短语含义;whole-query 或 n-gram context 更稳,但不可能为所有潜在 query 预先穷举。

Translation language models / 翻译语言模型

pt(w|d) = Σu∈d pt(w|u) p(u|d)

TLM 允许 document term u 通过 translation probability 为 query term w 提供概率质量,再把 translated document model 放进 smoothed query-likelihood model。Translation probabilities 可由 mutual information 或 embedding cosine similarity 估计,但 self-translation 必须足够强,不能为了语义扩展丢掉 exact matching。

Embeddings as semantic expansion / 向量语义扩展

Distributional representations 让上下文相似的词在向量空间接近。把 cosine similarity 归一化可得到 translation distribution,也能覆盖 syntactic variants。Similarity 不等于 identity;不受控的近邻会加入“主题相关但任务不相关”的词。

Query reduction / 查询精简

Verbose query 可能包含偏题词。IDF-r 保留 IDF 最高的一定比例词项;learned reduction 可用 query-performance predictors 及其他 features 预测保留比例。Expansion 和 reduction 表面方向相反,目标相同:让 query 更准确表达 information need。

Why fuse rankings? / 为什么融合排名

FamilyUses / 使用信息Examples
Voting哪些系统提名该文档plurality, Copeland
Rank aggregation文档在每个列表的位置Borda, Reciprocal Rank Fusion
Score aggregation通常先归一化,再组合 scoresCombSUM, weighted linear fusion, CombMNZ

Chorus effect 奖励多系统一致;skimming effect 从不同系统取得不同文档;dark-horse system 可能贡献独特好结果,也可能带来独特噪声。Fusion 常能超过最佳单一系统,但加入任意系统都提升并无保证。

RRF and score fusion / 排名与分数融合

RRF(d) = ΣLᵢ:d∈Lᵢ 1 / (v + rLᵢ(d))

RRF 只使用 rank,常见 v=60,越靠后的结果贡献越小。CombSUM 直接加分;linear fusion 为不同系统赋权;CombMNZ 再乘以支持该文档的列表数。某列表未返回该文档时,该列表不贡献。

Normalise before combining scores / 分数融合前先归一化

BM25、language model 和 neural scores 的来源与范围不同。可按每个 list 使用 min–max、shifted-sum 或 z-score。理想方案应 shift invariant、scale invariant,并尽量不让一个 outlier 改变所有其他文档。Min–max 可直接处理负分,但对极端 min/max 敏感。

Worked example:一次 Rocchio 更新
令 q₀=(1,0),relevant centroid 为 (0.8,0.6),non-relevant centroid 为 (0.2,0.9),且 α=1, β=0.75, γ=0.25,则 q′=(1.55,0.225)。这是下一轮检索的 query direction,不是 relevance judgement。
Worked example:RRF 怎样奖励一致性
取 v=60。A 在两列表位于第 1、4 名,得 1/61+1/64≈0.03202;B 位于第 2、1 名,得 1/62+1/61≈0.03252,因此 B 更高。计算前要说明 rank 是否从 1 开始。
Worked example:负分怎样做 min–max
一个列表的 scores 为 [-8,-5,-2],min–max 后是 [0,0.5,1]。给所有原始分数加 100 后结果不变,说明 shift invariance;若出现新的极端最大值,其他 normalized gaps 会一起变化。

Source status / 来源状态:已于 2026-10-20 核对正式发布的 49 页 Week 12 lecture,以及正式 practical notebook 与六个配套文件。官方 Week 12 Live Q/A deck 当时尚未开放;本站没有把任何 Week 12 题目冒充官方题,相关题目均明确标为 self-authored practice。原课程文件仅保留私有本地。

Public reading / 公开延伸:Stanford IR book: relevance feedback and query expansion · National Library of Medicine: MeSH.

Week 12 · Practical

Reformulate queries and fuse rankings / 查询重构与排序融合实践

Implement pseudo-relevance feedback (PRF) expansion, IDF-r reduction, and three rank-fusion methods. Evaluate every change with the same MS MARCO queries and qrels instead of assuming that a longer query or a fused list must be better.实现 PRF 扩展、IDF-r 精简以及三种排序融合方法。所有变化都要在同一组 MS MARCO queries 与 qrels 上评价,不能默认 query 更长或融合后就一定更好。

Data flow: 43 queries + index → baseline run → expand or reduce query → MAP/nDCG@10/recall@1000; four runs → per-query normalisation → Borda/CombSUM/CombMNZ → paired comparison

  1. Establish one evaluation baseline

    Load the 43 supplied queries, retrieve up to 1,000 documents with the same Lucene analyzer, and write qid, Q0, docid, rank, score and run tag. The supplied qrels contain many judged pairs, including relevance label 0; a qrel row is a judgment, not automatically a relevant document.先固定 analyzer、retrieval depth 与评价指标。qrels 中出现 docid 只表示它被判断过,label=0 仍是不相关;不能用“qrels 行数”当作相关文档数。

  2. Expand through pseudo-relevance feedback

    Retrieve an initial top-50 list, treat only the top n as pseudo-relevant, obtain each document vector, and aggregate candidate term evidence such as TF-IDF. Select the top m expansion terms, append them to the analyzed original query, rerun retrieval, then compare MAP, nDCG@10 and recall@1000. Exclude accidental noise and explain any query drift.n 是反馈文档数,m 是新增词数,不能混用。首轮 top documents 若属于错误词义,高权重词会把 query 继续拉偏;因此要同时检查总体指标与具体失败 query。

  3. Reduce a verbose query with IDF-r

    Analyze the query using the same Lucene analyzer, obtain document frequency for each resulting term, compute the practical’s IDF convention, and retain the n highest-IDF terms. Decide explicitly how duplicates, ties and n larger than the term count behave. A ratio-based variant must convert the ratio to a documented integer count.先分词再按 IDF 排序,保留区分度高的词。不要把 collection frequency 当 document frequency;删词可能提高效率与聚焦,也可能删掉 intent 所需的上下文。

  4. Normalise scores before score fusion

    The four supplied runs use TF, IDF, TF-IDF and BM25 scales. For each query and each run separately, min–max normalise scores before CombSUM or CombMNZ. Handle a constant-score list explicitly because its maximum equals its minimum. Borda uses rank positions, so raw score scale is irrelevant to its vote.归一化单位是“一个 query 的一条 run”,不是把所有 query 混在一起。若 max=min,直接除会得到零除;需要先声明处理规则。Borda 只看 rank,不看原始分数大小。

  5. Implement and compare three fusion rules

    With a one-based rank r in a list of length n, Borda contributes (n−r+1)/n; if enumerate starts at zero i, use (n−i)/n. CombSUM adds normalised scores over the union of documents. CombMNZ multiplies that sum by the number of runs in which the document has a positive score. Try combinations of the four runs, report MAP and nDCG, and use paired per-query tests for the chosen comparison.Borda 最容易出现 0/1-based off-by-one。CombSUM 奖励总证据,CombMNZ 还奖励多条 run 的共同支持。搜索“最佳组合”会增加多重比较风险,不能只报最高平均数而不说明选择过程。

Worked example / 拆开看一个例子

For two three-document rankings A=[d1,d2,d3] and B=[d2,d3,d4], Borda awards [1,2/3,1/3] by position. Totals are d2=5/3, d1=1, d3=1 and d4=1/3. If normalised score runs instead give d1 scores .9 and .2, d2=.4 once and d3=.8 once, CombSUM gives 1.1, .4 and .8; CombMNZ gives 2.2, .4 and .8 because only d1 has positive support from two runs.这个例子把 rank vote 与 score fusion 分开。Borda 不需要分数归一化;CombSUM/CombMNZ 则必须先让不同模型的分数可比较。若 min–max 后恰为 0,按 practical 定义它不增加 CombMNZ 的正分出现次数。

Check your understanding: Why can fusion perform worse than its strongest input run?

Fusion can add correlated errors, reward documents repeated by several weak systems, or distort evidence through unsuitable normalisation. Compare against every component run on the same qids, inspect per-query differences, and distinguish a mean gain from statistically reliable improvement.融合不是自动集成增益。多个弱模型可能重复同一错误;归一化也可能压缩有用差距。必须与每条 component run 在相同 qids 上比较,并检查哪些 query 变差。

Source map: prac-week12.ipynb: Exercises 1–3 and the supplied MS MARCO qrels, queries and four run files; released and reviewed 20 October 2026. Independent study explanation; original files remain on the course platform.

Back to Week 12 lecture concepts
Week 13

Efficient Inverted Indexes and Query Processing / 高效倒排索引与查询处理

Week 13 回到 inverted index,讨论一个工程问题:在超大规模语料上,如何又快又好地返回 top-k 结果。内容包括 compressed postings、quantized impact scores、dynamic pruning,以及 learned sparse retrieval 带来的效率代价,最后介绍期末口试的形式。

Why inverted indexes still matter / 倒排索引为何仍重要

有些搜索确实需要精确词项匹配;BM25 在没见过的数据上仍是强 baseline;倒排索引随语料增长扩展性好。SPLADE、TILDEv2 等 learned sparse 模型也把输出存进倒排索引,所以索引效率直接决定它们是否实用。在 web 规模下,快速返回足够好的结果,通常比慢慢返回最好的结果更有价值。

Postings, gaps and block access / 倒排项、差值与分块访问

一个 posting 由 document identifier 加 term frequency 或分数组成,同一列表中的 identifier 严格递增。只存相邻 identifier 的差值(delta coding)会得到更小、更易压缩的整数。代价是要取第 n 个 identifier 时,必须把前面的差值累加(prefix sum)。Block-based index 直接存每个块的首个 identifier,查找只需在一个块内累加,整块也可以不解压就跳过。

Compression is a trade-off / 压缩是一种取舍

Variable Byte(VByte)编码每个字节存 7 个数据位,剩下一位标记后面是否还有字节,所以小整数占用更少字节。实际系统大多用 block-based codecs。编码越小,编码和解码通常越慢;该选哪种 codec,取决于更看重空间还是查询延迟,也就是在 space–time Pareto frontier 上选位置。

Matching is separate from ranking / 匹配与排序是两回事

Boolean conjunction(AND)只返回包含所有查询词的文档,做法是推进最短的列表,并在其他列表中向前 seek。Disjunction(OR)接受包含任意查询词的文档。匹配规则并不决定排序:ranked conjunction 只给 AND 匹配到的文档打分,而 top-k disjunctive retrieval 相信 ranking function,对任何匹配文档都可以打分。

Precomputed and quantized impacts / 预计算与量化的 impact

Id,t = round( 2b · (sd,t − smin) / (smax − smin) )

对 BM25 这类可加模型,与查询有关的只有“用到哪些词”,所以每个词—文档贡献 sd,t 可在建索引时算好。浮点分数很难压缩,因此用全局 min–max 把它映射成 b 位整数,替代 term frequency 存进 postings;文档得分变成 impacts 之和。注意 2b 本身需要 b+1 位,所以实现通常乘以 2b − 1,或把最大值截断。

Top-k retrieval and dynamic pruning / Top-k 检索与动态剪枝

穷举式 top-k 检索给每个 posting 打分,同时用 min-heap 保留最好的 k 个文档;结果 rank-safe,但要解压并计算全部内容。Dynamic pruning 在保持同样 rank-safe 结果的前提下少做工作:建索引时存下每个列表的最大 impact Ut,用 heap 里的最小分数作为阈值 θ,把候选文档所在列表的上界相加;如果上界不可能超过 θ,就跳过该文档不打分,跳过的块也不用解压。前提是 scoring function 可加、预先存好列表上界,并且列表支持高效随机访问。判断是否跳过本身也有开销。

WAND and MaxScore / 两种剪枝算法

WAND(“weak” 或 “weighted” AND)按各列表当前文档排序,找到第一个累积上界超过 θ 的 pivot;pivot 之前的文档不可能进入 top-k。MaxScore 按上界给词排序,把合起来也达不到 θ 的低上界词标为 non-essential:候选只从 essential lists 产生,non-essential lists 只用来为这些候选查分。

DAAT, TAAT and SAAT / 三种遍历策略

StrategyTraversal order / 遍历顺序State / 状态
Document-at-a-time (DAAT)所有列表一起,一次处理一个文档当前 top-k 的 heap
Term-at-a-time (TAAT)一次处理一整个列表每个文档一个 accumulator
Score-at-a-time (SAAT)postings 按 impact 分组,从高到低处理accumulators;可以提前停止

Why learned sparse retrieval is harder to make fast / 为什么 learned sparse 更难加速

倒排索引高效靠的是稀疏性:一篇文档只涉及少数词,一个词只出现在少数文档里,一个查询只有少数词。Zipf’s law 让大多数词都很少见。Learned sparse 模型削弱了这三个假设:expansion 给文档和查询加了很多词;有些 token 被“挪作他用”,几乎出现在所有文档里,部分像 dense vector;长列表还可能得到很高的 learned impact,而 BM25 给长列表的 impact 很低。于是上界一直很高,剪枝去掉的候选变少,打分的 postings 变多,查询变慢。鼓励稀疏的 regularisation 是一种应对方法。

Final oral exam: format and preparation / 期末口试:形式与准备

Stage / 阶段What happens / 会发生什么How to prepare / 怎样准备
前 5 分钟拿到两道提示问题,同时核对身份证件为每题准备简短讲解:定义、机制、例子、局限
主要对话先回答其中一题,考官再追问细节、问题和改进方法练习解释方法为什么有效、会在哪里出问题,而不只是背定义
切换主题考官可能转到课程其他主题,以确认水平或给你冲更高档的机会把各周串起来:representation → retrieval → reranking → evaluation → RAG 与 agents

口试 30 分钟,闭卷,需要带身份证件;不能使用任何笔记或设备,结束时不会当场给分,因为成绩要在教学团队内统一审核。请通过课程官方渠道预约时间。把它当作一次技术面试:边想边说,说明假设,必要时及时纠正自己。

Worked example:差值与分块查找
Postings [3, 7, 8, 15, 30, 31, 45, 52] 变成差值 [3, 4, 1, 7, 15, 1, 14, 7]。不分块时,取第 7 个 identifier 要在第一个之后累加六个差值。每块 4 个、直接存首 identifier 3 和 30 时,第 7 个是 30 + 1 + 14 = 45,只需两次加法。
Worked example:Variable Byte 编码
从最高位开始把整数切成 7 位一组,除最后一个字节外,其余字节的 continuation bit 都设为 1。130 = 1·128 + 2 编码为 10000001 00000010;300 = 2·128 + 44 编码为 10000010 00101100;5 只需一个字节 00000101。普通 32 位整数总是占四个字节。
Worked example:量化一个 impact
取 smin=0、smax=20、b=8,BM25 贡献 7.5 映射为 256 × 7.5 / 20 = 96。最大分数会映射到 256,需要 9 位,所以实践中会乘以 255 或截断。
Worked example:什么时候可以跳过文档?
设列表上界 Ua=4、Ub=7、Uc=3,当前 heap 阈值 θ=9。只出现在 a、c 中的文档上界为 7 < 9,不打分直接跳过;出现在 a、b 中的文档上界为 11 ≥ 9,必须精确打分。对 MaxScore,把上界排序为 3(c)、4(a)、7(b):c 和 a 合计只有 7 < 9,是 non-essential,只有列表 b 产生候选。

Source status / 来源状态:已于 2026-10-27 核对 206 页的 Week 13 客座讲座(高效倒排索引)和 4 页期末考试说明。Week 13 没有 Live Q/A deck;本站所有 Week 13 题目均明确标为 self-authored practice。原课程文件仅保留私有本地。

Public reading / 公开延伸:Stanford IR book: index compression · Stanford IR book: efficient scoring and ranking.

Course clarifications · checked 13 October 2026

Project 2:主任务与 bonus 数据要分开

These are dated paraphrases of staff replies, not new assessment rules. Follow the latest course notebook and marking sheet if later updates differ. 这里是带日期的教师答复摘要,不是新增作业要求。以后若有更新,以最新版 notebook 和 marking sheet 为准。

ScopeVerified clarificationPractical consequence
Main Project 2 / Task 2The earlier “development set” wording refers to the sampled collection, queries and qrels; staff corrected the notebook wording.Use the simplified main-task sampled-data protocol. Do not reinterpret the bonus training queries as the main task's development set.
Dense-retriever fine-tuning bonusTrain queries/qrels are reserved for the bonus. A separate staff reply says dev queries/qrels have now been attached under the bonus folder.Obtain the official bonus files; train on train data and use dev data for checkpoint/hyperparameter selection according to the bonus instructions.

Important distinction: the main-task terminology correction and the later bonus dev-file addition concern different scopes. They are not contradictory. The bonus question identifies dev_queries.tsv and dev_qrels.txt. Staff confirmed availability; this website update has not checked the contents or counts of those new files. 主任务把 sampled data 称为 development set 的笔误,和 bonus 新增真正的 dev 文件,是两件事。不要把它们合并成“所有任务都用同一个 dev”。本次只核对教师回复,未核验新数据文件内容或条数。

For a scientific report, state exactly which data selected a model and which data evaluated it. Reusing a sample for selection and comparison is a course simplification, not evidence of an independent held-out generalisation estimate. 即使老师允许简化,也应如实记录选择模型与评价所用的数据;不要把重复使用的 sampled set 写成独立测试集。

Staff clarification: main-task sampled data (login required) · Staff clarification: bonus dev files (login required)

TILDEv2 modeling.py:目前仍待教师确认

A newer course question asks which modeling.py source to use. No staff answer was visible when checked. This page therefore does not approve a replacement implementation. Compare the class API and checkpoint compatibility before treating files from different versions as interchangeable. 目前看到的是同学的问题,不是老师已批准的方案。不能把“能否用 GitHub 版本”的提问当作已确认答案。

Open the pending course question (login required)

Formula sheet

公式速查表

先认清符号,再问每个因子在奖励或惩罚什么。

名称公式最重要的解释
P@krelevant in top k / k只看截断位置 k 内的精确率
Recallretrieved relevant / all relevant需要知道总相关文档数
F12PR/(P+R)precision 与 recall 的调和平均
AP(1/|Rel|) Σ P@k × rel(k)每遇到一个 relevant 就记录当时 precision
MAPmean(AP over queries)所有 query 同权
RR1/rank(first relevant)只在乎第一个 relevant
nDCG@KDCG@K / IDCG@K支持 graded relevance 与位置折扣
RBP(1-p)Σ r_i p^(i-1)p 显式建模用户继续浏览概率
Model matrix

模型对照:表示、打分、损失与负样本

遇到选择题时,把模型拆成这四列判断,比背一句描述可靠。

模型Representation / poolingSimilarity / interactionTraining定位
BM25稀疏 term statistics逐 term 加权求和通常无需训练;调 k1,b强 lexical baseline
TILDEv2query token IDs;document contextual token weightsexact match + per-term max weightdocument-side neural training/expansionfast learned sparse reranker
SPLADE(v2)mostly-zero vocabulary vectorssparse dot productMLM head + distillation + sparsity regularizationlearned sparse first-stage retrieval
PromptRepsLLM dense hidden state + sparse logitsnormalized hybrid interpolationzero-shot; no retriever trainingLLM-based hybrid representation
DPRCLS,query/doc 双编码dot productInfoNCE/NLL;in-batch + hard negativesdense first-stage retrieval
RepBERTmean token embeddingsinner productMultiLabelMarginLossBERT bi-encoder
ANCECLS / first tokendot productNLL/InfoNCE-style;ANN self-mined negatives动态 hard-negative mining
Contrievermean poolingcosineunsupervised contrastive;same-doc spans无监督 dense retriever
ColBERT保留所有 token embeddingslate interaction MaxSimcontrastive / pairwise CE效率与细粒度匹配折中
E5-MistralMistral decoder;EOS/last tokentemperature-scaled cosineInfoNCE;synthetic + supervised dataLLM embedding model
LLM2Vecdecoder 改双向 attention;poolingcosine 常见MNTP + unsupervised SimCSE + optional supervised把 decoder 转为通用 encoder
RepLLaMALlama2;EOS last hiddendot productInfoNCE;supervised + LoRALLM retriever
Cross-encoderquery 与 doc 联合编码分类/回归头输出 relevancepointwise/pairwise/listwise高精度 reranker
monoBERT[CLS] q [SEP] d [SEP] joint encodingrelevant-class softmax scorebinary cross-entropy;human positives + BM25 negativespointwise BERT reranker
BERT-MaxP / FirstP / SumPdocument passagesmax / first / sum of passage scorespassage labels transferred from documentlong-document score aggregation
PARADEpassage representationsaverage / max / attention / Transformer aggregationdocument-level rerankinglong-document representation aggregation
duoBERTquery + two candidate documentspairwise probability then win aggregationpairwise cross-entropylate-stage BERT reranker
LLM pointwisequery + one passage promptlabel/logit or relevance gradezero-shot prompting in Week 8simple LLM reranker
LLM pairwisequery + two passagesdirect preferencezero-shot promptingrobust but comparison-heavy
LLM listwisequery + candidate listgenerated permutationzero-shot promptinglist context, generation cost
LLM setwisequery + small candidate setbest-item logitszero-shot promptingfewer calls than pairwise
Corpus-in-Context (CiC)entire corpus + demonstrations in an LC-LLM promptgenerate relevant document IDsfew-shot prompting; no retriever trainingextreme model-based retrieval; corpus must fit context
Screenshot retrieverpage pixels/layout encoded by VLMdense similarity / learned retrieval scorevision-language contrastive traininglayout-rich and multimodal documents
DiffRetrievermultiple parallel representative tokensmulti-representation similaritydiffusion language-model representation learningefficient multi-vector generative retriever
Basic RAGquery + top-k external passagesautoregressive answer likelihoodretriever and generator may be frozen or adaptedIR4LLM; evidence-grounded answering
Rewrite-Retrieve-Readrewritten search query + retrieved passagesrewrite reward + downstream answer utilitypseudo-data warm-up and optional policy optimisationrepair ambiguous/underspecified queries
BlendFilterretrieved evidence + model knowledgeLLM filtering then answer generationprompted augmentation/filteringremove distracting retrieved knowledge
RARRdraft answer + researched evidencesupport check and revisionpost-hoc research/revisionretrofit attribution to generated text
PageRankdirected graph; one value per pagestationary random-surfer probabilitypower iteration, no relevance-label training requiredquery-independent authority feature
PRPestimated document relevance probabilitiesdescending probability ordera ranking principle, not a trained modelrelevance-oriented ordering
MMRcandidate scores + document similarities + selected setrelevance reward minus maximum redundancygreedy selection; tune the trade-off separatelydiversity-aware candidate reranking
Fast classification rule / 快速分类:先问输出是 lexical sparse、learned sparse、single dense vector 还是 token vectors;再问 matching 是 inverted-index lookup、dot/cosine、MaxSim 还是 joint cross-encoding;最后判断它用于 first-stage retrieval、reranking 还是 feedback learning。

Final-exam model family atlas / 模型家族记忆表

FamilySearches full corpus?Offline costOnline costBest atCharacteristic failure
Lexical sparse: BM25Yes, inverted indexlow indexing costvery lowexact names, rare terms, transparent scoresvocabulary mismatch / 词不相同就难匹配
Dense bi-encoder: DPR, ANCE, Contriever, E5, RepLLaMAYes, ANN vector indexencode every documentencode query + ANNsemantic paraphrasesfalse semantic neighbours; large dense index
Learned sparse: TILDEv2, SPLADEYes when indexed as sparse postingsneural document encodinglow to mediumlexical efficiency plus learned weighting/expansionexpansion noise; model/index coupling
Late interaction: ColBERTYes with specialised indexstore token vectorsmediumfine-grained token matchinglarger index and MaxSim computation
Cross-encoder: monoBERT, duoBERTNo, candidate rerankermodel traininghigh per pairdeep query-document interactioncannot recover documents absent from candidates
Prompted LLM rankerUsually no, candidate rerankerlittle task-specific traininghigh; repeated generation/logitszero-shot instructions and reasoning-rich comparisonprompt/order sensitivity and format failures
LC-LLM / CiCYes only if corpus fits contextconstruct/cache corpus prefixvery high without cachingconsolidated small-corpus searchposition bias, context cost, lost-in-the-middle
RAG systemRetriever doesindex + prompt/evaluation designretrieval + long generationcurrent, inspectable external knowledgedistractors, evidence conflict, unsupported answers
Exam memory chain: BM25 counts and normalises words → DPR compares one vector per text → ColBERT keeps token vectors → SPLADE/TILDEv2 return to sparse term dimensions with learned weights → monoBERT/LLM rankers deeply compare candidates → RAG passes selected evidence to a generator.
Acronym and definition glossary

缩写、英文全称与核心定义

沿用 3208 学习页的阅读顺序:缩写 → 完整英文名 → 中文名 → 一句话定义。不是每个模型名都是首字母缩写;名称型模型会明确标注其作用。

Short nameFull name中文Definition / 定义
IRInformation Retrieval信息检索从文档集合中找到并排序能够满足用户信息需求的内容。
TFTerm Frequency词频某个 term 在当前 document 中出现的次数。
DFDocument Frequency文档频率集合中包含某个 term 的 documents 数量。
CFCollection Frequency集合频率某个 term 在整个 collection 中出现的总次数。
IDFInverse Document Frequency逆文档频率根据 term 在多少文档中出现来衡量其区分度。
TF-IDFTerm Frequency-Inverse Document Frequency词频-逆文档频率结合文档内频率与集合内稀有性的词项权重。
VSMVector Space Model向量空间模型把 query 和 document 表示为向量并比较其方向或相似度。
BIMBinary Independence Model二元独立模型假设词项独立,用词在相关与非相关文档中的概率差异打分。
BM25Best Matching 25最佳匹配第 25 版结合 IDF、term-frequency saturation 与 document-length normalisation 的经典词汇检索模型。
RSJRobertson-Spärck Jones罗伯逊-斯帕克·琼斯权重BM25 常用的概率式 IDF 权重来源,用相关与非相关文档中的词项证据衡量区分度。
F1F-score with β = 1F1 分数Precision 与 recall 的调和平均,在两者同等重要时使用。
APAverage Precision平均精确率对一个 query,在每个相关结果出现位置记录 precision 并平均。
MAPMean Average Precision平均精确率均值对所有 queries 的 AP 取平均。
RR / MRRReciprocal Rank / Mean Reciprocal Rank倒数排名 / 平均倒数排名关注第一个相关结果的位置,再在 queries 间取平均。
DCGDiscounted Cumulative Gain折损累计增益让高相关文档得分更高,并按排名位置进行折扣。
IDCGIdeal Discounted Cumulative Gain理想折损累计增益同一 query 在理想相关性顺序下可获得的 DCG。
nDCGNormalized Discounted Cumulative Gain归一化折损累计增益以 DCG 除以 IDCG,使不同 query 的排名质量可比较。
RBPRank-Biased Precision排名偏置精确率用 persistence 参数模拟用户继续浏览下一结果的概率。
PageRankModel name, not an acronym链接权威度模型用带 teleport 的 random surfer 稳态概率定义 query-independent link authority。
PRPProbability Ranking Principle概率排序原则在相关概率估计准确且 relevance utility 独立等假设下,按 estimated relevance probability 递减排序。
MMRMaximal Marginal Relevance最大边际相关性一种 greedy reranking rule,在 query relevance 与相对已选结果的 redundancy 之间取舍。
α-nDCGAlpha-normalised Discounted Cumulative Gain考虑子主题的归一化折损累计增益对同一 intent 的重复覆盖进行折扣,需要子主题相关性判断。
ERR / ERR-IAExpected Reciprocal Rank / Intent-Aware Expected Reciprocal Rank期望倒数排名 / 意图感知期望倒数排名ERR 模拟用户满意后停止;ERR-IA 按意图权重汇总效用。普通 ERR 不自动衡量意图多样性。
RF / PRFRelevance Feedback / Pseudo-Relevance Feedback相关反馈 / 伪相关反馈从 relevance evidence 更新检索;PRF 不询问用户,直接假设初始 top results 相关。
RocchioModel name, not an acronymRocchio 查询更新让 vector-space query 靠近 relevant centroid,并远离 non-relevant centroid。
TLM / NTLMTranslation Language Model / Neural Translation Language Model翻译语言模型 / 神经翻译语言模型通过可翻译成 query term 的 document terms 建模;NTLM 用 embeddings 估计词项关系。
QPPQuery Performance Prediction查询性能预测预测 query 的潜在检索表现,也可帮助决定 verbose query 应保留多少词。
RRF / CombMNZReciprocal Rank Fusion / Combination Multiply Non-Zero倒数排名融合 / 非零计数加权组合RRF 用 reciprocal rank discount;CombMNZ 用支持列表数放大 score sum。
VByteVariable Byte coding可变字节编码按 7 位一组存整数,每个字节用一位标记后面是否还有字节。
WANDWeak (or Weighted) AND弱与 / 加权与剪枝用列表上界找 pivot,跳过不可能进入 top-k 的文档。
MaxScoreAlgorithm name, not an acronymMaxScore 剪枝算法按上界把词分成 essential 与 non-essential 列表。
DAAT / TAAT / SAATDocument-, Term- and Score-at-a-time按文档 / 按词 / 按分数遍历三种索引遍历顺序:逐文档、逐列表、按 impact 从高到低。
LSRLearned Sparse Retrieval学习式稀疏检索学习词权重(常含扩展词),但仍通过倒排索引检索。
BERTBidirectional Encoder Representations from Transformers双向 Transformer 编码表示使用双向上下文的 encoder 模型,可联合编码 query-document pair。
CLS / SEP / EOSClassification / Separator / End of Sequence token分类 / 分隔 / 序列结束标记分别用于汇总分类表示、分开输入片段,以及标记序列结束。
MLMMasked Language Modeling掩码语言建模遮住部分 tokens,再根据上下文预测它们的预训练目标。
FFNFeed-Forward Network前馈神经网络Transformer block 中对每个 token 独立执行的非线性特征变换。
GPTGenerative Pre-trained Transformer生成式预训练 Transformer以自回归 next-token prediction 为核心的 decoder 模型家族。
LLMLarge Language Model大型语言模型在大规模语料上训练、可通过自然语言 prompt 完成多种任务的模型。
DPRDense Passage Retrieval稠密段落检索分别编码 query 与 passage,并通过 dense-vector similarity 检索。
ColBERTContextualized Late Interaction over BERTBERT 上下文化后期交互模型保留 query 与 document 的 token vectors,以 MaxSim 进行细粒度 late interaction。
ANNApproximate Nearest Neighbour近似最近邻以少量精度损失换取大规模向量搜索速度。
ANCEApproximate Nearest Neighbor Negative Contrastive Estimation近似近邻负样本对比估计利用当前 dense model 的 ANN 检索错误动态挖掘 hard negatives。
NLLNegative Log-Likelihood负对数似然惩罚模型给正确目标分配较低概率的损失函数。
CECross-Entropy交叉熵衡量目标分布与模型预测分布差异的分类损失。
InfoNCEInformation Noise-Contrastive Estimation信息噪声对比估计让正样本相对于一组 negatives 获得更高相似度的对比学习目标。
TILDETerm Independent Likelihood moDEl词项独立似然模型从文档侧学习 vocabulary term likelihood,用于高效查询似然打分。
SPLADESparse Lexical and Expansion Model稀疏词汇扩展模型学习 vocabulary-aligned sparse weights,并可激活原文未出现的相关词。
ReLURectified Linear Unit修正线性单元使用 max(0,x) 去除负激活;SPLADE 用它产生非负词项权重。
FLOPSFloating-Point Operations浮点运算量SPLADE 中以预期计算开销为直觉的稀疏正则项,抑制过多词项被激活。
T5Text-to-Text Transfer Transformer文本到文本迁移 Transformer将所有任务表述为文本到文本生成;doc2query 可用它离线生成文档扩展词。
LTRLearning to Rank学习排序从 labels、preferences 或 lists 学习候选文档的排序函数。
OLTROnline Learning to Rank在线学习排序通过实时用户交互持续评价和更新 ranker。
IPSInverse Propensity Scoring逆倾向评分用观察概率的倒数重新加权点击,以校正曝光或位置偏差。
DBGDDueling Bandit Gradient Descent对决老虎机梯度下降用 interleaving 比较当前与探索 ranker,并向获胜方向更新。
COLTRCounterfactual Online Learning to Rank反事实在线学习排序使用记录了 propensities 的历史点击数据离策略评估候选 rankers。
RM3Relevance Model 3相关性模型 3从初始检索结果估计反馈模型并扩展原 query。
TRECText REtrieval Conference文本检索评测会议提供标准 collections、topics 和 evaluation tracks 的信息检索评测体系。
QrelsQuery Relevance Judgements查询相关性判断记录 query-document relevance labels 的标准答案文件,用于评价 run。
MS MARCOMicrosoft Machine Reading Comprehension微软机器阅读理解数据集由真实搜索查询和人工答案/段落构成,常用于训练与评价 neural rankers。
BEIRBenchmarking Information Retrieval信息检索基准套件跨多个领域测试 retriever 的 zero-shot generalisation。
JSONLJSON Lines逐行 JSON 格式每一行保存一个独立 JSON object,适合流式读取文档集合。
LoRALow-Rank Adaptation低秩适配冻结大部分原模型参数,只训练低秩更新矩阵的高效微调方法。
MNTPMasked Next Token Prediction掩码下一词预测通过遮住 token 并利用双向上下文训练 decoder 表示的目标。
SimCSESimple Contrastive Learning of Sentence Embeddings句向量简单对比学习以对比目标学习句子表示的方法。
PARADEPassage Representation Aggregation for Document Reranking面向文档重排序的段落表示聚合先编码长文档的各 passages,再聚合其 representations 做文档级评分。
LLM4IRLarge Language Models for Information Retrieval用于信息检索的大语言模型使用 LLM 改善 retrieval、reranking 或其他 IR tasks。
IR4LLMInformation Retrieval for Large Language Models用于大语言模型的信息检索通过 retrieval 为 LLM 提供外部、可更新的证据。
monoBERTModel name, not an acronym单文档 BERT 重排器对每个 query-document pair 独立输出 pointwise relevance score。
duoBERTModel name, not an acronym双文档 BERT 重排器比较两个 candidates,输出对 query 的 pairwise preference。
RAGRetrieval-Augmented Generation检索增强生成在生成前从外部 collection 检索证据,并把 query 与 evidence 一起交给生成模型。
LC-LLMLong-Context Large Language Model长上下文大语言模型能够接收远长于传统模型输入窗口的 prompt,但成本与位置偏差仍需考虑。
CiCCorpus-in-Context语料库置于上下文把整个可容纳 corpus 与示例放入 LC-LLM prompt,并生成相关 document IDs。
CoTChain-of-Thought思维链提示通过示例或指令诱导模型生成中间推理步骤;生成文字不等于可验证的内部推理。
VLMVision-Language Model视觉语言模型联合处理像素与文本;screenshot retrieval 用它保留页面布局和视觉内容。
HyDEHypothetical Document Embeddings假想文档嵌入先生成一个假想相关文档,再用其表示检索真实文档。
RARRRetrofit Attribution using Research and Revision通过研究与修订补加归因为已有答案搜索支持证据并修正无支持内容。
PPOProximal Policy Optimization近端策略优化一种限制单次策略更新幅度的强化学习方法;可训练 query rewriter。
AttributionEvidence attribution证据归因把生成答案中的 claims 与真实、支持它们的 references 或 citations 对齐。
Assignment 1

A1 完整实施流程

作业不是只写一个 BM25 函数,而是完成一个可验证、可调参、可比较的 IR 实验。

为什么分 train/test

Train qrels 用来选择 k1、b;test qrels 只用于最终一次无偏评价。若看着 test 结果反复改参数,就把 test 泄露成了 train,最终分数会过于乐观。

怎样讨论 tuned BM25

不要只写“分数提高”。解释 k1 改变 tf saturation、b 改变 length normalization,并结合 gain/loss queries 判断数据集是偏短文档、长文档,还是 query 中 rare terms 的作用更明显。

打开实验复习流程. 公开版仅保留方法讲解,不包含个人提交文件和实验输出。
# 一个清晰的 scorer 骨架(变量名按你的 notebook 调整)
def bm25(tf_vector, df_vector, doc_len, k1=1.2, b=0.75):
    score = 0.0
    for tf, df in zip(tf_vector, df_vector):
        idf = math.log(N / (1 + df))
        norm = 1 - b + b * (doc_len / average_doc_length)
        tf_weight = ((k1 + 1) * tf) / (tf + k1 * norm)
        score += idf * tf_weight
    return score
Recall practice

自测题

先在脑中回答,再展开答案。能够解释“为什么”才算掌握。

1. tf、df、cf 分别是什么?
tf:term 在当前 document 的次数;df:包含 term 的 documents 数;cf:term 在整个 collection 中的总次数。
2. 为什么 BM25 不让 tf 线性增长?
同一词从 0 到 1 很重要,但从 100 到 101 提供的信息很少。TF saturation 防止重复堆词无限提高分数。
3. nDCG 为什么要除以 IDCG?
不同 query 的相关文档数量和等级不同;归一化后才能把它们放在近似 0–1 尺度上比较和平均。
4. p-value < .05 说明什么?
在 H₀ 成立时,观察到当前或更极端差异的概率低于 5%。它不是系统 B 更好的概率,也不直接表示提升幅度。
5. Self-attention 为什么除以 √d_k?
维度增大时点积方差随之增大;缩放可避免 softmax 过尖,保持梯度稳定。
6. Encoder 和 decoder 的关键区别?
Encoder 一般可双向看完整输入,擅长理解;decoder 使用 causal mask,只看左侧前缀,擅长自回归生成。
7. Dense retrieval 为什么能处理同义词?
模型通过训练把语义相近表达映射到邻近向量,不再要求 query 与 document 必须共享完全相同的表面词项。
8. ColBERT 分数怎么计算?
对每个 query token 找到所有 document tokens 中的最高相似度,再把各 query token 的最大值相加,即 MaxSim。
9. Hard negatives 为什么有用又危险?
它们与 query 很像但被标为不相关,能提供强梯度;但 qrels 不完整时,其中可能实际相关,形成 false negatives。
10. 为什么不能在 test set 调 BM25 参数?
这会让参数适配 test qrels,破坏 test 作为未见数据的角色,导致对 generalization 的估计偏乐观。
11. Why is SPLADE sparse even though it uses BERT?
Sparsity describes the output: a vocabulary-sized vector with mostly zero entries. BERT is the neural mechanism that learns which vocabulary dimensions to activate.
12. TILDEv2 and SPLADE differ most clearly where?
TILDEv2 keeps query processing lightweight with token IDs and learns document-token weights, while SPLADE encodes both query and document into learned sparse vocabulary vectors.
13. Why use BM25 before TILDEv2 reranking?
BM25 cheaply reduces the full corpus to a candidate set. TILDEv2 then spends richer document-side scoring only on those candidates.
14. Pointwise, pairwise and listwise LTR differ how?
The training unit is respectively one document, a document preference pair, or an entire ranked list.
15. Why can clicks not be used as relevance labels directly?
Clicks depend on examination, position, selection, presentation and noise. An unclicked item may simply never have been seen.
16. What is the main purpose of IPS?
To correct exposure or position bias by weighting an observed click with the inverse probability that the item was examined.
17. How does balanced interleaving compare two rankers?
It alternates unseen documents from both rankers into one displayed list and attributes clicks to infer a pairwise preference under nearly identical context.
18. Counterfactual OLTR versus regular interleaving?
Counterfactual OLTR can reuse one propensity-logged interaction stream to estimate many candidates off-policy; pairwise interleaving normally needs a newly mixed list for each direct comparison.
19. Why must monoBERT follow a first-stage retriever?
Joint query-document encoding is too expensive for every document. BM25 first supplies a manageable candidate set; monoBERT can only reorder that set.
20. Score aggregation and representation aggregation differ how?
MaxP/FirstP/SumP combine scalar passage scores. PARADE first combines learned passage vectors, allowing a richer document-level decision.
21. monoBERT and duoBERT differ how?
monoBERT independently estimates relevance for one document; duoBERT directly estimates which of two documents should rank higher.
22. Pointwise, pairwise, listwise and setwise LLM reranking differ how?
They respectively judge one document, compare two, generate an order for a list, or select the best item from a small set.
23. Why are setwise LLM rankers potentially cheaper?
One inference compares several passages using logits, requiring fewer comparisons than pairwise ranking and less generation than a full listwise permutation.
24. Why can prompt wording change ranking effectiveness?
Role text, output constraints, formatting and component order alter the model context. Sensitivity also depends on model size, architecture and training.
25. LLM4IR 与 IR4LLM 的方向有什么不同?
LLM4IR 用 LLM 改善 retrieval/ranking;IR4LLM 用 retrieval 给 LLM 提供外部证据。一个 LLM reranker 属于前者,RAG 属于后者。
26. Why might CiC replace a retriever for a small corpus?
If the corpus fits the context window, the LC-LLM can read it directly and generate document IDs, avoiding a separate retrieval model. Cost, caching and position bias remain.
27. 为什么 long context 不等于模型会使用所有证据?
注意力使用有位置偏差,关键证据位于中间时可能被忽略;更多 distractors 也会稀释或冲突关键证据。
28. RAG 与 fine-tuning 的知识更新方式有什么不同?
RAG 在 inference 时替换外部 evidence;fine-tuning 修改参数。RAG 更容易更新和检查来源,fine-tuning 不需要每次提供证据但知识不容易单独替换。
29. What can happen when retrieved evidence contradicts model knowledge?
The model may ignore the evidence, preserve a wrong prior answer, or flip a correct answer. Groundedness must therefore be evaluated separately from retrieval relevance.
30. 为什么 prompt compression 不是免费压缩?
它减少 tokens 与成本,但 lexical deletion 或 learned memory tokens 都可能丢掉限定条件、来源或少数关键证据。
31. Rewrite-Retrieve-Read 的三个对象分别是什么?
Rewriter 产生更适合搜索的 query;retriever 找 evidence;reader 根据原问题与 evidence 生成答案。Rewrite drift 是主要风险。
32. What is the difference between relevance and groundedness?
Relevance asks whether a passage helps the information need; groundedness asks whether answer claims are actually supported by supplied evidence.
33. 为什么 nDCG 提升不保证答案分数提升?
生成还受 passage truncation/order、prompt、模型先验和证据冲突影响。只能通过对齐 query 的差值与 correlation 实证检查。
34. What does attribution add beyond a fluent answer?
It identifies evidence supporting each claim, allowing verification. A fluent answer without valid support may still hallucinate.
35. Screenshot retrieval 解决了 text-only retrieval 的什么损失?
它保留布局、图像、表格和空间关系等纯文本抽取可能丢失的信号,但增加视觉编码成本。
36. RARR 的顺序是什么?
先有 draft answer,再 research evidence,最后 revise unsupported content 并补充 attribution;它是 post-generation repair,而不是 first-stage retrieval。
37. Link authority 与 anchor text 为什么是不同信号?
Authority 来自链接图以及 endorsing pages 的重要性;anchor text 是链接作者对 target page 的文字描述。前者是 structural evidence,后者是 descriptive evidence。
38. 什么样的 incoming link 会贡献更多 PageRank?
来源页自身 PageRank 更高、out-links 更少时,传入质量更大;teleportation 与 dangling-page handling 让概率分布保持良定义。
39. 为什么 PageRank 不是 relevance score?
它是 query-independent graph importance。Ranker 可把它当 document prior 或 feature,与 query-dependent lexical/semantic evidence 组合,再用目标指标实证验证。
40. MMR 如何弥补纯 PRP 排名的局限?
只按 relevance 排序可能重复同一 aspect;MMR greedy 地平衡 relevance 与相对已选结果的 similarity,提升 novelty 与 coverage,但可能牺牲一部分 relevance。
41. Explicit、implicit 与 pseudo relevance feedback 有何不同?
Explicit 是用户直接判断;implicit 从行为推断并继承行为偏差;PRF 不询问用户,直接假设初始 top results 相关。
42. Rocchio 三部分分别做什么?
Original query 锚定方向;relevant centroid 产生吸引;non-relevant centroid 产生排斥。α、β、γ 控制三者强度。
43. 为什么 PRF 可能让检索变差?
若首轮顶部文档属于错误词义或主题,它们的词会把 reformulated query 拉离真实 information need,形成 query drift。
44. CombSUM 前为什么要 normalise scores?
不同 rankers 的范围、正负号和分布可能不兼容;不做 per-list normalisation,数值天然较大的系统会无理由支配融合结果。
45. RRF 与 CombMNZ 的证据有什么不同?
RRF 只用 rank positions,并折扣深层结果;CombMNZ 使用 normalized scores,并用返回该文档的列表数量放大 score sum。
46. Delta coding 为什么有用,代价是什么?
递增 identifier 之间的差值是小整数,便于压缩。代价是取第 n 个 identifier 要做 prefix sum;block-based index 直接存每块首 identifier,把累加限制在一个块内。
47. 为什么存量化后的 impact,而不是 term frequency?
可加模型的词—文档贡献可以预先算好;映射成小整数后既能压缩,查询时也只需把 impacts 相加。
48. Dynamic pruning 什么时候可以安全跳过一个文档?
当它所在各列表的上界之和不可能超过当前 top-k 阈值时。最终 top-k 不变,所以仍是 rank-safe。
49. DAAT、TAAT 与 SAAT 有什么区别?
DAAT 一次处理一个文档、同时遍历所有列表并用 heap;TAAT 一次处理一个列表、每个文档一个 accumulator;SAAT 按 impact 从高到低处理 postings,可以提前停止。
50. 为什么 learned sparse 索引更难剪枝?
Expansion 和被挪用的 token 产生了很长且 learned impact 很高的列表,上界一直很高,能跳过的候选变少,要打分的 postings 变多。
Source map

本手册整理范围

内容来自你提供的本地 INFS7410 课程目录;大体积 corpus、index、venv 只作为实验资源,不纳入复习正文。

  • Week 1 lecture notes / slides / practical notebook
  • Week 2 lecture notes / slides / practical notebook
  • Week 3 lecture notes / slides / practical notebook
  • Week 4 lecture notes / transformer worksheets / practical solution
  • Week 5 lecture notes / slides and DPR, ColBERTv2, RepLLaMA practical
  • Week 5 InfoNCE practical solution
  • Week 5 MarginLoss practical solution
  • Week 5 HardNegatives practical solution
  • Week 6 practical: TILDEv2 and SPLADEv2
  • Week 6 learned sparse lecture notes
  • Week 7 lecture and MQ2007 practical: logistic LTR, DBGD, interleaving and click simulation
  • Week 8 full slide deck: BERT/LLM reranking, long-document aggregation and prompt variation
  • Week 8 practical: BM25 + monoBERT pipeline and LLM pointwise/pairwise reranking
  • Week 9 lecture and practical: long-context retrieval, RAG construction, prompting, evidence checks and thinking-mode comparison
  • Week 10 full slide deck, live Q/A deck and practical: search agents, ReAct, Search–Visit, SIEVE, DCI, RISE, AgentIR, LRAT and ITER.
  • Week 11 lecture and five-page PageRank practical: link analysis, anchor text, document priors, PRP, ambiguity, redundancy, MMR and diversity evaluation. Original course files remain private.
  • Week 12 released lecture and practical: relevance feedback, PRF expansion, IDF-r reduction, translation language models, Borda/CombSUM/CombMNZ rank fusion and score normalisation. Its official Live Q/A deck was not available when checked; any Week 12 questions here are self-authored practice.
  • Week 13 guest lecture on efficient inverted indexes (compressed postings, quantized impacts, dynamic pruning, DAAT/TAAT/SAAT and learned sparse efficiency) and the final oral exam briefing. No Week 13 Live Q/A deck was available; Week 13 questions here are self-authored practice.
  • Project 2 notebook, sampled BRIGHT Biology collection and marking sheet
  • A1 project notebook
  • A1 marking sheet
Coverage / 范围:This edition integrates Weeks 1–13. Week 6 connects lexical and dense retrieval; Week 7 covers click feedback; Week 8 neural/LLM reranking; Week 9 evidence-based generation; Week 10 closes the agent loop; Week 11 adds web ranking; Week 12 adds feedback-driven query reformulation and rank fusion.
Question directory

独立编写的练习题与复习 Q/A

按周查找 67 个复习主题:依据课程知识点,用重新组织的场景与计算示例练习,不是官方原题或官方答案册。点击主题后先尝试作答,再展开讲解;不公开课件的原始题目文字。

打开全部题目、搜索与复习进度

Week 1 (7)
  1. Information need, query and relevance
  2. Search architecture
  3. Stemmers
  4. Zipf’s law
  5. Domain-specific stoplists
  6. Text processing true/false
  7. Search in the LLM era
Week 2 (6)
  1. Precision, AP and MAP
  2. Reciprocal rank
  3. Normalised DCG
  4. Rank-Biased Precision
  5. Working with qrels
  6. Implement evaluation functions
Week 3 (6)
  1. Lexical models: features and limitations
  2. Normalised term frequency
  3. Inverse document frequency
  4. TF-IDF term weights
  5. TF-IDF query score
  6. Repeated terms and BM25 components
Week 4 (5)
  1. Vocabulary mismatch and self-attention
  2. Transformer components
  3. BERT versus GPT
  4. Input embedding matrix
  5. Use encoders/decoders for ranking
Week 5 (5)
  1. Encoder text representations
  2. Decoder similarities
  3. InfoNCE calculation
  4. Margin loss calculation
  5. Harder negatives
Week 6 (5)
  1. Why TILDE is efficient
  2. SPLADE term weights
  3. PromptReps hybrid retrieval
  4. SPLADEv1 sum pooling
  5. SPLADEv2 max pooling
Week 7 (6)
  1. Offline Learning to Rank
  2. Cascade ranking
  3. Worked interleaving preference
  4. IPS estimator
  5. DBGD with balanced interleaving
  6. Counterfactual OLTR
Week 8 (4)
  1. monoBERT versus dense retrieval
  2. monoBERT limitations
  3. Handle long documents: MaxP and PARADE
  4. Four decoder reranking methods
Week 11 (9)
  1. Link authority and anchor text
  2. PageRank calculation logic
  3. PageRank versus relevance
  4. PRP assumptions and MMR diversification
  5. Why must PageRank preserve probability mass? / PageRank 为什么不能丢概率?
  6. Does λ always mean the same thing? / 不同公式中的 λ 是同一个参数吗?
  7. Calculate the second MMR result / 手算 MMR 的第二名
  8. Can MMR be computed once for every document? / MMR 能只打分一次然后排序吗?
  9. Does ordinary nDCG prove good intent coverage? / 普通 nDCG 高能证明覆盖更多意图吗?
Project 2 (1)
  1. Which data belongs to the main task and which to the bonus? / 主任务和 bonus 分别使用哪组数据?
Week 12 (8)
  1. Which feedback signal should be trusted most? / 哪类反馈最可信?
  2. Calculate a Rocchio update / 手算一次 Rocchio 更新
  3. Why can pseudo-relevance feedback drift? / PRF 为什么会漂移?
  4. Compare two RRF candidates / 比较两个 RRF 候选
  5. Why normalise scores before CombSUM? / CombSUM 前为什么归一化?
  6. What must stay consistent in IDF-r reduction? / IDF-r 精简中哪些设置必须一致?
  7. Avoid Borda's indexing trap / 避免 Borda 的下标错误
  8. Compare CombSUM and CombMNZ / 比较 CombSUM 与 CombMNZ
Week 13 (5)
  1. Use block starts to find a document identifier / 用块首 identifier 查找文档
  2. Encode integers with Variable Byte / 用 Variable Byte 编码整数
  3. Quantize a BM25 contribution / 量化一个 BM25 贡献
  4. Decide which documents need scoring / 判断哪些文档需要打分
  5. Explain why learned sparse retrieval is slower / 解释 learned sparse 为什么更慢
没有找到匹配内容。试试更短的关键词。