INFS7410 Information Retrieval

Complete review map

From term matching to evidence-grounded generation

The course forms one connected Information Retrieval (IR) pipeline: index documents, retrieve candidates, rerank them, evaluate and learn from feedback, then provide evidence to a generator. Study sequence: retrieval → reranking → evaluation → RAG → attribution.

Text → terms Terms → index Index → ranking Ranking → evaluation Sparse → dense semantics Dense → learned sparse Qrels → clicks Candidates → BERT / LLM reranking Evidence → RAG answer Claims → attribution

9connected weeks

30+key formulas

35+retrieval, ranking and RAG methods

7 + projectspracticals and assignments

Mental model

What an information retrieval system does

Place each weekly topic inside one software pipeline instead of memorising isolated functions.

Mastered

Corpus
raw document collection

→

Analyzer
tokens, stopwords, stems

→

Inverted index
term → postings

→

Ranker
BM25 / dense model

Query
user information need

→

Top-k run
ranking for each query

→

Qrels
relevance judgements

→

Evaluation
MAP / nDCG / tests

Four objects to keep separate: the corpus contains all documents; a query expresses a need; a run is the system ranking; and qrels are the reference relevance judgements. Evaluation compares a run with qrels.

Candidate retrieval

BM25, dense and learned sparse models search the full collection.

→

Ranking

Rerankers and LTR models reorder a manageable candidate set.

→

Evidence

Qrels support offline metrics; clicks require bias-aware online methods.

Week 1

Indexing, text statistics, and preprocessing

Understand how a search engine converts a large text collection into a structure that can be searched efficiently.

Week 1 complete

IR, databases, and relevance

Database queries usually require exact matching and return deterministic answers. IR handles natural language and uncertain information needs, so it returns a ranked list. Relevance means whether a document satisfies a user's need in context, not merely whether it shares words with the query.

Core chain: representation + matching + ranking + evaluation.

Corpus, document, token, and term

A corpus/collection is the full set of documents. A document is one record, a token is one occurrence produced by tokenisation, and a term is the normalised unit stored in the index.

tf occurrences in one document df documents containing the term cf total occurrences in the collection

Zipf's law

A small number of terms are extremely frequent, while most terms are rare. Frequency falls approximately as a power law with rank.

frequency(r) ≈ C / r^s

This produces a long tail. Very frequent items are often function words; very rare items may be names, spelling variants, or noise.

Text processing

Tokenisation: split text into indexable units.
Case folding: usually convert text to lowercase.
Stopping: remove frequent, weakly discriminative words, with some risk of losing phrase meaning.
Stemming: apply rules such as Porter stemming; the result need not be a real word.
Lemmatisation: use lexical knowledge to recover dictionary forms.

Inverted index

A forward representation is document → terms. An inverted index stores term → postings list, allowing the engine to find candidate documents without scanning the entire corpus.

Level	Posting contents	Supported capability
Boolean	docID	presence and AND / OR / NOT
Counts	docID + tf	TF, TF-IDF, and BM25 ranking
Positions	docID + tf + positions	phrase and proximity queries
Doc vectors	terms and statistics per document	custom scorers read tf/df; A1 requires `-storeDocvectors`

Common error: df is not the total number of occurrences. A term appearing 20 times in one document contributes 1 to df but 20 to cf.

What the Week 1 practical develops

Reading a JSONL corpus, counting and inspecting documents, plotting Zipf-style term distributions, building Lucene indexes with different stemming/stopword configurations, and verifying the index with index_reader.stats().

Week 2

Evaluation: proving that a system is better

Offline evaluation requires queries, qrels, and runs. Each measure represents a different user model.

Week 2 complete

Precision and Recall

Precision = |Relevant ∩ Retrieved| / |Retrieved|

Recall = |Relevant ∩ Retrieved| / |Relevant|

Precision asks how clean the returned set is. Recall asks how much of the relevant set was found. Web results often emphasise early precision; legal and medical search may place more weight on recall.

F-measure

F_β = (1 + β²)PR / (β²P + R)

F1 = 2PR / (P + R)

F1 is the harmonic mean of precision and recall. β>1 emphasises recall; β<1 emphasises precision.

AP and MAP

AP = (1 / |Rel|) Σ_k=1..K P@k × rel(k)

MAP = (1 / |Q|) Σ_q∈Q AP(q)

P@k contributes only when the item at rank k is relevant. Relevant documents that were never retrieved remain in |Rel|, reducing AP.

RR and MRR

RR(q) = 1 / rank(first relevant result)

MRR = mean_q RR(q)

These measures care only about the first relevant answer, making them suitable for question answering and navigational search.

DCG and nDCG

DCG@K = Σ_k=1..K (2^rel(k) − 1) / log₂(k + 1)

nDCG@K = DCG@K / IDCG@K

Graded relevance receives exponential gain and lower ranks receive a discount. IDCG is the score of the ideal ordering for the same query.

RBP

RBP = (1 − p) Σ_i=1..d r_i pⁱ⁻¹

p is the probability that the user continues to the next result. Expected viewing depth is 1/(1-p).

Paired t-test

Score the same queries with systems A and B, calculate one difference per query, then test whether the mean difference departs from zero.

d_q = metric_B,q − metric_A,q
H₀: mean(d) = 0

p < .05 is commonly called significant; p < .01 strongly significant.

Interpreting a p-value

It is the probability of observing the current or a more extreme result if H₀ were true. It is not the probability that H₀ is true and does not measure effect size.

Check evaluator keys: KeyError: ndcg_cut_5 normally means that measure was not requested when the evaluator was created.

The Week 2 practical pipeline

search(..., k=1000) creates a run → pytrec_eval.parse_run reads it → the evaluator scores each query → aggregation produces the mean → scipy.stats.ttest_rel compares paired query scores. Here k is the number of retrieved documents, not BM25's k1.

Week 3

Lexical matching and classical ranking

Start with term presence, then add frequency, rarity, length normalisation, and probabilistic reasoning.

Week 3 complete

Course diagrams: open the course platform (login required)

Model	Core idea	Score / role	Main limitation
Boolean	Satisfy logical conditions	AND / OR / NOT postings	Little or no ranking
Coordination	Match more distinct query terms	number of matched query terms	Ignores frequency and rarity
TF	More occurrences in a document are better	Σ f(q_i,D)	Favours common terms and long documents
IDF	Rare terms are more discriminative	log(N/(1+df_i))	Ignores within-document tf
TF-IDF	Local frequency × global rarity	Σ tf_i,D × idf_i	Linear tf and crude length treatment
VSM	Compare query and document vector directions	cosine(q,d)	Still depends on lexical overlap
BIM	Contrast term probability in relevant/non-relevant sets	probabilistic relevance weight	Term independence and probability estimates
BM25	IDF + tf saturation + length normalisation	strong sparse baseline	No native synonym or semantic matching

Course TF, IDF, and TF-IDF

TF(D,Q) = Σ_i=1..|Q| f(q_i,D)

IDF(q_i) = log(N / (1 + df_i))

TF-IDF(D,Q) = Σ tf_i,D × idf(q_i)

1 + df smooths the denominator. IDF depends on the term and collection, not on the current document.

Vector Space Model

cos(q,d) = (q · d) / (||q|| ||d||)

A dot product is affected by vector magnitude. Cosine normalises magnitude and compares direction. For L2-normalised vectors, dot product and cosine produce the same ranking.

BM25, factor by factor

score(D,Q) = Σ_i IDF(q_i) × [(k₁+1)tf_i] / [tf_i + k₁(1 − b + b·|D|/avgdl)]

IDF: rewards discriminative rare terms.
TF saturation: the gain from tf 1→2 is larger than 100→101.
Length normalisation: controls the natural advantage of longer documents.
k1: controls how quickly tf saturates; larger values are more nearly linear.
b: controls length normalisation, from none at 0 to full at 1.

RSJ-IDF ≈ log((N − df + 0.5) / (df + 0.5))

Several BM25 variants exist. In A1, follow the formula in the supplied notebook and marking sheet rather than mixing definitions.

Why the Week 3 scorer receives three arguments

tf_vector stores each query term's tf in the current document; df_vector stores each term's collection df; and doc_len is the current document length. The scorer returns one total score used to rank and write the run.

What the three assertions test

assert score(tf=3) > score(tf=1) checks tf monotonicity; assert score(short_doc) > score(long_doc) checks length normalisation; and assert np.isfinite(...) catches NaN or infinity. A false condition raises AssertionError.

Week 4

Transformers: from tokens to contextual embeddings

Classical sparse models require lexical overlap. Transformers learn context-dependent representations that support semantic matching.

Week 4 complete

Course diagrams: open the course platform (login required)

Tokenisation and embedding

A tokenizer maps text to token IDs. nn.Embedding(V,D) is a trainable lookup table mapping each ID to a D-dimensional vector. D is a model hyperparameter; BERT-base uses 768.

input_i = token_embed_i + position_embed_i + segment_embed_i

The usual shape is N × D: N tokens, each represented by D values.

Position and segment embeddings

Self-attention alone does not encode order, so position embeddings are added. For sentence pairs, BERT uses segment/token-type embeddings and [SEP] boundaries.

WordPiece: rare words can be split into subwords; ## marks a continuation piece.

Self-attention

Q = XW_Q, K = XW_K, V = XW_V

Attention(Q,K,V) = softmax(QK^T / √d_k)V

Each token produces a query (what am I looking for?), key (what can I match?), and value (what content can I provide?).
QKᵀ produces an N×N score matrix.
Division by √d_k prevents large dot products from saturating softmax.
Softmax gives row-wise weights summing to one; multiplying by V forms context-aware representations.

Multi-head attention

Several heads learn relationships in different projection subspaces. Their outputs are concatenated and projected back to D dimensions.

FFN, hidden, and residuals

hidden = GELU(W₁x + b₁)
out = W₂hidden + b₂

The FFN transforms each token independently, often D→4D→D. hidden is an intermediate activation tensor, not a manually chosen parameter. The learned parameters are W and b. Residual connections preserve information; LayerNorm stabilises training.

Encoder and BERT

An encoder can use left and right context, making it suitable for understanding, classification, ranking, and embeddings. BERT's main pretraining objective is masked language modelling.

L_MLM = −Σ_i∈M log P(x_i | x_\M)

Decoder and GPT

A decoder uses a causal mask and sees only the left prefix, enabling token-by-token generation. An encoder backbone cannot naturally generate in the same way without an added decoder or modified objective and attention mask.

P(y₁…y_T) = Π_t=1..T P(y_t | y<t)

Why the Week 4 practical produced NoneType.shape

If out = tf_block(x) is None, forward() or a called method is missing return x/return out. Python functions without return statements return None, so out.shape fails.

Week 5

Dense retrieval, objectives, and negatives

Map queries and documents into a vector space where “car repair” can be close to “automobile maintenance.”

Week 5 complete

Course diagrams: open the course platform (login required)

Bi-encoder

score(q,d) = φ(η_q(q), η_d(d))

Queries and documents are encoded separately. Document vectors can be precomputed and indexed for ANN search, making bi-encoders suitable for first-stage retrieval.

Cross-encoder

[query; document] is encoded jointly, allowing deep token interaction. This is accurate but prevents offline document encoding, so it is usually used to rerank a smaller candidate set.

Pooling choices

CLS/first token: use the first special token's contextual embedding.
Mean pooling: average valid token vectors.
EOS/last token: common with decoders because the last token has seen the full prefix.
All-token: retain every token vector, as in ColBERT.

Similarity

dot(q,d) = Σ q_id_i

cos(q,d) = q·d / (||q|| ||d||)

For L2-normalised vectors, dot product and cosine rank identically. “Mean-pooling similarity” is imprecise: mean pooling aggregates a representation; it is not a similarity function.

InfoNCE / contrastive loss

L = −log [ exp(sim(q,d⁺)/τ) / (exp(sim(q,d⁺)/τ) + Σ_j exp(sim(q,d⁻_j)/τ)) ]

The objective raises positive similarity relative to a set of negatives. A lower temperature τ makes softmax sharper. In-batch negatives reuse other queries' positive documents as negatives at little extra encoding cost.

Multi-label margin loss

L = mean_(d⁺,d⁻) max(0, 1 − [s(q,d⁺) − s(q,d⁻)])

A pair contributes zero loss once the positive score exceeds the negative score by at least the margin.

ColBERT late interaction

MaxSim(q,d) = Σ_i=1..m max_j=1..n E_q,i · E_d,j

For each query token, find the most similar document token, then sum those maxima. This is not a single dot product between one query vector and one document vector.

Course diagrams: open the course platform (login required)

Negative type	Source	Advantage	Risk
Random	random collection documents	cheap and stable	often too easy
In-batch	other positives in the batch	almost no extra encoding	false negatives
BM25 hard	high-ranked but judged non-relevant	lexically similar and challenging	depends on qrel completeness
ANN/self-mined	current model's mistaken neighbours	targets current weaknesses	costly index refresh

What a smoke query or smoke test means

A smoke test checks that the pipeline runs at all; it does not prove quality. Run one smoke_query through TF and BM25, inspect top-5 output, and check the structure and finite scores. Formal conclusions still require all test queries and qrels.

Week 6

Learned sparse retrieval

Vocabulary-aligned sparse representations retain inverted-index efficiency while neural models learn contextual term weights and expansion.

Week 6 complete

The missing middle: BM25 is sparse and fast but lexical; DPR is semantic but dense. Learned sparse models use neural encoders while keeping mostly-zero vocabulary dimensions that can be interpreted as terms.

Model	Query	Document	Matching	Online cost
BM25	observed terms	term statistics	hand-crafted weighted sum	very low
DPR	dense vector	dense vector	dot product + ANN	query encoder
TILDEv2	token IDs	contextual token weights, often expanded offline	exact ID match + max weight	no query encoder
SPLADE(v2)	learned sparse vocabulary vector	learned sparse vocabulary vector	sparse dot product	query encoder
PromptReps	dense hidden state + sparse logits	same	hybrid interpolation	LLM inference

Expansion

doc2query generates likely queries offline and appends them to documents before indexing. This reduces vocabulary mismatch without adding query-time generation.

TILDEv2

score(q,d) = Σ_t∈q max_{i: token(d_i)=t} w_i,d

The query is only tokenised. Exact query-token matches retrieve learned contextual document weights; repeated terms contribute their maximum weight.

SPLADE

score(q,d) = v_q · v_d, v ∈ R^|V| and mostly zero

An MLM head activates weighted vocabulary terms, including related terms absent from the text. ReLU, pooling and FLOPS/L1 regularisation control sparsity.

Sparse can still be neural

Sparse/dense describes the output representation. SPLADE uses BERT but emits a sparse vocabulary vector; DPR uses BERT but emits a dense embedding.

Practical pipeline: BM25 + TILDEv2

Retrieve

BM25 searches the full corpus.

→

Read candidates

Use hit IDs and raw document text.

→

Rerank

TILDEv2 scores query-document pairs.

The first stage protects efficiency and candidate recall; the second stage improves ordering within the candidate set.

Data flow inside tildev2_scoring

Document text becomes token IDs and learned weights. Query text becomes token IDs with stopwords removed. Equal IDs are matched, repeated matches use the maximum document weight, and matched query-term weights are summed into one reranking score.

Week 7

Learning to rank, click bias, and online evaluation

LTR learns to order candidates. Online LTR replaces scarce static labels with abundant but biased user interactions.

Week 7 complete

Retriever → features → LTR ranker

Retriever

BM25, dense or learned sparse candidates.

→

Features

Lexical, neural, freshness and authority signals.

→

LTR ranker

Learned ordering of the candidate list.

Offline LTR

Pointwise: predict one relevance score.
Pairwise: predict which of two documents should rank higher.
Listwise: optimise an entire ranked list.

Clicks are not labels

Position bias changes examination; selection bias excludes unseen documents; presentation bias changes attraction. No click does not necessarily mean non-relevant.

Inverse Propensity Scoring

Δ_naive(f) = Σ λ(rank_i)c_i

Δ_IPS(f) = Σ [λ(rank_i)/P(o_i=1)]c_i

IPS corrects position/exposure bias in expectation by dividing a click by its examination probability. Rarely examined observations receive more weight, which can also increase variance.

Balanced interleaving

Maintain a scan pointer for each ranked list. Inspect the list with the smaller pointer, breaking ties with a randomly chosen starting ranker. Advance the pointer even for duplicates, but append only unseen documents. This balances inspected depths, not necessarily the number contributed by each ranker.

DBGD

Sample a candidate parameter direction, compare current and candidate rankers through interleaving, and move toward the winner.

Counterfactual OLTR

A stochastic logging policy records displayed actions and propensities. IPS then estimates the risk of current and candidate rankers from the same interaction log.

Main advantage over regular interleaving: one logged randomized policy can evaluate multiple candidate rankers off-policy instead of creating a fresh interleaved comparison for every pair. It depends on accurate propensities, exploration/support, and manageable variance.

Method	Data	Output	Main trade-off
Offline qrels	queries + runs + qrels	MAP/nDCG	cleaner but expensive and static
Interleaving	live clicks on a mixed list	pairwise preference	sensitive, but requires live exposure
DBGD	repeated interleaving	parameter update direction	online adaptation versus exploration risk
Counterfactual IPS	logged clicks + propensities	off-policy value/risk	log reuse versus variance and support assumptions

Week 8

Transformer-based reranking: BERT and LLM rankers

Encoder rerankers jointly encode query-document pairs. Decoder-based Large Language Models (LLMs) judge, compare, or order passages through prompts. Stage one retrieves candidates; stage two makes more expensive, fine-grained relevance judgements.

Week 8 complete

Retrieval → interaction → final ranking

First-stage retrieval

BM25 searches the full collection and returns top-k candidates.

→

Neural reranking

BERT jointly encodes pairs, or an LLM evaluates candidates through a prompt.

→

New order

Sort by scores, pairwise preferences, or an output permutation.

A reranker cannot recover a relevant document omitted by Stage 1. Increasing candidate depth can protect recall but increases latency. Candidate depth connects effectiveness with efficiency.

monoBERT: pointwise cross-encoder

Bidirectional Encoder Representations from Transformers (BERT) reads [CLS] query [SEP] document [SEP]. A classifier converts the contextual [CLS] vector into a relevance score.

s(q,d) = P(relevant = 1 | q,d)

Each document is judged independently. Training commonly uses human positives and BM25 hard negatives.

Training objective

L = -Σ_pos log(s) - Σ_neg log(1-s)

Binary cross-entropy rewards high positive scores and low negative scores. The output is a ranking signal, not necessarily a calibrated real-world probability.

Long-document evidence

Method	Aggregation	Interpretation
BERT-FirstP	first passage score	cheap but ignores later evidence
BERT-MaxP	maximum passage score	one strong passage represents the document
BERT-SumP	sum of passage scores	accumulate evidence
Birch	first-stage score + weighted sentence scores	sentence-level evidence
PARADE	average, max, attention, or Transformer over passage vectors	representation aggregation

Key distinction: MaxP/FirstP/SumP combine scalar scores; Passage Representation Aggregation for Document Reranking (PARADE) combines learned passage representations before the final score.

duoBERT: pairwise reranking

duoBERT reads a query and two candidates to estimate p(dᵢ > dⱼ | q). Pairwise wins are then aggregated.

s(dᵢ) = Σ_j≠i p(dᵢ > dⱼ | q)

This can refine a monoBERT list, but pair comparisons add substantial encoder cost.

Multi-stage trade-off

BM25 → monoBERT → duoBERT allows separate candidate depths and model choices. More tuning knobs create a wider effectiveness-efficiency trade-off space, but also more system complexity.

Four LLM reranking families

Family	Input	Signal	Main trade-off
Pointwise	query + one passage	Yes/No logit, label, or grade	simple; absolute judgement may be unstable
Pairwise	query + passages A and B	A > B preference	robust; many comparisons
Listwise	query + passage list	generated permutation	list context; generation/context cost
Setwise	query + small passage set	best-item logits	fewer calls than pairwise; avoids full-list generation

“Zero-shot” here means no new contrastive retriever training after obtaining an instruction-tuned LLM; it does not mean the underlying LLM was never trained.

Prompt variation

Role text, formatting, output restrictions, and query/passage order affect results. The slides report greater sensitivity for pointwise and listwise prompts, while pairwise and setwise are generally more robust. Model size, architecture, and training still matter.

Why setwise can cost less

One call compares several passages using logits. This needs fewer comparisons than pairwise ranking and less generation than listwise permutation output.

Week 8 practical data flow

BM25 top-k

Retrieve 10 or 20 candidates and read raw passage text.

→

monoBERT

Tokenise each pair, obtain relevant-class probability, and sort descending.

→

Evaluate

Write the reordered run and compare it with qrels.

The second exercise uses TinyLlama for pointwise Yes/No scores, then asks for a pairwise LLM prompt comparing two passages. Retrieval depth and evaluation cutoff are separate parameters.

LLM4IR versus IR4LLM

Large Language Models for Information Retrieval (LLM4IR) applies LLMs to tasks such as reranking. Information Retrieval for Large Language Models (IR4LLM) retrieves external evidence for an LLM. The Week 8 ranking exercise is LLM4IR.

Why automatic prompt engineering is harder for ranking

A ranking demonstration includes a query, several passage meanings, relative relevance, and an ordering. Multiple passages may share a valid relevance level, so the desired output is more complex and less unique than one classification label.

Week 9

Information Retrieval in the Age of LLMs

LLM4IR uses Large Language Models for retrieval and ranking; IR4LLM supplies external evidence to an LLM. This week connects retrieval models to RAG, long context and attribution.

Week 9 complete

The end-to-end evidence pipeline

Retrieve

BM25, DPR, TILDEv2, SPLADE, CiC, or a specialist model selects evidence.

→

Refine

Rewrite, rerank, filter, fuse, summarise, or compress.

→

Generate and attribute

Answer from evidence and link claims to real sources.

Retrieval effectiveness and answer quality are related but different. Better nDCG can make good evidence available, yet prompt order, truncation, conflicting model knowledge, and evidence use still determine the answer.

Prompting recap

Zero-shot gives instructions only; one/few-shot adds demonstrations; Chain-of-Thought (CoT) elicits intermediate reasoning tokens. Those tokens are generated text, not a guaranteed faithful trace of internal reasoning.

LLMs for retrieval and ranking

RepLLaMA and LLM2Vec generate dense representations. PromptReps and DiffRetriever explore multiple generated representations. Pointwise, pairwise, listwise, and setwise prompts produce ranking judgements.

Long-context search versus RAG

Dimension	LC-LLM / Corpus-in-Context	RAG
Input	very large corpus inside the prompt	small retrieved top-k
Benefit	consolidates a multi-stage pipeline	scales to large and changing collections
Cost	long-context attention and latency	retriever/reranker maintenance
Failure	position bias; lost-in-the-middle	missing, distracting, or conflicting evidence

Corpus-in-Context (CiC, “seek”) uses consistent document IDs, corpus-grounded examples and a fixed list output. Prefix caching reduces repeated corpus-prefix work, but does not remove position bias.

Specialised interfaces

Screenshot retrievers use a Vision-Language Model to preserve layout and visual evidence. DiffRetriever uses a diffusion language model to produce representative tokens in parallel and matches multiple representations.

Why RAG?

External evidence can be newer, private, domain-specific, updateable and inspectable. Unlike further pretraining, the collection remains separate from the generator's parameters.

Modular RAG

Stage	Examples	Main risk
Pre-retrieval	routing, rewriting, expansion, HyDE	query drift
Retrieval	lexical, dense, learned sparse, hybrid	gold evidence absent from top-k
Post-retrieval	rerank, filter, fuse, summarise	latency or useful evidence removed
Prompt	lexical/embedding compression	lost qualifiers or provenance
Generation	read, cite, revise, retrieve again	model prior overrides evidence

Rewrite-Retrieve-Read

A rewriter makes an underspecified query search-friendly, a retriever selects evidence, and a reader answers. The rewriter can be warmed up with pseudo-data and later optimised from downstream reward.

BlendFilter

Combine retrieval with model-generated query/answer signals, then use an LLM to filter irrelevant knowledge before generation. Retrieved results still require evidence selection.

Prompt compression

Lexical methods delete low-information tokens; embedding methods map long text to compact memory tokens. Both save tokens but may lose evidence detail and provenance.

Evidence conflict

A model may ignore corrective evidence or change a correct answer after contrary evidence. Relevance, answer correctness and groundedness must be evaluated separately.

Noise, position, and evidence amount

High-scoring distractors can sharply hurt answers; more distractors often worsen results; gold evidence in the middle can be underused; random context sometimes helps particular models. These are empirical model-dependent behaviours, not universal recipes.

Project 2 link: keep the same LLM, prompt, decoding and token budget across no-retrieval, BM25, neural and oracle evidence conditions.

Attribution

Attribution links answer claims to supporting references. Directly generated citations may be fabricated; retrieval-based attribution restricts candidates to real documents.

RARR

Retrofit Attribution using Research and Revision researches support for an existing answer, then revises unsupported content while preserving as much of the draft as possible.

Why does better retrieval not guarantee a better answer?

nDCG measures ranked evidence. Generation additionally depends on truncation, ordering, prompt design, model priors, and whether the generator follows the passages. Measure the relationship per query rather than assuming causality.

Formula sheet

Formula reference

Identify the variables, then ask what each factor rewards or penalises.

Formulas complete

Name	Formula	Interpretation
P@k	`relevant in top k / k`	precision within cutoff k
Recall	`retrieved relevant / all relevant`	coverage of the relevant set
F1	`2PR/(P+R)`	harmonic mean of P and R
AP	`(1/\|Rel\|) Σ P@k × rel(k)`	record precision whenever a relevant result appears
MAP	`mean(AP over queries)`	each query has equal weight
RR	`1/rank(first relevant)`	only the first relevant result matters
nDCG@K	`DCG@K / IDCG@K`	graded relevance plus position discount
RBP	`(1-p)Σ r_i p^(i-1)`	p models continuation probability

Name	Formula	Intuition
IDF	`log(N/(1+df))`	rarer collection terms discriminate better
TF-IDF	`Σ tf × idf`	within-document importance × collection rarity
Cosine	`q·d/(\|\|q\|\|\|\|d\|\|)`	compare direction while reducing magnitude effects
BIM weight	`log((N-df)/df)`	probabilistic term advantage
BM25	`Σ IDF × saturated_tf(length-normalised)`	rarity, saturation, and document length

Name	Formula	Intuition
Softmax	`exp(z_i)/Σexp(z_j)`	convert scores to weights summing to one
Attention	`softmax(QKᵀ/√d_k)V`	relevance-weighted value aggregation
InfoNCE	`-log exp(s+)/[exp(s+)+Σexp(s-)]`	classify the positive among candidates
Margin	`max(0, m-(s+-s-))`	positive must lead by at least m
MaxSim	`Σ_i max_j E_qi·E_dj`	best document-token match for each query token
Autoregressive	`Π_t P(y_t\|y_<t)`	predict the next token from the prefix
monoBERT	`P(relevant=1\|q,d)`	pointwise score from joint encoding
Binary cross-entropy	`-Σ_pos log(s)-Σ_neg log(1-s)`	raise positives and suppress negatives
duoBERT	`s(d_i)=Σ_{j≠i}p(d_i>d_j\|q)`	aggregate pairwise wins

Name	Formula	Interpretation
Naive clicks	`Σ λ(rank_i)c_i`	retains exposure bias
IPS	`Σ [λ(rank_i)/P(o_i=1)]c_i`	corrects examination bias in expectation
Off-policy weight	`P(action\|new)/P(action\|logging)`	reweights logged actions for a candidate policy
Counterfactual update	`Risk_IPS(f) < Risk_IPS(f₀)`	prefer lower estimated counterfactual risk

Model matrix

Representation, scoring, loss, and negatives

For model questions, classify each claim by these components instead of memorising one sentence.

Model matrix complete

Model	Representation / pooling	Similarity / interaction	Training	Role
BM25	sparse term statistics	weighted sum over terms	usually untrained; tune k1,b	strong lexical baseline
TILDEv2	query token IDs; contextual document-token weights	exact match + per-term max weight	document-side neural learning and expansion	fast learned sparse reranker
SPLADE(v2)	mostly-zero vocabulary vectors	sparse dot product	MLM head, distillation, sparsity regularisation	learned sparse retrieval
PromptReps	LLM dense state + sparse logits	normalised hybrid score	zero-shot	hybrid representation
DPR	CLS bi-encoder	dot product	InfoNCE/NLL; in-batch and hard negatives	first-stage dense retrieval
RepBERT	mean token embeddings	inner product	MultiLabelMarginLoss	BERT bi-encoder
ANCE	CLS / first token	dot product	NLL/InfoNCE-style; ANN self-mined negatives	dynamic hard-negative mining
Contriever	mean pooling	cosine	unsupervised contrastive; same-document spans	unsupervised dense retriever
ColBERT	all token embeddings	late-interaction MaxSim	contrastive / pairwise CE	efficient fine-grained matching
E5-Mistral	Mistral decoder; EOS/last token	temperature-scaled cosine	InfoNCE; synthetic and supervised data	LLM embedding model
LLM2Vec	decoder with bidirectional attention; pooling	often cosine	MNTP + unsupervised SimCSE + optional supervision	decoder converted to encoder
RepLLaMA	Llama2; EOS last hidden state	dot product	InfoNCE; supervision + LoRA	LLM retriever
Cross-encoder	joint query-document encoding	classification/regression relevance head	pointwise, pairwise, or listwise	high-accuracy reranker
monoBERT	`[CLS] q [SEP] d [SEP]`	relevant-class softmax score	binary cross-entropy; BM25 negatives	pointwise BERT reranker
BERT-MaxP / FirstP / SumP	document passages	max / first / sum passage score	transferred document labels	long-document score aggregation
PARADE	passage vectors	average / max / attention / Transformer aggregation	document-level ranking	representation aggregation
duoBERT	query + two documents	pairwise preference aggregation	pairwise cross-entropy	late-stage BERT reranker
LLM pointwise / pairwise	one document / document pair in a prompt	label logits / preference	zero-shot prompting	individual judgement / comparison
LLM listwise / setwise	candidate list / small set	permutation / best-item logits	zero-shot prompting	list context / efficient selection
Corpus-in-Context	whole corpus in an LC-LLM prompt	generated document IDs	few-shot prompting; no retriever training	small-corpus model-based retrieval
Screenshot retriever	page pixels and layout	VLM similarity	vision-language contrastive learning	layout-rich documents
DiffRetriever	parallel representative tokens	multi-representation matching	diffusion language model	generative multi-vector retrieval
Basic RAG	query + top-k evidence	autoregressive answer	retriever and generator may be frozen	evidence-based IR4LLM
Rewrite-Retrieve-Read	rewritten query + evidence	downstream answer reward	pseudo-data and optional policy optimisation	underspecified questions
BlendFilter	retrieved and model-generated knowledge	filter then generate	prompted LLM modules	remove distractors
RARR	draft + researched evidence	support check and revision	post-hoc research/revision	retrofit attribution

Example multiple-choice logic: the ColBERT and DPR pairings are correct, as is RepBERT with mean pooling, inner product, and Multi-Label Margin Loss. ANCE normally uses dot product, not cosine. E5-Mistral's EOS pooling and InfoNCE are plausible, but “mean-pooling similarity” confuses pooling with the similarity function.

Acronym and definition glossary

Full names and key definitions

Short name → full English name → definition. Model names that are not acronyms are identified separately.

Glossary complete

Short name	Full name	Definition
IR	Information Retrieval	Finding and ranking documents that satisfy an information need.
TF / DF / CF	Term Frequency / Document Frequency / Collection Frequency	Counts within one document, across documents containing the term, and across all term occurrences.
IDF	Inverse Document Frequency	A term's discriminative weight based on how few documents contain it.
TF-IDF	Term Frequency-Inverse Document Frequency	Combines local frequency with collection rarity.
VSM / BIM	Vector Space Model / Binary Independence Model	Vector similarity and probabilistic term-independence approaches to ranking.
BM25	Best Matching 25	A lexical model combining IDF, tf saturation, and length normalisation.
RSJ / F1	Robertson-Spärck Jones / F-score with β = 1	A probabilistic IDF weighting foundation for BM25, and the harmonic mean of precision and recall.
AP / MAP	Average Precision / Mean Average Precision	Average precision at relevant ranks for one query, then averaged across queries.
RR / MRR	Reciprocal Rank / Mean Reciprocal Rank	Measures where the first relevant result appears.
DCG / IDCG / nDCG	Discounted Cumulative Gain / Ideal DCG / Normalized DCG	Combines graded relevance and rank discount, then normalises by the ideal order.
RBP	Rank-Biased Precision	Models a user's probability of continuing down the ranking.
BERT	Bidirectional Encoder Representations from Transformers	An encoder model that can jointly contextualise query and document tokens.
CLS / SEP / EOS	Classification / Separator / End of Sequence token	Special tokens for classification summaries, segment boundaries, and sequence endings.
MLM / FFN	Masked Language Modeling / Feed-Forward Network	A masked-token pretraining objective and the per-token nonlinear sublayer in a Transformer block.
GPT / LLM	Generative Pre-trained Transformer / Large Language Model	Decoder-style generation family and the broader class of large pretrained language models.
DPR	Dense Passage Retrieval	A bi-encoder that retrieves by dense query-passage vector similarity.
ColBERT	Contextualized Late Interaction over BERT	Retains token vectors and applies MaxSim for fine-grained late interaction.
ANN / ANCE	Approximate Nearest Neighbour / Approximate Nearest Neighbor Negative Contrastive Estimation	Fast approximate vector search and a retriever that mines its ANN mistakes as hard negatives.
NLL / InfoNCE	Negative Log-Likelihood / Information Noise-Contrastive Estimation	Objectives that penalise low target probability and contrast positives against negatives.
CE	Cross-Entropy	A classification loss comparing target and predicted distributions.
TILDE	Term Independent Likelihood moDEl	Learns document-side vocabulary term likelihoods for efficient matching.
SPLADE	Sparse Lexical and Expansion Model	Learns sparse vocabulary weights and semantic expansion terms.
ReLU / FLOPS	Rectified Linear Unit / Floating-Point Operations	Produces non-negative activations, while the FLOPS-inspired regulariser discourages overly dense SPLADE vectors.
T5	Text-to-Text Transfer Transformer	A sequence-to-sequence model that can generate offline doc2query expansions.
LTR / OLTR	Learning to Rank / Online Learning to Rank	Learns an ordering from labelled data or continuously from user interactions.
IPS	Inverse Propensity Scoring	Reweights observations by inverse exposure probability to correct bias in expectation.
DBGD	Dueling Bandit Gradient Descent	Uses interleaving outcomes to move toward a winning exploratory ranker.
COLTR	Counterfactual Online Learning to Rank	Evaluates candidate rankers off-policy using logged clicks and propensities.
RM3 / TREC	Relevance Model 3 / Text REtrieval Conference	A pseudo-relevance feedback model and a major standard IR evaluation programme.
Qrels	Query Relevance Judgements	Ground-truth query-document relevance labels used to evaluate a run.
MS MARCO / BEIR	Microsoft Machine Reading Comprehension / Benchmarking Information Retrieval	Widely used neural-ranking training data and a cross-domain retrieval benchmark suite.
JSONL	JSON Lines	Stores one independent JSON object per line for streaming datasets.
LoRA / MNTP / SimCSE	Low-Rank Adaptation / Masked Next Token Prediction / Simple Contrastive Learning of Sentence Embeddings	Efficient fine-tuning and representation-learning methods used by later embedding models.
PARADE	Passage Representation Aggregation for Document Reranking	Aggregates encoded passage vectors for long-document ranking.
LLM4IR / IR4LLM	Large Language Models for Information Retrieval / Information Retrieval for Large Language Models	Using LLMs to improve IR versus retrieving evidence to support an LLM.
monoBERT / duoBERT	Model names, not acronyms	Pointwise relevance scoring versus pairwise document preference.
RAG / LC-LLM	Retrieval-Augmented Generation / Long-Context Large Language Model	Retrieve a small evidence set for generation versus place much more source text directly in context.
CiC / CoT	Corpus-in-Context / Chain-of-Thought	Direct corpus prompting for retrieval and generated intermediate reasoning steps.
VLM / HyDE	Vision-Language Model / Hypothetical Document Embeddings	Models pixels and language; or retrieves by embedding a generated hypothetical answer document.
RARR	Retrofit Attribution using Research and Revision	Researches evidence for a draft and revises unsupported claims.
PPO	Proximal Policy Optimization	A reinforcement-learning method that can optimise a trainable query rewriter while limiting update size.

Assignment 1

End-to-end A1 workflow

A1 is not just a BM25 function. It is a reproducible IR experiment with validation, tuning, comparison, and analysis.

A1 review complete

Open the experiment study workflow. No personal submission or experiment outputs are included in this public guide.

Why separate train and test?

Train qrels select k1 and b. Test qrels are used once for unbiased final evaluation. Repeatedly changing parameters after inspecting test results leaks test information into training.

How to discuss tuned BM25

Do not write only “the score improved.” Explain how k1 changes tf saturation and b changes length normalisation, then use gain/loss queries to diagnose the effect.

# Clear scorer skeleton; adapt names to the supplied notebook
def bm25(tf_vector, df_vector, doc_len, k1=1.2, b=0.75):
    score = 0.0
    for tf, df in zip(tf_vector, df_vector):
        idf = math.log(N / (1 + df))
        norm = 1 - b + b * (doc_len / average_doc_length)
        tf_weight = ((k1 + 1) * tf) / (tf + k1 * norm)
        score += idf * tf_weight
    return score

Recall practice

Self-test

Answer first, then expand. Being able to explain why is the real test.

Self-test complete

1. What are tf, df, and cf?

tf is the term count in the current document; df is the number of documents containing the term; cf is the term's total count across the collection.

2. Why does BM25 saturate tf?

The jump from zero to one occurrence is more informative than 100 to 101. Saturation prevents repetition from increasing the score without limit.

3. Why divide DCG by IDCG?

Queries differ in the number and grades of relevant documents. Normalisation provides a comparable scale, usually between zero and one.

4. What does p-value < .05 mean?

If H₀ were true, the probability of the observed or a more extreme difference is below 5%. It is not the probability that system B is better.

5. Why scale attention by √d_k?

Dot-product variance grows with dimension. Scaling prevents an excessively sharp softmax and helps stable gradients.

6. Encoder versus decoder?

An encoder typically sees both left and right context for understanding. A decoder uses a causal mask and sees only the left prefix for generation.

7. Why can dense retrieval match synonyms?

Training maps semantically related expressions to nearby vectors rather than requiring identical surface terms.

8. How does ColBERT calculate its score?

For every query token, find the highest similarity to any document token, then sum those maxima.

9. Why are hard negatives useful and risky?

They look relevant and provide strong gradients, but incomplete qrels can make truly relevant documents appear as false negatives.

10. Why not tune BM25 on the test set?

It adapts parameters to test judgements and makes the estimate of generalisation optimistically biased.

11. Why is SPLADE sparse although it uses BERT?

It outputs a vocabulary-sized vector with mostly zero dimensions. BERT learns which dimensions should be active.

12. TILDEv2 versus SPLADE?

TILDEv2 uses lightweight query token IDs and learned document-token weights; SPLADE encodes both sides into sparse vocabulary vectors.

13. Why put BM25 before TILDEv2?

BM25 cheaply narrows the full collection, then TILDEv2 reranks only plausible candidates.

14. Pointwise, pairwise, and listwise?

They learn from one document, a document preference pair, or an entire ranked list respectively.

15. Why are clicks biased?

Clicks depend on examination, position, selection and presentation as well as relevance.

16. What is the main purpose of IPS?

To correct exposure bias by weighting observations with inverse examination probability.

17. How does balanced interleaving work?

Mix two rankers' unseen results into one list and attribute clicks to infer pairwise preference.

18. Why use counterfactual OLTR?

It reuses one propensity-logged interaction stream to estimate several candidate rankers off-policy.

19. Why must monoBERT follow a retriever?

Joint encoding is too expensive for the whole collection. BM25 first creates a manageable candidate set that monoBERT can reorder.

20. Score aggregation versus representation aggregation?

MaxP/FirstP/SumP combine scalar passage scores; PARADE combines passage vectors before making a document-level decision.

21. monoBERT versus duoBERT?

monoBERT scores one document independently; duoBERT directly compares two documents.

22. Four LLM reranking families?

Pointwise judges one item, pairwise compares two, listwise generates an order, and setwise selects the best from a small set.

23. Why may setwise ranking cost less?

It compares several passages per inference using logits, reducing pairwise calls and avoiding full list generation.

24. Why does prompt wording matter?

Roles, output constraints, formatting, and component order alter the model context; sensitivity also varies by model.

25. LLM4IR versus IR4LLM?

LLM4IR applies LLMs to retrieval/ranking; IR4LLM retrieves external evidence for a generator.

26. Why can long context still miss evidence?

Position bias and lost-in-the-middle mean that fitting text into a context window does not guarantee that the model uses it.

27. Relevance versus groundedness?

Relevance asks whether a passage helps the query; groundedness asks whether answer claims are supported by the supplied evidence.

28. Why can one distractor harm RAG?

Semantic alignment without answer support introduces a competing signal that can mislead the generator.

29. RAG versus fine-tuning?

RAG changes inspectable external evidence at inference; fine-tuning changes model parameters.

30. What does RARR do?

It researches evidence for a draft, adds attribution, and revises unsupported content after generation.

Source map

Scope of this guide

The guide synthesises the supplied INFS7410 folder. Large corpora, indexes, and virtual environments are experimental resources rather than revision content.

Source map complete

Week 1 lecture notes, slides, and practical
Week 2 lecture notes, slides, and practical
Week 3 lecture notes, slides, and practical
Week 4 notes, Transformer worksheets, and solution
Week 5 lecture notes and practical solutions
Week 6 TILDEv2 and SPLADEv2 practical
Week 6 learned sparse notes
Week 7 full slide deck: LTR, IPS, interleaving and COLTR
Week 8 full slide deck and practical: neural/LLM reranking
Week 9 full slide deck: long-context search, modular RAG, evidence conflict, noise and attribution
Project 2 notebook, sampled BRIGHT Biology collection and marking sheet

Coverage: this edition integrates Weeks 1–9 and connects retrieval representations, candidate ranking, click feedback, neural reranking, RAG and evidence attribution.

Revision questions

Review 44 topics through independent explanations and worked examples. Original course PDFs, slide screenshots and full slide transcripts are not distributed here.

Open revision Q/A · Original course materials (login required)

No matching content. Try a shorter term.