Applied NLP

NLP applications: classification, tagging, retrieval, and reranking

Task picker

Predict the output type before choosing a model. Ask whether you need one label, one label per token, a ranked list, or a missing token. Two tasks may accept the same text but require different targets and metrics.

Plain-language start

Keep one message in mind: “My card was charged twice.” Classification returns one label such as billing. Token tagging can mark “card” as a product. Retrieval returns similar help articles. Reranking reorders those candidate articles using the message and each article together.

First question. What is one input, and what exact output must the system produce?

A review may need one sentiment label. A sentence may need one label for every name. A search box may need a ranked document list.

1. Define the output and evaluation unit.
2. Build a rule, majority, or TF-IDF baseline.
3. Add a pretrained encoder only when context or semantic matching helps.
4. Compare both systems on the same untouched evaluation set.
5. Inspect errors before adding more complexity.

Task 1: sequence classification

Definition. Sequence classification assigns one label to a whole input. Examples include sentiment and support intent.

Shapes. IDs (B,T) become hidden states (B,T,D), pooled vectors (B,D), then class logits (B,C).

• Baseline: majority class, then bag of words or TF-IDF.
• Metrics: accuracy for balanced equal-cost errors; otherwise add per-class precision, recall, and F1.
• Error check: read mistakes from every class, especially rare important classes.

Boundary example. “The camera is excellent, but delivery was late” mixes aspects. Decide whether the target is overall sentiment, delivery sentiment, or aspect extraction before changing models.

Task 2: token classification and named entities

Definition. Token classification predicts a label at each token. Named-entity recognition can mark people, organizations, and places.

Example. Rita joined Acme in Pune can use labels B-PER O B-ORG O B-LOC. B begins a span, I continues it, and O is outside.

• Shapes: hidden states (B,T,D); token logits (B,T,C).
• Loss: exclude padding positions.
• Subwords: write one label-alignment rule and use it in training and evaluation.
• Metric: use entity-level F1, not only token accuracy.

Why entity F1? Finding New but missing York should not receive full credit for New York.

Task 3: semantic retrieval

Goal. Semantic retrieval matches meaning when words differ. Example: query reset my login secret and document change your password.

Method. A bi-encoder encodes queries and documents independently. Document vectors are stored once; a query vector is compared with them by cosine or dot product.

offline: document -> encoder -> vector -> index
online:  query    -> encoder -> vector -> nearest candidates

Checkpoint rule. A raw masked-language checkpoint is not automatically a good retrieval model. Follow an embedding checkpoint’s required prefixes, pooling, and normalization.

• Recall@k: did a relevant item appear in the top k?
• MRR: how early did the first relevant item appear?
• nDCG: how good is an ordering with graded or multiple relevant items?

Task 4: reranking

Definition. A cross-encoder reads a query and candidate together, then returns one relevance score.

Trade-off. Joint attention can improve fine distinctions, but scoring millions of documents this way is too expensive. First retrieve 50 or 100 candidates, then rerank them.

query -> TF-IDF or bi-encoder -> top 100 candidates
      -> cross-encoder pair scores -> final top 10

1. Measure candidate recall before reranking.
2. Measure final ranking quality after reranking.
3. Measure latency for both stages.
4. Reject a costly gain when it does not meet the product budget.

Hard limit. If the relevant document is absent from the candidate set, the reranker cannot recover it.

Task 5: masked-language modeling

Definition. Masked-language modeling predicts selected hidden tokens using context on both sides.

Not classification. MLM produces vocabulary logits per token. Classification produces task-label logits for a sequence or token.

1. Split whole documents before creating random masks.
2. Continue pretraining on carefully selected domain text.
3. Keep an unchanged downstream baseline.
4. Check whether downstream sentiment, tagging, or retrieval improves.
5. Inspect bias and memorization risks for sensitive text.

Metrics by hand: precision, recall, and F1

• TP: a correct prediction of the chosen class.
• FP: an incorrect prediction of that class.
• FN: a missed example of that class.

precision = TP / (TP + FP)
recall = TP / (TP + FN)
F1 = 2 * precision * recall / (precision + recall)

• Macro F1: average class F1 scores; every class has equal weight.
• Micro F1: combine all decisions first; frequent classes have more influence.
• Weighted F1: average class F1 scores weighted by class size.

Reporting rule. State exactly which F1 average you use.

# Runnable binary example: TP=2, FP=1, FN=1
tp, fp, fn = 2, 1, 1
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print(round(precision, 3), round(recall, 3), round(f1, 3))

# Expected: 0.667 0.667 0.667

This snippet was tested with standard Python. In multiclass work, repeat the one-versus-rest calculation for each class before taking a macro average.

Leakage patterns that look like progress

• Duplicate leakage: near-copies cross splits. Group duplicates before splitting.
• Author leakage: one person appears in train and test. Split by user when future users are the target.
• Time leakage: training sees later messages. Train on the past and test on the future.
• Feature leakage: vocabulary, IDF, tokenizer training, thresholds, or normalization see all data. Fit them on training only.
• Label leakage: an input field or template directly reveals the answer. Inspect examples and the data-producing system.

Use validation data to select a model and threshold. Use the final test once after those decisions. Repeatedly checking test results turns the test set into another validation set.

A university-style experiment report

1. Define input, output, and unit of prediction.
2. Describe label collection and uncertainty.
3. Show train, validation, and test counts per class.
4. Explain split groups, duplicate handling, and dates.
5. Compare majority, classical, and proposed models on the same split.
6. Report metrics, seeds, latency, memory, and representative errors.
7. State limits and use confidence intervals when possible.

Incomplete claim: “BERT achieved 92%.” Add the dataset, split, metric, checkpoint, seeds, and comparison.

Scope limit. Product-review sentiment does not prove quality on medical notes or another language.

Capstone 1: support-message classifier

Goal. Classify messages into billing, account, delivery, and other.

1. Build majority and TF-IDF nearest-centroid baselines.
2. Fine-tune a small pretrained encoder.
3. Split by customer and, if available, time.
4. Report macro F1, per-class recall, confusion matrix, and CPU latency.

Acceptance check. Compare against TF-IDF on validation and untouched test; provide OOV and empty-message fallbacks; categorize ten errors. A clear report remains useful even when the encoder does not win.

Capstone 2: entity extraction

Goal. Extract product, organization, and location spans from short messages.

1. Write annotation rules before labelling.
2. Keep each whole document in one split.
3. Align word labels to subwords and ignore padding in loss.
4. Compare with a dictionary matcher.

Acceptance check. Report entity-level precision, recall, and F1 by type. Include boundary errors, unknown names, and one disagreement that exposes an unclear annotation rule.

Capstone 3: search and reranking

Goal. Retrieve answers for natural-language questions from a small course-note collection.

1. Build TF-IDF retrieval.
2. Add a bi-encoder.
3. Optionally rerank candidates with a cross-encoder.
4. Create relevance judgements, including multiple relevant documents where suitable.

Acceptance check. Report Recall@10, final MRR or nDCG, and separate stage latency. Include keyword, lexical-mismatch, and no-answer queries; allow “no supported result”.

Solved practice

Recap

• Sentiment and intent use sequence classification.
• Named entities use token or span labels.
• Bi-encoders retrieve at scale; cross-encoders rerank a short list.
• Masked-language modeling is pretraining, not a universal task head.
• Fit preprocessing on training data, compare a baseline, and preserve the final test.

Further reading: the BERT paper for encoder fine-tuning tasks, the Sentence-BERT paper for bi-encoder sentence retrieval, the Sentence Transformers retrieve-and-rerank documentation, and the Stanford ranked retrieval evaluation chapter.