02.31 · UNIT 08 · Train, evaluate, and serve your transformer · Lab
NLP applications: classification, tagging, retrieval, and reranking
An NLP task starts by defining the exact output and how success will be measured.
PLAIN-LANGUAGE INTRODUCTION
What is this?
An NLP task starts by defining the exact output and how success will be measured.
One simple example
For query “card charged twice”, a retriever gives duplicate-charge help score 0.82 and reset-PIN score 0.15.
What goes in?
A query plus a fixed collection of candidate documents.
What comes out?
A ranked document list with one relevance score per candidate.
Why does it matter?
A clear output selects the right labels, baseline, and metric.
What is it not?
A retrieval score is not automatically a calibrated probability.
WORK THROUGH THE IDEA
See the idea in more detail
- This example uses retrieval. The required output is a ranked list of help articles.
- The illustrative scores are
0.82for duplicate charges and0.15for resetting a PIN. - Sort larger scores first. The duplicate-charge article ranks above the reset-PIN article.
- Compare with a count-based baseline on the same queries and candidate collection.
- Common mistake: accuracy for one label per document does not measure ranking quality well.