Notebook infographic: stemming, lemmatization, and stopword removal — where preprocessing helps and where it hurts
NLP Foundations & Text Representation

Stemming, Lemmatization, and Stopword Removal: When Text Preprocessing Helps and When It Hurts

Three classic tricks that shrink a vocabulary — and a rule for knowing when to use each one.

Core Concept

Classical text preprocessing shrinks the vocabulary a model must learn. Collapse running, runs, runner into one feature, delete words that appear in every sentence, and the matrix of counts stops being 99% zeros — lower sparsity, faster training. The catch: the benefit is model-dependent. For count-based models it removes a real problem; for modern transformers it removes nothing and can actively damage the input. One question decides the call: does the model treat every surface form as an independent feature, or does it already have a bounded subword vocabulary?

Key Components

01Stemming

Crude suffix-stripping rules (Porter, Snowball): running→run, studies→studi. Fast, rule-based, and the output need not be a real word — that is accepted, because the goal is token collision, not grammar.

02Lemmatization

Dictionary + part-of-speech context returns a genuine base form: was→be, better→good. Slower and linguistically correct — it resolves ambiguity stemming cannot (study the noun vs study the verb).

03Stopword Removal

Deletes high-frequency function words — the, is, of — on the assumption they carry little signal. The lists are language- and domain-specific: a generic English list mangles clinical, legal, and code text.

04The Negation Hazard

not, no, never sit in default stopword lists. Strip them and “the movie was not good” becomes identical to “the movie was good” — you just deleted the exact signal a sentiment classifier exists to read.

How It Works

  1. Why it helps: in bag-of-words, TF-IDF, naive Bayes, and classical topic models, every surface form is an independent, unrelated feature. Collapsing variants and dropping boilerplate cuts distinct token counts, which lowers sparsity and speeds training.
  2. Why it hurts: subword tokenizers (BPE, WordPiece, SentencePiece) already bound the vocabulary at a fixed size and split unseen words into known pieces — the sparsity problem stemming was invented to solve no longer exists. Feeding a pretrained model stemmed text produces out-of-distribution input it never saw during pretraining.
  3. The falsification test: train two sentiment classifiers on the same labelled set — one TF-IDF + logistic regression, one fine-tuned transformer — each with and without preprocessing. Preprocessing should help or barely move the TF-IDF arm and measurably hurt the transformer arm.
  4. The negation probe: hand-build a test set of negated sentences. Once not is stripped, watch accuracy collapse — that is the stopword list eating meaning, measurable in a single run.

Where It Still Earns Its Place

Keyword search & BM25 indexing TF-IDF baselines Topic modelling Tag clouds Low-resource / CPU-only settings
Rule of thumb: if the pipeline counts terms (interpretable term counts, classical models), preprocess freely. If the pipeline understands context (anything with subword pretraining), feed it the raw text — and never touch not.

Checkpoint — Answer Before You Move On

  1. Porter stemming turns studies into studi — not a real word. Why is a non-word output acceptable in a stemming pipeline, and what does that trade-off buy you? Think: what is the pipeline actually optimizing?
  2. A sentiment pipeline applies a default English stopword list. Why do “the movie was not good” and “the movie was good” become identical, and which class of words is being destroyed? Look at what the two sentences differ by.
  3. A team adds lemmatization to the input of a fine-tuned transformer and accuracy drops. Give two distinct reasons why preprocessing is expected to be neutral-or-harmful for transformer pipelines. One reason is about the tokenizer; the other is about what the model saw during pretraining.