Remove Stopwords
Data Processing · NLP
Easy
Problem
Remove every token that appears in stopwords. Comparisons are case-sensitive, and retained tokens must remain in their original order. Return a new list without modifying either input.
Theory
Stopwords are common words that appear frequently in text but carry little semantic meaning on their own. Examples include "the", "is", "at", "which", "on", "a", "an", "and", "or", "but". Removing stopwords reduces noise in text data, allowing models to focus on content-bearing words.
Why Remove Stopwords?
Dimensionality reduction: In bag-of-words representations, stopwords create many high-frequency features that add little value.
Improved signal-to-noise ratio: Content words (nouns, verbs, adjectives) carry more meaning than function words.
Computational efficiency: Fewer tokens to process means faster training and inference.
Better similarity measures: Without stopwords, document similarity focuses on topical content rather than grammatical structure.
Common English Stopwords
Articles: a, an, the
Pronouns: I, you, he, she, it, we, they, me, him, her, us, them
Prepositions: in, on, at, by, for, with, about, against, between, into, through, during, before, after
Conjunctions: and, but, or, nor, so, yet, both, either, neither
Auxiliary verbs: is, am, are, was, were, be, been, being, have, has, had, do, does, did
Other common words: this, that, these, those, what, which, who, whom, whose, where, when, how, why
Standard Stopword Lists
NLTK English stopwords: ~179 words, widely used baseline
spaCy stopwords: ~326 words, more comprehensive
Scikit-learn stopwords: ~318 words, optimized for ML applications
Custom lists: Domain-specific additions or removals
Considerations:
- Lists vary between libraries
- Some tasks need custom modifications
- Language-specific lists required for non-English text
The Removal Process
Input: List of tokens (words) from a document Output: Filtered list with stopwords removed
Steps:
- Obtain or define the stopword set
- Convert to efficient lookup structure (set for O(1) lookup)
- Iterate through tokens
- Keep tokens not in stopword set
- Return filtered list
Case handling: Typically convert both tokens and stopwords to lowercase before comparison
Worked Example
Original text: "The quick brown fox jumps over the lazy dog"
Tokens: ["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]
Stopwords: {"the", "a", "an", "over", "is", "are", ...}
After lowercase: ["the", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]
After removal: ["quick", "brown", "fox", "jumps", "lazy", "dog"]
Observation: Removed "the" (twice) and "over". The content-bearing words remain.
When NOT to Remove Stopwords
Sentiment analysis: "not good" vs "good" - removing "not" changes meaning entirely
Question answering: "what", "where", "when", "how" are stopwords but critical for understanding queries
Named entity recognition: "The White House" - "The" is part of the entity name
Machine translation: All words contribute to proper translation
Language modeling: Predicting the next word requires seeing all words
Phrase matching: "to be or not to be" loses meaning without stopwords
Language-Specific Considerations
Different languages have different stopwords:
- French: le, la, les, un, une, de, du, des, et, ou
- German: der, die, das, ein, eine, und, oder, aber
- Spanish: el, la, los, las, un, una, y, o, pero
Challenges:
- Morphologically rich languages have more word forms
- Some languages have fewer or different function words
- Transliteration issues for non-Latin scripts
Efficient Lookup
For processing large corpora, lookup efficiency matters:
Set-based lookup: O(1) average time per token
- Convert stopword list to set once
- Check membership with
token in stopword_set
List-based lookup: O(n) time per token where n is list length
- Much slower for large stopword lists
- Should be avoided
Case normalization: Apply once before all comparisons
Preprocessing Pipeline Position
Typical text preprocessing order:
- Tokenization: Split text into words
- Lowercase: Normalize case
- Stopword removal: Remove common words
- Stemming/Lemmatization: Reduce words to base form
- Vectorization: Convert to numerical representation
Order matters: Stopword removal usually comes after lowercasing but before stemming.
Impact on Different Models
TF-IDF: Stopwords get low IDF scores anyway (appear in many documents), so removal has moderate impact
Word embeddings: Pre-trained embeddings include stopwords; removal depends on downstream task
Topic modeling (LDA): Stopwords should be removed to prevent topics dominated by function words
Neural models: Often keep stopwords since transformers can learn to ignore them
Custom Stopword Modifications
Domain-specific additions:
- Medical: "patient", "study", "results" might be too common in medical literature
- Legal: "whereas", "hereby", "pursuant"
- Technical: "system", "method", "data"
Strategic removals:
- Keep negation words for sentiment: remove "not" from stopword list
- Keep question words for QA systems
Quality Considerations
Too aggressive removal: May lose important context
Too conservative removal: May retain noise
Evaluation approach: Compare model performance with and without stopword removal on a validation set
Where Stopword Removal Shows Up
Search Engines: Query processing ignores stopwords in simple keyword search
Document Classification: Text categorization benefits from focusing on content words
Topic Modeling: LDA and related algorithms need stopword-free input
Keyword Extraction: Identifying important terms excludes common words
Text Summarization: Extractive methods focus on content-bearing sentences
Information Retrieval: Indexing often excludes stopwords
Plagiarism Detection: Comparing content words rather than grammatical structure
Social Media Analysis: Processing tweets and posts with limited characters
Examples
Example 1
- Input
tokens = ["this", "is", "a", "test"], stopwords = ["is", "a"]- Output
["this", "test"]- Explanation
- The two matching stopwords are removed while the remaining order is preserved.
Example 2
- Input
tokens = ["hello", "world"], stopwords = ["the", "and"]- Output
["hello", "world"]
Example 3
- Input
tokens = ["a", "an", "the"], stopwords = ["a", "an", "the"]- Output
[]
Hints
- Build blocked = set(stopwords) before scanning the tokens.
- Filter with [token for token in tokens if token not in blocked].
Requirements
- Convert stopwords to a set for membership checks
- Preserve the order and duplicates of retained tokens
- Treat uppercase and lowercase tokens as different
- Return a new list
Constraints
- tokens and stopwords are lists of strings
- Use Python only
Starter Code
def remove_stopwords(tokens: list, stopwords: list) -> list:
"""
Returns a list of tokens.
"""
# Write code here
pass