NLTK (Natural Language Toolkit)

NLTK, which stands for Natural Language Toolkit, is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

Purpose and Functionality:
NLTK is widely used in computational linguistics, artificial intelligence, and machine learning for various natural language processing (NLP) tasks. Its modular design allows users to combine different functionalities to create complex NLP pipelines. Key functionalities include:

1. Tokenization: Breaking down text into smaller units like words or sentences.
2. Stemming and Lemmatization: Reducing words to their root or base form to standardize them for analysis.
3. Part-of-Speech (POS) Tagging: Identifying the grammatical role of each word (e.g., noun, verb, adjective).
4. Named Entity Recognition (NER): Identifying and classifying named entities in text (e.g., person names, organizations, locations).
5. Parsing: Analyzing the grammatical structure of sentences.
6. Classification: Training models to categorize text into predefined classes (e.g., sentiment analysis).
7. Corpora and Lexical Resources: Access to a vast collection of text and speech, along with lexical databases like WordNet.
8. Frequency Distribution: Analyzing the occurrence frequency of words in a text.

Key Features:
- Extensive Modules: Offers a rich set of modules for almost every NLP task.
- Corpus Access: Provides programmatic access to many linguistic corpora (collections of text).
- Educational Tool: Often used as a teaching and research tool for NLP.
- Open Source: Freely available and actively maintained by a large community.

NLTK is often the first library learned by those entering the field of NLP due to its comprehensive features and relatively low learning curve for basic tasks. While more specialized libraries like SpaCy or Hugging Face Transformers might be preferred for production-level systems or state-of-the-art deep learning models, NLTK remains an invaluable tool for prototyping, academic research, and fundamental text processing.

import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.probability import FreqDist
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

 --- 1. Download necessary NLTK data (run once) ---
 nltk.download('punkt')          For tokenization
 nltk.download('averaged_perceptron_tagger')  For POS tagging
 nltk.download('stopwords')       For stopwords
 nltk.download('wordnet')         For lemmatization
 nltk.download('omw-1.4')         For WordNet open multilingual wordnet

 --- 2. Sample Text ---
text = "NLTK is a powerful library for natural language processing. It provides tools for tokenization, stemming, lemmatization, and more. Learning NLTK makes NLP tasks easier and more accessible."

print("Original Text:\n", text, "\n")

 --- 3. Tokenization ---
 Word Tokenization
words = word_tokenize(text)
print("Word Tokenization:\n", words)

 Sentence Tokenization
sentences = sent_tokenize(text)
print("\nSentence Tokenization:\n", sentences)

 --- 4. Part-of-Speech Tagging ---
 Requires 'averaged_perceptron_tagger' data
pos_tags = nltk.pos_tag(words)
print("\nPOS Tagging (first 10 words):\n", pos_tags[:10])

 --- 5. Frequency Distribution ---
 Remove stopwords for better insights
stop_words = set(stopwords.words('english'))
filtered_words = [word.lower() for word in words if word.isalnum() and word.lower() not in stop_words]
fdist = FreqDist(filtered_words)
print("\nFrequency Distribution (Top 5):\n", fdist.most_common(5))

 --- 6. Stemming ---
 Using Porter Stemmer
ps = PorterStemmer()
stemmed_words = [ps.stem(w) for w in filtered_words]
print("\nStemming (first 10 filtered words):\n", stemmed_words[:10])

 --- 7. Lemmatization ---
 Using WordNet Lemmatizer
lemmatizer = WordNetLemmatizer()
lemmatized_words = [lemmatizer.lemmatize(w) for w in filtered_words]
print("\nLemmatization (first 10 filtered words):\n", lemmatized_words[:10])

 Example of lemmatization with POS context (verb)
print("\nLemmatizing 'running' as a verb:", lemmatizer.lemmatize("running", pos="v"))
print("Lemmatizing 'ran' as a verb:", lemmatizer.lemmatize("ran", pos="v"))
print("Lemmatizing 'better' as an adjective:", lemmatizer.lemmatize("better", pos="a"))