nlp-text-analysis

NLP Text Analysis

Overview

This skill covers the full workflow for analyzing and preprocessing text datasets in NLP research: profiling raw data, making principled preprocessing decisions, selecting tokenizers and embeddings, and evaluating model outputs with standard metrics. Apply this skill whenever working with text corpora, NLP benchmarks, or language model outputs.

When to Use This Skill

Use this skill when:

• Exploring a new text dataset before modeling (corpus statistics, quality checks)
• Deciding how to preprocess text (tokenization, normalization, cleaning)
• Choosing an embedding strategy for a downstream task
• Evaluating NLP model outputs (translation, summarization, QA, classification)
• Selecting benchmark datasets for a given NLP task
• Augmenting a small text dataset to improve generalization

---

Text Dataset Characterization

Before any modeling, profile your corpus systematically. Use scripts/text_profiler.py for automated profiling, or run the analyses below interactively.

Core Statistics

import pandas as pd
import collections
import re

# Load dataset (CSV, TSV, or JSONL)
df = pd.read_csv("dataset.csv")  # or pd.read_json("data.jsonl", lines=True)
texts = df["text"].dropna().tolist()

# Basic whitespace tokenization for profiling
def simple_tokenize(text):
    return text.lower().split()

tokens_per_doc = [simple_tokenize(t) for t in texts]
token_lengths = [len(t) for t in tokens_per_doc]
char_lengths = [len(t) for t in texts]

# Token count distribution
import numpy as np
print(f"Total documents:    {len(texts)}")
print(f"Total tokens:       {sum(token_lengths):,}")
print(f"Avg tokens/doc:     {np.mean(token_lengths):.1f}")
print(f"Median tokens/doc:  {np.median(token_lengths):.1f}")
print(f"Max tokens/doc:     {max(token_lengths)}")
print(f"Min tokens/doc:     {min(token_lengths)}")
print(f"Std tokens/doc:     {np.std(token_lengths):.1f}")

# Vocabulary size
all_tokens = [tok for doc in tokens_per_doc for tok in doc]
vocab = collections.Counter(all_tokens)
print(f"\nVocabulary size (whitespace, lowercased): {len(vocab):,}")
print(f"Singleton tokens (freq=1):               {sum(1 for v in vocab.values() if v == 1):,}")

# Top 20 most frequent tokens
print("\nTop 20 tokens:")
for tok, freq in vocab.most_common(20):
    print(f"  {tok:<20} {freq:>8,}")