"algo-nlp-ner"

Named Entity Recognition

Overview

NER identifies and classifies named entities in text into predefined categories (Person, Organization, Location, Date, Money, etc.). Approaches: rule-based (regex, gazetteers), statistical (CRF), neural (BiLSTM-CRF, transformer-based). Modern NER uses spaCy or Hugging Face models with F1 scores 85-95%.

When to Use

Trigger conditions:

• Extracting structured entities from unstructured text
• Building knowledge graphs from documents
• Preprocessing for information retrieval or question answering

When NOT to use:

• For text classification (categorizing whole documents, not extracting entities)
• For relation extraction between entities (need additional RE model)

Algorithm

IRON LAW: NER Performance Depends on DOMAIN Match
A model trained on news text (OntoNotes) performs poorly on medical
records or legal documents. Domain-specific entities (drug names,
legal citations, product SKUs) require domain-specific training data
or fine-tuning. Always evaluate on YOUR domain's data.

Phase 1: Input Validation

Determine: target entity types (standard: PER, ORG, LOC, DATE, MONEY or custom), input language, domain. Select appropriate pre-trained model or prepare training data. Gate: Entity types defined, model or training data available.

Phase 2: Core Algorithm

Pre-trained model approach:

1. Load model (spaCy, Hugging Face NER pipeline)
2. Process text through the pipeline
3. Extract entity spans with type labels and confidence scores

Fine-tuning approach:

1. Annotate 200+ domain-specific examples in BIO format
2. Fine-tune transformer model (BERT, RoBERTa) on annotated data
3. Evaluate on held-out test set

Phase 3: Verification

Evaluate: precision, recall, F1 per entity type. Check: boundary detection (exact span match) and type classification accuracy. Gate: F1 > 0.80 per entity type on domain-relevant test data.

Phase 4: Output

Return extracted entities with types, positions, and confidence.