data-quality-standards

Data Quality Standards for Somali Dialect Classifier

Quality Dimensions

1. Completeness

• All required fields present (text, label, source, timestamp)
• No null or empty text fields
• Labels properly assigned (Northern/Southern/Central)

2. Accuracy

• Text is in Somali (not English, Arabic, or other languages)
• Labels match actual dialect (validated by native speakers)
• Geographic metadata aligns with dialect labels

3. Consistency

• Uniform text encoding (UTF-8)
• Consistent label format (standardized names)
• Timestamp format standardized (ISO 8601)

4. Uniqueness

• No exact duplicates
• Near-duplicate detection (>95% similarity flagged)
• Source URL deduplication

5. Validity

• Text length within acceptable range (10-5000 characters)
• No corrupted/garbled text
• No HTML tags or formatting artifacts

---

Quality Metrics

Critical Metrics

Language Purity:

• Target: >98% Somali text
• Method: Language detection (langdetect, fastText)
• Action: Remove non-Somali text

Duplicate Rate:

• Target: <2% duplicates
• Method: Exact match + fuzzy matching (Levenshtein distance)
• Action: Keep first occurrence, remove duplicates

Label Confidence:

• Target: >90% inter-annotator agreement
• Method: Multiple annotators for sample
• Action: Re-label low-confidence examples

Text Quality Score:

• Target: Average score >7/10
• Components: Length, vocabulary richness, grammar
• Action: Filter texts with score <5

---

Validation Pipeline

Stage 1: Basic Validation

def basic_validation(record):
    checks = {
        'has_text': bool(record.get('text', '').strip()),
        'has_label': record.get('label') in ['Northern', 'Southern', 'Central'],
        'valid_length': 10 <= len(record.get('text', '')) <= 5000,
        'valid_encoding': is_valid_utf8(record['text'])
    }
    return all(checks.values()), checks

Stage 2: Language Detection

from langdetect import detect

def validate_language(text):
    try:
        lang = detect(text)
        return lang == 'so'  # Somali ISO code
    except:
        return False