data-exploration

Data Exploration Skill

Overview

This skill provides comprehensive capabilities for Exploratory Data Analysis (EDA) on datasets from CKAN portals and direct CSV files. It focuses on understanding data structure, assessing quality, identifying patterns, and generating insights.

Capabilities

1. Dataset Discovery & Metadata Analysis

• CKAN Dataset Exploration: Search and retrieve datasets from CKAN portals
• Organization Analysis: Examine publishers and their datasets
• Metadata Validation: Assess completeness and quality of metadata
• Resource Evaluation: Analyze available data resources and formats

2. Structural Analysis

• Schema Discovery: Identify columns, data types, and relationships
• Data Profiling: Generate statistical summaries and distributions
• Format Assessment: Evaluate data formatting and standards compliance
• Field Analysis: Examine individual columns and their characteristics

3. Quality Assessment

• Completeness Checking: Identify missing values and null patterns
• Consistency Validation: Verify internal logical consistency
• Accuracy Evaluation: Cross-check totals and calculated fields
• Temporal Analysis: Assess time-series completeness and gaps
• Outlier Detection: Identify anomalous values and patterns

4. Statistical Analysis

• Descriptive Statistics: Calculate measures of central tendency and dispersion
• Distribution Analysis: Examine value distributions and patterns
• Correlation Analysis: Identify relationships between variables
• Trend Analysis: Detect temporal patterns and changes
• Segmentation: Group and compare data subsets

5. Insight Generation

• Pattern Identification: Discover meaningful patterns in data
• Anomaly Detection: Find unusual values or inconsistencies
• Quality Scoring: Assign quality metrics to datasets
• Recommendation Generation: Provide actionable improvement suggestions

Tools & Techniques

DuckDB Integration

For advanced CSV analysis using SQL:

# Schema analysis
duckdb -jsonlines -c "DESCRIBE SELECT * FROM read_csv('url')"

# Statistical summarization
duckdb -jsonlines -c "SUMMARIZE SELECT * FROM read_csv('url')"

# Data sampling
duckdb -jsonlines -c "SELECT * FROM read_csv('url') USING SAMPLE N"

# Custom queries
duckdb -jsonlines -c "SELECT column_name, COUNT(*), AVG(value) FROM read_csv('url') GROUP BY column_name"