data-loading

BigQuery Data Loading & Export

Parent Skill: /bigquery
Path: /bigquery/data-loading

Purpose

Efficiently import data from various sources (GCS, local files, streaming) and export BigQuery results.

When to Use

Trigger automatically when:

• Keywords: load, import, export, CSV, JSON, Parquet, streaming
• User uploads file and mentions BigQuery
• User asks to "save query results"

Chat commands:

/bigquery/data-loading import CSV from gs://bucket/data.csv
/bigquery/data-loading load this file into table events
/bigquery/data-loading export query results to GCS
/bigquery/data-loading setup streaming for real-time data

Requirements

Verification

# Check GCS access
gsutil ls gs://your-bucket/

# Check BigQuery permissions
bq show your_project:dataset

---

Loading Patterns

Pattern 1: Load CSV from GCS

# Using bq CLI
bq load \
  --source_format=CSV \
  --skip_leading_rows=1 \
  --autodetect \
  dataset.table_name \
  gs://bucket/data.csv

# Using Python client
from google.cloud import bigquery

client = bigquery.Client()

job_config = bigquery.LoadJobConfig(
    source_format=bigquery.SourceFormat.CSV,
    skip_leading_rows=1,
    autodetect=True,
    write_disposition='WRITE_TRUNCATE'  # or WRITE_APPEND
)

uri = "gs://bucket/data.csv"
table_id = "project.dataset.table"

load_job = client.load_table_from_uri(
    uri, table_id, job_config=job_config
)

load_job.result()  # Wait for completion
print(f"Loaded {load_job.output_rows} rows")

Best Practices:

• Use --autodetect for schema inference (development only)
• Specify explicit schema for production
• Use WRITE_APPEND for incremental loads
• Partition target table by date if loading daily

---

Pattern 2: Load JSON (Newline-Delimited)