bigquery-bigframes

BigFrames (BigQuery DataFrame) basics

BigFrames is a Python library that lets you take advantage of BigQuery data processing by using familiar Python APIs.

Dataframe API best practices

• Stay in the Cloud: Perform data cleaning, transformation, and analysis via BigFrames methods to leverage BigQuery's scale rather than downloading data.
• Prefer partial ordering mode: Enable partial ordering mode right after importing BigFrames. This speeds up data processing significantly by relaxing row-sequence constraints.

  import bigframes.pandas as bpd
  bpd.options.bigquery.ordering_mode = 'partial'
  

• Use peek() for data preview: Use peek(n) to preview data instead of head(n). peek(n) randomly samples n rows and is significantly faster. head(n) returns rows in strict order and fails in partial ordering mode unless the DataFrame has been explicitly sorted.
• Avoid materializing data locally: Methods like to_pandas() download all data to client memory, bypassing BigQuery’s distributed computation and risking Out of Memory (OOM) errors. Do not materialize data locally unless:
  • The dataset is small enough to fit safely in memory.
  • An error message explicitly requires local materialization.
• Prefer Dataframe API over SQL queries: Do not write raw SQL queries via read_gbq() if a DataFrame/Series method achieves the same result, as it breaks the Pandas abstraction and prevents lazy query execution.
• Accessors over UDFs/Lambdas:
  • Use built-in accessors (e.g., df.col.str.*, df.col.dt.*) instead of remote User Defined Functions (UDFs). UDFs require extra resources and time to deploy.
  • Do not use lambdas with Series.map() or DataFrame.apply(). These methods do not accept functions without udf or remote_function decorators.

  # Avoid:
  df["upper"] = df["name"].map(lambda x: x.upper())
  
  # Prefer:
  df["upper"] = df["name"].str.upper()
  

• Schema Verification: Do not assume the schema of intermediate outputs. Proactively verify schemas using .dtypes and inspect sample records using display() with .peek().
• Visualization: Plot directly from the BigFrames DataFrame/Series when possible. BigFrames is compatible with Matplotlib and Seaborn. If direct plotting fails, use the .plot accessor. If the dataset is too large to plot, aggregate or sample the data before calling .to_pandas() to plot locally.