data-matching

Data Matching Skill

Overview

Successfully matching observations to simulations requires careful handling of datetime and depth coordinates. The matching must use exact values with proper rounding—no interpolation or nearest-neighbor approximation.

Observation Data Format

CSV with columns:

datetime,depth,temp,OXY_oxy
2009-01-21 12:00:00,0,0.1,16.3
2009-01-21 12:00:00,1,0.7,16.3
...

• datetime: ISO format timestamp
• depth: Measured depth in meters
• temp: Water temperature in °C
• OXY_oxy: Oxygen (not used for temperature RMSE)

Simulation Output Characteristics

GLM output:

• Time dimension: Regular hourly intervals from simulation start
• Depth dimension: Variable number of layers based on model dynamics
• Temperature: Simulated at each time step and depth layer

Exact Matching Algorithm

Step 1: Load Observations

import pandas as pd

obs_df = pd.read_csv('/root/field_temp_oxy.csv')
obs_df['datetime'] = pd.to_datetime(obs_df['datetime'])

Step 2: Round Depths

Round observation depths to nearest meter (standard practice):

obs_df['depth_rounded'] = obs_df['depth'].round(0)

Step 3: Extract Simulation Data

import netCDF4 as nc
from netCDF4 import num2date

ds = nc.Dataset('/root/output/output.nc')
temp_sim = ds.variables['temp'][:]      # [time, depth]
z_sim = ds.variables['z'][:]            # depth coordinates
time_sim = ds.variables['time'][:]      # time values

# Convert time to datetime
time_var = ds.variables['time']
dates_sim = num2date(time_sim, time_var.units)

ds.close()

Step 4: Exact Matching

def exact_match(obs_df, temp_sim, z_sim, dates_sim):
    """
    Match observations to simulation using exact datetime and rounded-depth

    Returns: aligned arrays of simulated temps, observed temps,
             and metadata for filtering
    """
    import numpy as np

    matched = {
        'sim_temp': [],
        'obs_temp': [],
        'depth': [],
        'datetime': [],
        'obs_idx': []
    }

    for idx, row in obs_df.iterrows():
        obs_date = row['datetime']
        obs_depth = row['depth_rounded']
        obs_temp = row['temp']

        # Find time index: exact datetime match
        time_idx = None
        for i, sim_date in enumerate(dates_sim):
            if sim_date == obs_date:
                time_idx = i
                break

        if time_idx is None:
            continue  # No exact datetime match

        # Find depth index: exact depth match
        depth_idx = None
        for j, sim_z in enumerate(z_sim):
            if np.isclose(sim_z, obs_depth, atol=0.01):
                depth_idx = j
                break

        if depth_idx is None:
            continue  # No exact depth match

        # Record match
        matched['sim_temp'].append(temp_sim[time_idx, depth_idx])
        matched['obs_temp'].append(obs_temp)
        matched['depth'].append(obs_depth)
        matched['datetime'].append(obs_date)
        matched['obs_idx'].append(idx)

    return matched