fixed-tensor-testing

Testing with Fixed Input Tensors

Purpose

Fixed tensor testing ensures deterministic, reproducible results for loss functions and model outputs. Enables verification without training dependencies.

Creating Fixed Tensors

1. Deterministic Seeding

import torch
import numpy as np

# Set all random seeds
torch.manual_seed(42)
np.random.seed(42)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(42)

2. Creating Test Tensors

# Fixed log probabilities (typically negative)
log_probs = torch.randn(batch_size, seq_len)
# Ensure reasonable log prob range (e.g., -2 to 0)
log_probs = torch.clamp(log_probs, min=-5.0, max=0.0)

# Fixed sequence lengths
seq_lengths = torch.randint(10, 100, (batch_size,))

# Fixed input IDs (for masking)
input_ids = torch.randint(0, vocab_size, (batch_size, seq_len))

3. Example Test Setup

batch_size = 4
seq_len = 10

# Generate fixed tensors
torch.manual_seed(123)
log_probs = torch.randn(batch_size, seq_len)
log_probs = torch.clamp(log_probs, -5.0, 0.0)

# Create mask (e.g., padding)
mask = torch.ones(batch_size, seq_len, dtype=torch.bool)
mask[:, seq_len-2:] = False  # Last 2 tokens are padding

# Lengths accounting for mask
seq_lengths = mask.sum(dim=1).float()

Saving Test Results

Save as .npz (NumPy Compressed)

import numpy as np

results = {
    'losses': loss.cpu().detach().numpy(),
    'log_probs': log_probs.cpu().detach().numpy(),
}

np.savez_compressed('/path/to/results.npz', **results)

Load .npz Files

data = np.load('/path/to/results.npz')
losses = data['losses']
print(f"Shape: {losses.shape}, dtype: {losses.dtype}")

Assertions and Validation

Basic Checks

# Loss should be finite and positive
assert torch.isfinite(loss).all(), "Loss contains NaN or Inf"
assert loss.item() > 0, "Loss should be positive"

# Shape validation
assert loss.shape == expected_shape, f"Shape mismatch: {loss.shape}"

# Range checks
assert loss.item() < 100, "Loss unreasonably large"
assert log_probs.min() >= -6.0, "Log probs too small"