factor-data

Factor Data

Market data is the input contract for every FactorMiner workflow. This skill makes sure a dataset is schema-valid and split-covered before a mining run burns iterations on a broken file.

Canonical schema

FactorMiner expects an OHLCV panel with one row per (asset, timestamp):

Column	Meaning	Notes
datetime	Bar timestamp	Parseable date/datetime
asset_id	Instrument id	Aliases: code, ticker, symbol
open high low close	Prices	—
volume	Share/contract volume	—
amount	Dollar/turnover volume	vwap derived as amount / volume when missing

returns and vwap are derived automatically when absent. Column aliasing is handled by the loader, so near-canonical files pass.

Workflow

1. Validate

Always validate first:

factorminer validate-data path/to/market_data.csv --json

Read the report. It lists detected columns, applied aliases, derived fields, and train/test split coverage. If either split has zero rows, stop — fix the file or the config's data.train_period / data.test_period before mining. Use --strict to treat warnings as failures in CI.

2. Resample (optional)

If the bars are finer than the research horizon (e.g. 5-minute bars for a daily study), resample:

factorminer resample-data raw_5m.csv bars_1h.parquet --rule 1h

3. Fetch from an MCP connector (optional)

To pull data from a financial-data MCP connector instead of a local file, write a small MCP-source config and run fetch-data. The config maps the connector's tool and field names onto the canonical loader-required schema, including volume and amount:

factorminer mcp-connectors

# factset_source.yaml
transport: http
url: https://mcp.factset.com/mcp
headers:
  Authorization: "Bearer ${FACTSET_TOKEN}"
tool: get_prices
arguments:
  ids: ["AAPL-US", "MSFT-US"]
  start: "2022-01-01"
  end: "2024-12-31"
  frequency: "1d"
records_path: data.prices
field_mapping:
  datetime: date
  asset_id: fsym_id
  open: price_open
  high: price_high
  low: price_low
  close: price_close
  volume: volume
  amount: turnover