Methodology: keeping simulations honest

This note describes how research at MAA is intended to be conducted. It is a statement of process, not of results. No results have been published.

What this document is — and is not

It is a commitment about how experiments will be designed, run, and labeled, so that the reasoning can be inspected and criticized. It is not evidence that any approach works. Where something below is a design goal rather than a built capability, it is a design goal.

Rule 1: point-in-time data only

An experiment may use only information that would have been available at the simulated moment. In practice that means:

• every record carries an as-of timestamp, and queries are bounded by it;
• data that is revised later (earnings restatements, index membership, corporate actions) is stored as it looked at the time, not as it looks today;
• delisted and discontinued instruments are kept in the universe, to avoid survivorship bias.

Rule 2: language models are a leakage risk

A model trained on text from after the simulated date may “know” what happened next. That makes a simulation look better than it could have been, and the effect is hard to measure. Our approach is to treat any model-assisted historical experiment as suspect unless the model’s training cutoff precedes the simulated period, and to record model versions with every run.

Rule 3: assumptions are written before results

Transaction costs, slippage, liquidity limits, and rebalancing frequency are specified in the experiment configuration before it runs. Changing an assumption after seeing a result starts a new, separately registered experiment.

Rule 4: validate out of sample, and count attempts

Hold-out periods and walk-forward validation are the default. Every experiment — including the ones that fail — goes into a registry. A single “good” result among many attempts is expected by chance alone, so the number of attempts must be known to interpret any result.

Rule 5: reproducibility

Each run records its configuration, data version, code version, model and prompt versions, and random seeds where supported. A result that cannot be reproduced is treated as a lead, not a finding.

Rule 6: honest labels

Label	Meaning
Historical simulation	Run on past data. Subject to hindsight, overfitting, and cost-model error.
Paper trading	Forward test against live feeds with no real capital.
Live	Real capital in real markets. Not applicable — MAA does not trade live.

These categories are never blended in a single figure, and a historical simulation is never described as if it were a track of real trading.

Status

This methodology is at the design stage and will be revised as the research progresses. Revisions will be dated on this page.