Quant Toolbox — Research Desk

Distributional forecasting, scored in public

We measured one thing: how Peter Cotton’s laplace forecaster (the skaters package) scores, out of sample, against four honest classical baselines — on the economic series class its author recommends it for, and, reported separately, on the asset-price stratum his project lists as unscored.

Our publication rule, stated publicly and binding on this desk: we publish results as they come out, including every loss — per series, in the same table and the same size type as the wins. No post-hoc series drops. No metric swaps.

Pass 1 results — Panel A: FRED change series (the recommended domain)

Mean held-out log score is the primary metric (higher is better); CRPS is the secondary (lower is better). ΔLL is Laplace’s log-score margin over the strongest baseline on that series. The CRPS verdict names the strongest baseline on CRPS. Calibration columns are Laplace’s.

SeriesFreq.Test nLaplace log scoreBest baseline log scoreΔLLLaplace CRPSBest baseline CRPSCRPS verdictPIT mean95% coverage
DGS10daily5001.8341.542 (AR(1)-G)+0.2920.027340.0277 (AR(1)-G)Win0.5010.95
DGS2daily5001.9751.389 (AR(1)-G)+0.5850.029030.0308 (SES-G)Win0.4990.94
VIXCLSdaily500-1.867-2.219 (AR(1)-G)+0.3520.97720.9526 (AR(1)-G)Loss0.4880.95
DEXUSEUdaily5003.9363.806 (AR(1)-G)+0.1290.0026630.002792 (AR(1)-G)Win0.4880.95
DCOILWTICOdaily500-2.097-2.818 (SES-G)+0.7221.271.302 (SES-G)Win0.5060.94
ICSAweekly500-11.590-34.563 (Naive-G)+22.97330,13735,706 (Naive-G)Win0.4990.92
WRESBALweekly498-12.517-12.746 (SES-G)+0.23037,81038,544 (AR(1)-G)Win0.4900.95
CPIAUCSLmonthly382-0.317-1.683 (Naive-G)+1.3660.31390.3323 (AR(1)-G)Win0.5070.94
UNRATEmonthly3772.585-3.126 (AR(1)-G)+5.7120.15330.1432 (SES-G)Loss0.4980.93
INDPROmonthly500-0.811-2.259 (SES-G)+1.4480.36190.3442 (SES-G)Loss0.4970.95
PAYEMSmonthly421-6.657-16.386 (Naive-G)+9.729202.6183.1 (AR(1)-G)Loss0.5070.95
HOUSTmonthly325-6.001-6.028 (AR(1)-G)+0.02754.254.55 (AR(1)-G)Win0.5090.94

Read plainly: on the primary metric, Laplace beat every baseline on all 12 series. On CRPS it beat the naive baselines on all 12 and lost on 4 of 12 — VIXCLS and PAYEMS to AR(1)-G, UNRATE and INDPRO to SES-G. Those losses are rows in the table above, not a footnote.

Panel B: asset prices (out-of-scope stress test — always reported separately)

SeriesFreq.Test nLaplace log scoreBest baseline log scoreΔLLLaplace CRPSBest baseline CRPSCRPS verdictPIT mean95% coverage
SP500daily5003.2843.138 (AR(1)-G)+0.1470.0052630.005323 (AR(1)-G)Win0.5040.94
NASDAQCOMdaily5002.9582.847 (AR(1)-G)+0.1120.0072240.007177 (AR(1)-G)Loss0.5110.94
GOLDAMGBD228NLBMdaily—Not scored — skipped under the pre-registered data-defect clause: FRED's endpoint returns an HTML page, not CSV, for this series (verified twice, 2026-11-19). No substitute was inserted after the fact.

Panel B is the stratum the skaters project itself lists as unscored (its issue #338). Laplace won the log score on both scored series and split CRPS. The forecaster’s own documentation recommends it for non-price economic series; Panel B is a stress test outside that recommendation and is never pooled with Panel A.

Protocol

Reproduce it

python3 -m venv /tmp/cotton-venv
/tmp/cotton-venv/bin/pip install skaters numpy
/tmp/cotton-venv/bin/python benchmark.py

The full harness — pre-registration, code, data, machine-readable results, and disclosures — is published with these results as the desk’s standing artefact.

Limitations — volunteered, at headline level

Credit

Methodology: Cotton, Peter — Transforms All the Way Down — Automatic Online Distributional Forecasting by Conjugation (SSRN 7220478; https://skaters.microprediction.org), cited as its author requests. This page is Quant Toolbox’s own work and words; factual credit only — no endorsement by the cited author is stated or implied.

© 2026 Quant Toolbox LLC · Research desk · Results published as they come out, including every loss.