Quant Toolbox — Research Desk
Distributional forecasting, scored in public
We measured one thing: how Peter Cotton’s laplace forecaster (the skaters package) scores, out of sample, against four honest classical baselines — on the economic series class its author recommends it for, and, reported separately, on the asset-price stratum his project lists as unscored.
Our publication rule, stated publicly and binding on this desk: we publish results as they come out, including every loss — per series, in the same table and the same size type as the wins. No post-hoc series drops. No metric swaps.
Pass 1 results — Panel A: FRED change series (the recommended domain)
Mean held-out log score is the primary metric (higher is better); CRPS is the secondary (lower is better). ΔLL is Laplace’s log-score margin over the strongest baseline on that series. The CRPS verdict names the strongest baseline on CRPS. Calibration columns are Laplace’s.
| Series | Freq. | Test n | Laplace log score | Best baseline log score | ΔLL | Laplace CRPS | Best baseline CRPS | CRPS verdict | PIT mean | 95% coverage |
| DGS10 | daily | 500 | 1.834 | 1.542 (AR(1)-G) | +0.292 | 0.02734 | 0.0277 (AR(1)-G) | Win | 0.501 | 0.95 |
| DGS2 | daily | 500 | 1.975 | 1.389 (AR(1)-G) | +0.585 | 0.02903 | 0.0308 (SES-G) | Win | 0.499 | 0.94 |
| VIXCLS | daily | 500 | -1.867 | -2.219 (AR(1)-G) | +0.352 | 0.9772 | 0.9526 (AR(1)-G) | Loss | 0.488 | 0.95 |
| DEXUSEU | daily | 500 | 3.936 | 3.806 (AR(1)-G) | +0.129 | 0.002663 | 0.002792 (AR(1)-G) | Win | 0.488 | 0.95 |
| DCOILWTICO | daily | 500 | -2.097 | -2.818 (SES-G) | +0.722 | 1.27 | 1.302 (SES-G) | Win | 0.506 | 0.94 |
| ICSA | weekly | 500 | -11.590 | -34.563 (Naive-G) | +22.973 | 30,137 | 35,706 (Naive-G) | Win | 0.499 | 0.92 |
| WRESBAL | weekly | 498 | -12.517 | -12.746 (SES-G) | +0.230 | 37,810 | 38,544 (AR(1)-G) | Win | 0.490 | 0.95 |
| CPIAUCSL | monthly | 382 | -0.317 | -1.683 (Naive-G) | +1.366 | 0.3139 | 0.3323 (AR(1)-G) | Win | 0.507 | 0.94 |
| UNRATE | monthly | 377 | 2.585 | -3.126 (AR(1)-G) | +5.712 | 0.1533 | 0.1432 (SES-G) | Loss | 0.498 | 0.93 |
| INDPRO | monthly | 500 | -0.811 | -2.259 (SES-G) | +1.448 | 0.3619 | 0.3442 (SES-G) | Loss | 0.497 | 0.95 |
| PAYEMS | monthly | 421 | -6.657 | -16.386 (Naive-G) | +9.729 | 202.6 | 183.1 (AR(1)-G) | Loss | 0.507 | 0.95 |
| HOUST | monthly | 325 | -6.001 | -6.028 (AR(1)-G) | +0.027 | 54.2 | 54.55 (AR(1)-G) | Win | 0.509 | 0.94 |
Read plainly: on the primary metric, Laplace beat every baseline on all 12 series. On CRPS it beat the naive baselines on all 12 and lost on 4 of 12 — VIXCLS and PAYEMS to AR(1)-G, UNRATE and INDPRO to SES-G. Those losses are rows in the table above, not a footnote.
Panel B: asset prices (out-of-scope stress test — always reported separately)
| Series | Freq. | Test n | Laplace log score | Best baseline log score | ΔLL | Laplace CRPS | Best baseline CRPS | CRPS verdict | PIT mean | 95% coverage |
| SP500 | daily | 500 | 3.284 | 3.138 (AR(1)-G) | +0.147 | 0.005263 | 0.005323 (AR(1)-G) | Win | 0.504 | 0.94 |
| NASDAQCOM | daily | 500 | 2.958 | 2.847 (AR(1)-G) | +0.112 | 0.007224 | 0.007177 (AR(1)-G) | Loss | 0.511 | 0.94 |
| GOLDAMGBD228NLBM | daily | — | Not scored — skipped under the pre-registered data-defect clause: FRED's endpoint returns an HTML page, not CSV, for this series (verified twice, 2026-11-19). No substitute was inserted after the fact. |
Panel B is the stratum the skaters project itself lists as unscored (its issue #338). Laplace won the log score on both scored series and split CRPS. The forecaster’s own documentation recommends it for non-price economic series; Panel B is a stress test outside that recommendation and is never pooled with Panel A.
Credit
Methodology: Cotton, Peter — Transforms All the Way Down — Automatic Online Distributional Forecasting by Conjugation (SSRN 7220478; https://skaters.microprediction.org), cited as its author requests. This page is Quant Toolbox’s own work and words; factual credit only — no endorsement by the cited author is stated or implied.