Foolishly Naive Starting Points, Fundamental Flaws

The first version of the Civic Stress Index claimed to measure “structural stress in American institutions” using 7 formulas with sophisticated-sounding names — VAR-Granger, Loss-Averse Hawkes, Bayesian Hierarchical, Markov-Switching VAR — computed daily and displayed on a public dashboard, with a claimed provenance of having been “validated” against a documented 170-formula review process.

We archived all of it. Here’s why, in the order we found the problems.

Where this actually came from

Before the specific problems below: the 170 formulas weren’t designed one at a time from theory. They were generated by an automated, iterative process — 15 mathematical base templates times 6 parameter variations to produce 90 candidates, then a second “backward induction” pass expanding that to 160 — a mechanical combinatorial search over formula shapes, filtered afterward by which ones didn’t trip an obvious specification error.

The formulas that “passed” that filter didn’t pass because they were shown to model the real mechanism correctly. They passed because their free coefficients were tuned — backfit — to match the known outcome of a small number of historical reference episodes, by the project’s own account only “6–9 effective calibration episodes.” That’s an extremely small sample to fit multi-parameter formulas against under any circumstances, and it gets worse: several of the indicators being fit against — institutional trust chief among them — were never real data in the first place, just a fabricated placeholder standing in for it. Tuning coefficients to match known answers using data that was itself made up isn’t validation; it’s curve-fitting to noise and calling the fit a result.

That’s the actual root cause behind almost every problem below. The magic-number coefficients aren’t unsourced by accident — they’re what an automated backfitting process spits out when it’s optimizing to hit a target rather than deriving a mechanism. And it’s exactly why the formulas fail the historical test further down: something tuned to reproduce a handful of known past outcomes, using bad inputs, has no reason to generalize to a real situation it wasn’t fit against.

The formulas didn’t implement what they were named after

VAR (vector autoregression), Granger causality, and Hawkes processes are all fundamentally about relationships that unfold over time — lagged effects, self-exciting kernels, decay. The formula labeled “VAR-Granger Reduced-Form Linearization” was, in its entirety:

score = 35 + (50 - approval) * 0.35 + (gdelt - 100) * 0.15 + (cpi - 300) * 0.2

A static combination of that same day’s numbers. No lag, no time-dependence, nothing resembling the method in its name. Every one of the 7 production formulas followed this same pattern: a real, credible statistical name attached to a two- or three-term linear regression that had nothing to do with it.

The “7 validated formulas” claim wasn’t true

We keep a real review document recording individual verdicts for the formulas that were actually tested — some marked passing, most marked failed or flagged, each with a stated reason. Checking the 7 formulas that shipped to production against that record: only 2 had ever been individually reviewed and passed. One was explicitly flagged “empirically unverified.” The other four were never reviewed at all — just named in a closing summary paragraph, with the real verdict deferred to a methodology page that turned out to contain no content whatsoever.

Every coefficient was a magic number

Constants like 0.35, 0.15, 0.4, 3 appeared throughout the formulas with no citation and no fitting procedure visible anywhere in the code — consistent with them being leftovers of the backfitting process described above rather than derived from theory or a documented regression. The uncertainty ranges shown alongside each score were just that day’s score plus or minus a fixed constant — not derived from anything statistical, despite documentation elsewhere insisting that credible intervals were the only defensible output given how little calibration data existed.

Half the “formulas” were the same formula

Three of the seven took a single input — a fabricated approval-rating number — and did nothing to it but multiply and add different constants. They weren’t independent tests of different mechanisms; they were the same univariate line with three different slopes, dressed up as three different named methodologies.

The clearest test: it couldn’t tell COVID, the George Floyd protests, or January 6th apart from an ordinary month

We ran all 7 formulas against real economic data for five genuinely different historical periods — a quiet baseline month, COVID’s onset, the peak of the largest protest movement in US history, the aftermath of the Capitol riot, and a deliberately uneventful comparison month. Every formula stayed within a few points across all five. In two of them, the “quiet” comparison month scored higher than the actual crisis months. Not just insensitive — pointing the wrong way.

What we ignored along the way

This is the part worth being honest about, more than any individual bug: we reused sophisticated statistical language without ever checking the code matched it. We repeated “7 validated formulas” across multiple documents without once tracing that claim back to the one place that actually recorded verdicts. A completely empty methodology page sat live on the site for weeks without anyone treating its emptiness as a red flag about everything built around it. We fixed a “flat line” data bug and reported it as real progress without ever asking whether the formulas consuming that data could turn it into a historically sensible signal — that question only got asked when someone was explicitly asked to stop debugging symptoms and evaluate the whole thing critically.

What we kept

The real data-fetching infrastructure — FRED, BLS, GDELT, and NewsAPI integration — survived this cleanup intact. That work was sound; it was the scoring layer built on top of it that wasn’t. The Civic Stress Index is being rebuilt from that foundation, with the methodology and the code checked against each other from the start this time, not fixed reactively after the fact.

The full technical retrospective, including the formula-by-formula code and the historical test data, is preserved in the project repository under archive/csi-early-attempts/ for anyone who wants the complete accounting.