Cascade-Correlation on Modern Compute

See the NN-Revival series page for methodology, scope, and other entries. Prior notes: #1 — Weightless Neural Networks and #2 — CMAC.

Abstract

Cascade-Correlation (Fahlman & Lebiere, 1990) learns by growing structure on demand: it starts with inputs wired straight to outputs, and whenever training error stalls it adds one hidden unit, trained to correlate with the residual error and then frozen permanently, with each new unit cascaded on top of all previous ones. It was designed to defeat the moving-target problem — the mutual interference of layers trained together under backprop — and its signature triumph was two-spirals, a task fixed-size backprop nets of the era struggled with. We re-ran it faithfully in its native domain (two-spirals + a low-dim regression surface) with a labeled out-of-domain MNIST comparability arm, under the series protocol: three seeds, natural budgets, predictions pre-registered and frozen before any runs, results frozen and SHA-256-hashed, all figures generated from frozen JSON.

The mechanism reproduces cleanly — it solves two-spirals (100% train) by growing ~22 frozen units — but the founding legend does not survive modern training. Given a fair modern shot (mini-batch Adam, good init, depth), a plain two-layer MLP solves two-spirals at 99.65% held-out where Cascade-Correlation reaches only ~79% (82% even at a generous oracle early-stop), and beats it ~7× on the regression surface and by 13–23 points on MNIST. This is not evidence the 1990 idea was wrong; it is a story of asymmetric investment — the mainstream path received three decades of cumulative tooling that dissolved the very problem Cascade-Correlation was built to route around. The residual value is in the mechanism, not the accuracy: it genuinely allocates capacity to fit the problem (units grown scale with difficulty), trains cheaply one unit at a time without full-net backprop, and is markedly parameter-compact — but it overshoots useful depth and cannot back out, and its connection count grows quadratically. A toy-scale study; not a SOTA claim.

Why

Cascade-Correlation is the cleanest pre-1995 instance of grow-on-demand learning — allocate structure as the problem demands it, rather than fixing width and depth up front. That makes it the natural third probe in this series: after #1 (weightless memory) and #2 (sparse addressing), #3 tests growth. Its historical fame rests on a specific empirical claim — that it cracked two-spirals where backprop of the day could not — so the honest question is narrow: implement it faithfully, give both it and a modern backprop baseline a fair fight, and report what actually happens.

We claim no new architecture. Cascade-Correlation is Fahlman & Lebiere's. The candidate-pool, correlation objective, and frozen cascade are theirs.

Scope

This series re-examines architectures in their original form under modern controls. Cascade-Correlation is not domain-specialized (it is a general supervised learner); what is dated is its mechanism — constructive growth with a correlation objective and no end-to-end backprop. Its native proving ground is small supervised problems with hard nonlinear structure, so the faithful anchor is two-spirals plus a low-dim regression surface; MNIST is a labeled, out-of-domain comparability arm carried for cross-entry comparison, not a task where Cascade-Correlation is expected to shine.

One interpretive frame, stated up front and honestly. The comparison in this note is developmentally asymmetric: Cascade-Correlation received a few years of attention before the field consolidated around end-to-end backprop, while the backprop MLP has had three decades of cumulative improvement (better initialization, Adam, normalization, routine depth). Several of those improvements directly dissolve the moving-target problem Cascade-Correlation was invented to avoid. So a modern MLP winning here is overtaking, not refutation — the 1990 result was real for 1990. We flag this as interpretation, not measurement: we did not test the counterfactual "would it win with comparable modern investment," and make no claim about it.

Setup

Tasks.

• Two-spirals — the standard 194-point benchmark (97 points per spiral, 3 turns), plus a held-out test set sampled at finer angular resolution between the training points. (Two-spirals is classically a fit-the-set task; the held-out generalization lens is our modern addition, and is where overfitting shows up.)
• Regression — the 2-D nonlinear surface reused from study #2 (cross-entry continuity). Held-out RMSE.
• MNIST — binarized, via research-harness::mnist; one-vs-rest, k ∈ {200, 500} per class. Out-of-domain comparability.

Arms. cascor (faithful: candidate pool ≥ 8, correlation-trained, frozen cascade) vs backprop MLP baselines. For the native comparison we report both a matched single-hidden-layer MLP (same unit count as Cascade-Correlation grew) and the best-found MLP over an architecture × activation × learning-rate sweep — the fair comparator to a deep cascade is a net allowed depth.

Protocol. Seeds {0, 1, 2}; mean ± half-range. Predictions frozen in SPEC.md 2026-08-20, before any runs; outcomes append-only. Results frozen and SHA-256-hashed via research-harness::checksum; every figure below generated from the frozen JSON by gen_tables.py.

Two fairness guards, both applied. (1) The MLP baseline was initially broken (full-batch, ~55% — chance); it was refit with mini-batch Adam and an architecture sweep until it cleared chance decisively. (2) Cascade-Correlation was then given a matching hyperparameter sweep (candidate pool, candidate/output training budgets, activation, stopping) so its result is an honest ceiling, not under-tuning — this raised its two-spirals held-out from ~76% to ~82%. Neither arm is strawmanned.

Results

All figures from frozen artifacts: results/full_campaign_frozen/, results/cascor_sweep_frozen/, results/stage_a_frozen/.

Q1 — two-spirals (3 seeds, mean ± half-range)

Cascade-Correlation separates the training set perfectly and generalizes to ~79% (82% if allowed a generous held-out-picked stopping point — an optimistic oracle, and still short). The matched single-layer MLP genuinely fails (~61%) — so at matched capacity the constructive approach does beat naive fixed backprop. But a fair MLP allowed depth solves the task at 99.65%. The "backprop can't do two-spirals" claim is a fact about 1990's training, not about backprop.

Q1 — regression (#2 surface, 3 seeds)

On the smooth surface even the matched MLP edges Cascade-Correlation, and the best MLP beats it ~7× on held-out RMSE at ~5× fewer parameters. No hard boundary to reward growth here.

Q2 — adaptive capacity (the mechanism working)

Spirals, units grown vs difficulty (cascor):

Regression, units grown vs training-set size (cascor):

The easy spiral needs ~8 units; the hard variants grow to ~16–20 (not perfectly ordered among the hard cases, but easy ≪ hard). More regression data → more units to reach the solve threshold, with better held-out error. Capacity is genuinely allocated to fit — the mechanism does what it claims.

Q3 — cost and failure mode (N4, N6)

Overshoot (N4), two-spirals: train accuracy always reaches 1.0 while held-out peaks mid-growth then drifts down — the train−test gap is 0.2083±0.0261, versus ~0.00 for the best MLP. Because Cascade-Correlation is monotone-additive — it can only add frozen units, never remove or adjust one — it overshoots useful depth and cannot back out. Output- weight magnitude grows then plateaus (~4–5) and a high fraction of cascade units saturate. On the smooth regression surface this overshoot is largely absent (held-out improves to the solve point) — the overshoot appears where train-fit and generalization diverge (the hard boundary), not on the smooth surface.

Quadratic growth (N6): each unit connects to all previous units, so connections ≈ ½H² + 3½H + 3 (inputs=2). Measured/analytic:

Cost is quadratic in units grown — quadratic in model growth (a different axis from the transformer's context-length quadratic; Cascade-Correlation has no context axis).

Per-unit cost (N3): only the candidate pool and the output layer train per install — never the whole net — so per-unit training is local and cheap; total time grows with units and with fan-in as the cascade deepens. (Absolute wall-clock is still large here because we used plain gradient descent, not the original's Quickprop.)

Q4 — MNIST comparability (out-of-domain, 3 seeds)

Off its native turf, Cascade-Correlation loses by 13–21 points — but at 11–17× fewer parameters. Compact, not competitive. (Full-dataset one-vs-rest was skipped as wall-clock-unbounded; k=200/500 is the comparability arm.)

Scorecard (pre-registered → outcome; predictions never edited)

The author's high-confidence prediction (N4) landed; the author-set medium-confidence N1 was the headline miss. Both are kept exactly as frozen. No retcon.

How to read this result

Cascade-Correlation lost every accuracy comparison we ran — and that is the honest headline. But the reason matters, and it is not "the 1990 idea was bad." Two things are simultaneously true:

1. It worked, and still works, as advertised. It solves two-spirals by growing a small frozen cascade; it allocates capacity to difficulty; it trains one cheap local step at a time. The mechanism is faithful and it does what Fahlman & Lebiere said.
2. The problem it was built to beat has been dissolved by thirty years of investment in the other path. The moving-target problem was a real obstacle for 1990 backprop; Adam, modern initialization, and routine depth largely remove it. A general mechanism that receives three decades of cumulative tooling will overtake a specialized one that received a few years — and then was forgotten. That is what we are measuring: overtaking, not error.

So the verdict is (mostly) earned retirement in the honest sense of the series: not a flaw exposed, but a road-not-taken that the field stopped developing. Its live, under-explored value is structural — grow-to-fit capacity, frozen features that never re-interfere, cheap backprop-free-ish updates, real parameter-compactness — not leaderboard accuracy. Whether that mechanism would repay comparable modern investment is a genuinely open question, and one this study deliberately does not answer.

Exploratory rethinks of the overshoot (not pre-registered)

After freezing the faithful result, three fixes for the N4 overshoot were tried as exploration — no pre-registered predictions, all three run, none cherry-picked: grow-then-prune (ablation), a keep-diverse candidate pool, and relevance gating. A first pass was confounded (pruning decided on a single tiny validation split and over-pruned, hurting held-out); a corrected, leakage-clean pass made every prune/gate decision on 5-fold cross-validation within the training set, never the test set. Under that clean signal the result is a firm null: conservative pruning makes no safe edits at all — the grown units are genuinely used — gating moves the cross-validation loss imperceptibly and does not transfer to held-out, and refreshing units in place changes nothing. None closes the gap to the MLP. This sharpens N4: the overshoot is structural — used capacity that overfits the hard boundary, not removable redundancy — so it cannot be pruned away. (A clean win here would have been re-tested under pre-registration in a later entry, not claimed from this exploratory pass.)

What it's good for

Not accuracy on any task here. The constructive/frozen mechanism is attractive where its structure is the point:

• you want the model to size itself to the problem rather than guess width/depth;
• you want cheap, local, backprop-free-ish per-unit training on small supervised problems;
• you value parameter-compactness and interpretable growth over the last points of accuracy;
• frozen features that never re-interfere are useful (a property worth revisiting for continual learning — banked).

Not for high-dimensional perception, and not where a well-trained MLP is available and accuracy is what you're buying.

Limitations

• Toy scale, small baselines. A one- or two-hidden-layer MLP is the comparator; a real CNN would beat every arm on MNIST.
• Two-spirals held-out is our lens. The task is classically fit-the-set; we added a held-out set to measure generalization, and that is where the overshoot appears.
• The best-curve (82%) number is generous to CasCor — it uses held-out to pick the stopping point (an oracle early-stop). The honest no-peeking figure is ~79%. Even the generous number loses.
• Plain gradient descent, not Quickprop. The mechanism is faithful; the optimizer is not the 1990 original, so absolute wall-clock is not comparable to the historical result.
• MNIST is k=200/500 one-vs-rest only (full OVR was wall-clock-unbounded); labeled out-of-domain.
• The asymmetric-investment framing is interpretation, not measurement. We did not test whether the mechanism would win with comparable modern development, and claim nothing about it.

Series note

Three entries now sort three different ways: #1 (WiSARD) — revival partly works; the write rule, not the representation, is the ceiling. #2 (CMAC) — revival works; sparse tiled addressing genuinely avoids the saturation #1 exposed. #3 (Cascade-Correlation) — (mostly) earned retirement; a faithful, elegant mechanism overtaken by cumulative investment in mainstream backprop, its residual value structural rather than performance. Two partial-revivals and one retirement is the series doing its sorting job — evidence it is an honest audit, not a vindication campaign. Banked for later: frozen-feature growth for continual learning (P8, echoing #1's sectioning). The three exploratory rethinks of the N4 overshoot were run post-freeze (see Exploratory rethinks above); the honest outcome is a firm null — the overshoot is structural, not patchable by these.

Reproducibility

• Code: github.com/Vulkgryph/cascor-study — package cascor-study under NN-Revival; bundles the research-harness scaffolding vendored in-repo (MNIST loading, aggregation, summary/freeze, SHA-256 checksums).
• Spec: SPEC.md — predictions frozen 2026-08-20 before the runs; outcomes append-only.
• Frozen results (do not overwrite): results/full_campaign_frozen/, results/cascor_sweep_frozen/, results/stage_a_frozen/.

SHA-256 of frozen JSON (verify with the manifest; same shasum -a 256 command as #1/#2):

cd cascor-study/results && shasum -a 256 -c FROZEN_SHA256.txt

Tables: every figure above is generated from the frozen JSON by cascor-study/gen_tables.py (using research-harness's harness_tables.py) → results/GENERATED_TABLES.md. No result numbers are hand-typed.