Weightless Neural Networks on Modern Compute

See the NN-Revival series page for methodology, scope, and future entries.

Abstract

We re-ran classic weightless neural networks — chiefly Aleksander's WiSARD and Kanerva's Sparse Distributed Memory — on binarized MNIST under a modern experimental protocol: shared model interface, natural (not matched) budgets, three seeds, and predictions pre-registered and frozen before multi-seed runs. A small MLP still leads full-data accuracy (0.9611±0.0016). Classic WiSARD is fast to write and competitive in the few-shot regime, but does not dominate 1-shot and never reaches 90% even at large N. Gradient-trained contents under the same n-tuple addressing close most of the accuracy gap to the MLP (0.9584±0.0044 at N=500, just below the MLP's 0.9611), at a large parameter and wall-clock cost. SDM on raw pixels is weak and radius-brittle. A dual-mode hybrid (frozen gradient RAMs + plastic one-shot writes), used as a class-incremental 0–7 → 8–9 probe, shows that sectioning prevents cell overwrite while a second interference channel — plastic logit domination of a shared sum — remains. This is a toy-scale study on binarized MNIST with small models; it is not a SOTA claim. A modern CNN would beat every arm here.

Why

Weightless nets were a serious research line from the 1960s–1990s and then largely left the mainstream conversation. The question for this note is narrow and empirical: if you implement them carefully on current CPUs, with honest budgets and frozen predictions, what still works, what fails, and which knobs (addressing vs contents; write vs gradient) actually matter?

We do not claim a new architecture family. WiSARD and SDM are prior art. Stable/plastic partitioning for continual learning is a known idea (e.g. progressive networks, PackNet, complementary learning systems). Applying a frozen/plastic RAM partition to a weightless substrate is under-explored in the current literature; that is the scope of the hybrid experiment, not an invention claim.

Scope

This series re-examines architectures in their original form under modern controls — to bring the primary designs to light and measure them honestly, not to survey or fold in the decades of descendants that followed. Where we extend one, the extension is ours and labeled as such (here: trained_wisard, hybrid). Vanilla WiSARD's N-scaling saturation is thus a property of the original write-based form; the sparse/routed, memory-reduced, and differentiable variants in the later literature are deliberately out of scope — not overlooked.

Setup

Data. MNIST, pixels binarized at 0.5. Real IDX required for multi-seed runs (train=60000, test=10000); no silent fallback.

Protocol. Seeds {0, 1, 2}; mean ± range with range = (max − min)/2. Few-shot k ∈ {1, 5} averaged over 10 subset draws per seed. Predictions written in SPEC.md before the corresponding multi-seed runs and never edited afterward — outcomes append only.

Two-knob framing. Every arm is a choice on two axes:

Arms (natural budgets, logged at run time).

Results

All figures below are from frozen multi-seed artifacts: results/baseline_full_frozen/, results/new_arms_full_frozen/, results/hybrid_full_frozen/.

Accuracy, few-shot, and train time (binarized MNIST)

WiSARD N-scaling (n=8, full train, 3 seeds)

Knee near N=1k–5k. Further N buys memory and time, not accuracy; at N=10k accuracy falls (bleach search selected threshold 0 — saturation/redundancy, not under-bleaching).

trained_wisard scaling and SDM peak

Q1 (sample-efficiency curve) and Q3 (N-scaling sweep) are independent multi-seed runs of the same N=100, n=8 config. Full accuracy at N=100 is 0.9422±0.0020 (Q1) / 0.9390±0.0027 (Q3) — a ~0.003 gap consistent with seed variance across separate campaigns. Below, Q1 is used for head-to-head arm comparisons; Q3 is used only for the N-scaling series.

• trained_wisard N=100 (Q3): 0.9390±0.0027 (256,000 params, ~44 s).
• trained_wisard N=500 (Q3): 0.9584±0.0044 (1,280,000 params, ~226 s) — just below the baseline MLP (0.9584 vs 0.9611).
• SDM best cell: M=2048, r=360 → 0.4273±0.0569; r=350/370 collapse to ~0.30; M=8192 does not rescue (0.3965±0.0194).

Hybrid continual learning (classes 0–7 offline → 8–9 online)

Selected cells (3 seeds):

Findings

1. The write rule is a ceiling, not the representation.

Under identical seeded n-tuple addressing, replacing one-shot bit-writes with gradient CE tables lifts full accuracy from 0.8767 (WiSARD N=1000, Q1) to 0.9422 (trained_wisard N=100, Q1). Separately, the Q3 scaling run puts trained_wisard at 0.9584 for N=500 — just below the baseline MLP's 0.9611. The addresses were already useful; bit-writes under-used them. Cost is honest and large: 256k–1.28M floats and ~45–226 s vs WiSARD's ~0.5 s.

2. n-tuple partitions beat random linear features and global Hamming balls.

Reservoir (frozen sparse binary proj + ridge) reaches 0.8414 full — above chance, below MLP/ramnet/WiSARD. SDM with the same weightless write rule as WiSARD tops out at 0.4273. Addressing scheme dominates contents rule when the address geometry is wrong for raw binary pixels.

3. Learning the address (STE ramnet) does not buy a win over a plain MLP.

Ramnet full 0.9385 < MLP 0.9611, and is the slowest baseline arm (25.7 s). Pre-registered: underperforms MLP — TRUE.

4. Few-shot and speed edges are real but narrow.

• Pre-registered "WiSARD wins sample-efficiency at k≤10" is only partly true: WiSARD leads at k=5 (0.650) and k=10 (0.753), but at k=1 reservoir wins (0.419 > WiSARD 0.390). Gaps are small, not "≫".
• Write phase is ~11× faster than MLP (379 ms vs 4241 ms). Total WiSARD time including bleach search is only ~8× (532 ms). Speed claims must split write vs bleach.

5. Scaling WiSARD does not hit 90% on this protocol.

Pre-registered P2.4 (≥90% cheaply at N≈1k–5k) is FALSE. Best mean is 0.8769 at N=1000. N=10k degrades to 0.8474.

6. SDM is a dead end on raw binarized pixels here.

Brittle radius (peak only at r=360 among {350, 360, 370}), weak absolute accuracy, no help from larger M.

7. Dual-mode hybrid: backprop-free adaptation without cell overwrite — with a second failure mode.

At write scale 1×10⁻⁴, sectioned forget = 0.000 while naive forget = 0.725. Sectioning stops writes from corrupting gradient-trained cells. But plastic mass still shares the readout sum: at higher scales, retain collapses even though frozen cells are untouched (logit domination). Usable sweet spots: (0.90, 5×10⁻⁴) overall 0.802 with forget 0.103; (0.50, 1×10⁻⁴) overall 0.815 with forget 0.082. Naive has no usable operating point under this protocol — any scale that adapts also destroys retention.

What it's good for

Not leaderboard accuracy. The weightless write path is attractive when:

• train-time budget is measured in milliseconds and parameters must be zero;
• few-shot regimes around k=5–10 matter more than full-data ceiling;
• a frozen substrate must absorb new classes online without backprop (hybrid sectioning), accepting that write scale must be calibrated so plastic does not dominate the shared sum.

Edge devices and continual "add a class without replaying the base set" are the natural niches — not ImageNet.

Limitations

• Binarized MNIST only. No grayscale, no CIFAR, no language.
• Small models. A one-hidden-layer MLP already wins full accuracy; a CNN would beat every arm in this note. "MLP-level" here means small dense net level, not modern deep-net level.
• Plastic adaptation is write-quality. Hybrid 8–9 accuracy is bounded by one-shot additive writes into a minority of RAMs, not by a second gradient phase.
• Write-scale calibration is protocol-sensitive. O(1) scales with ~6k samples/class swamp frozen CE logits; the operating band used was {10⁻⁴, 5×10⁻⁴, 10⁻³, 5×10⁻³}.
• Natural budgets, not matched parameters. Comparisons state each arm's footprint; they are not iso-parameter bake-offs.

Future work

This is note #1 in a Forgotten Architectures series: re-implement older inductive biases under modern controls (frozen specs, multi-seed, honest negatives). Candidates for later notes include sparse/routed (e.g., CMAC-style) addressing — sending each input to a specialized subset of RAMs, to test whether specialization avoids the N-scaling saturation seen here — other weightless variants, hashing-based memories, and hybrid stable/plastic designs with separate readouts (to test whether decoupling the sum removes soft forgetting).

Reproducibility

• Code: pending public repository — the ramnet-study package under NN-Revival.
• Spec: ramnet-study/SPEC.md — predictions frozen before each multi-seed campaign; outcomes append-only.
• Frozen results (do not overwrite):
  • ramnet-study/results/baseline_full_frozen/ — 4-arm Q1/Q2, 3 seeds
  • ramnet-study/results/new_arms_full_frozen/ — wisard / trained_wisard / sdm, 3 seeds
  • ramnet-study/results/hybrid_full_frozen/ — Q5 sectioned vs naive, 3 seeds

SHA-256 of frozen JSON (expected values; verify with the command below):

shasum -a 256 \
  ramnet-study/results/baseline_full_frozen/summary.json \
  ramnet-study/results/new_arms_full_frozen/summary.json \
  ramnet-study/results/hybrid_full_frozen/hybrid_summary.json

Commands: cargo run --release -- --full · --new-arms · --hybrid-full (require real MNIST under data/mnist/).

Predictions that came out FALSE or only partly true (P1.2, P1.3 total-time form, P2.4 ≥90%, N=10k monotone, naive-usable hybrid band) are left in the scorecard as written. No retcon.