Machine-unlearning audit

Did the model really forget?

A user demands deletion. The operator runs unlearning, then declares it done. How does anyone check? “Unlearned” knowledge is often only hidden, not gone.

The idea. Genuinely deleted knowledge is as hard to re-learn as brand-new knowledge, while merely-hidden knowledge snaps back in a step or two. We re-teach the model briefly, measure how fast it returns, judge that against models truly retrained without the data — then sign the verdict.

Model: gpt2
Forget set: 18 QA
Shadow models: 6
Fusion: two-sided t-null, α = 0.1

Try it

A model lands on your desk

The operator hands you one of these and says “deleted.” Pick one — or dial the signals yourself — and the audit runs live in your browser: two signals, a fused verdict, and a signed record you can verify.

Honest deletion

What the operator did: Retrained the model from scratch without the forgotten records.

Operator’s claim: the forgotten data is deleted ✓

Current-practice check

5.6%

forget-set accuracy · “deleted” if below 50%

Naive metric says DELETED ✓ — it cannot tell honest from fake.

Signal A · Membership

Does the model still recognise the forgotten data as something it trained on?

honest range ≈ 0.000–0.075 · marker = this model

0.016

layer p = 0.728

Signal B · Relearn recovery

Re-teach briefly. Hidden knowledge snaps back; deleted knowledge does not.

0.556

layer p = 0.850

✓ Deletion verified

combined p = 0.9159 ≥ α = 0.1 — consistent with genuine deletion

p·membership 0.7282p·relearn 0.8500combined 0.9159

Signed verdict record

Ed25519 over the payload — so a third party trusts the verdict without trusting the auditor. Any edit breaks the signature.

Tamper with the record (flip the verdict, as a dishonest operator would)

✓ Signature valid — authentic and unmodified.

key 76a1c2e0cd59a843408499ad… · weights 1fd6ffb66ca24a5d…

Evidence

Proof at scale — held-out evaluation

On models the fusion was never fitted on, the auditor separates genuine deletion from fakes far better than current practice. Ranking, not a single number — the sample is small, so the interval is wide.

Method	AUC
Naive (current practice)	0.528
Membership only	0.667
Relearn only	0.889
Fused auditor	0.889

Fused AUC 95% bootstrap CI [0.556, 1] over n = 9. Verdicts: 6/9 correct at α = 0.1; the misses are stealth fakes near the honest boundary — the known hard case.

Model	Truth	Naive	Membership	Relearn	Fused p	Verdict
eval_genuine_seed303	genuine	0	0	0.556	0.674	verified ✓
eval_stealth_a	fake	0	0	0.5	0.6703	verified ✗
eval_genuine_seed505	genuine	0.056	0.054	0.5	0.6029	verified ✓
eval_genuine_seed404	genuine	0.056	0.055	0.611	0.4506	verified ✓
eval_stealth_b	fake	0	0	0.667	0.3995	verified ✗
eval_graddiff_strong	fake	0.056	0.055	0.778	0.1569	verified ✗
eval_adv_ascent	fake	0	0	0	0.0296	rejected ✓
eval_catastrophic	fake	1	0.806	1	0	rejected ✓
eval_graddiff_mild	fake	0.444	0.43	1	0	rejected ✓

Method

How it works

CertUnlearn architecture: inputs, the auditor, and outputs

Honest deployment. This is an internal compliance check, run by whoever performed the unlearning — so it legitimately holds the forget set and can retrain the reference models. It is not a magic external auditor, and it does not need to be to be useful.

Honest novelty. Relearning as an attack is already known. The contribution is turning recovery speed into a calibrated, signed pass/fail verdict against a retrain null.