Skip to content

paper-watch: CADMA capacity-aware coverage decomposition (arXiv:2611.12243) — toy study - #562

Open
mikkel wants to merge 1 commit into
masterfrom
paper-watch/cadma-metrics
Open

mikkel wants to merge 1 commit into
masterfrom
paper-watch/cadma-metrics

Conversation

@mikkel

@mikkel mikkel commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Paper watch, cycle 2 — CADMA coverage decomposition

Paper: "CADMA: Capacity-Aware Coverage Decomposition for Generative Model Assessment" (Meng et al.), arXiv:2611.12243 — https://arxiv.org/abs/2611.12243 (Sections 3 and 4.1 read before building). Research note; nothing merges — review only.

What this adds

  • lib/toy_metrics.py: cadma_scores(real, fake, k=5, regularize=True).
  • tests/test_cadma_metrics.py: the study below (constructions + k-sensitivity + real toy-GAN runs), with local equivalent implementations of the two control metrics, since master does not carry cycle 1's naeem_coverage / gini_distance_correlation (they live on paper-watch/gini-distance-metric, PR Paper watch: EmbeddGAN salvage — Gini distance correlation as an evaluation metric #559; branches deliberately not merged).

Formulas (paper Sec. 3)

Real set R (size N), generated set F (size M = N). rho_i = distance from r_i to its k-th nearest neighbour within R; radius regularization (Eq. 10) eps_i = min(rho_i, mean(rho)). Admissible matrix A_ij = 1[ ||r_i - f_j|| <= eps_i ]; Z* = maximum-cardinality bipartite matching under A (Hopcroft–Karp via scipy.sparse.csgraph.maximum_bipartite_matching), N_mat = |Z*|.

  • Capacity coverage C_cap = N_mat / N (Eq. 7)
  • Mode-dropping score D_drop = Coverage - C_cap, Coverage = Naeem coverage on the same eps radii (Eq. 8)
  • Mode-collapse score D_collapse = (M_cov - N_mat) / N, M_cov = # generated samples inside ≥ 1 real neighbourhood (Eq. 9)

So Coverage = C_cap + D_drop exactly. Fully uncovered real samples lower C_cap directly and count in neither diagnostic (paper footnote 1) — this matters for reading the toy-GAN result below.

Study design (ground truth by construction, ≥ 3 seeds wherever sampling noise exists)

7-mode Gaussian mixture (ring radius 3.0, σ = 0.35), N = M = 1400, seed-mean over 3 seeds.

  1. Density shift (paper Sec. 4.1.1): generated mass progressively moved from 6 modes into the remaining 1, stages t ∈ {0, .2, .4, .6, .8, 1}; reference effective coverage decreases linearly 1 → 1/7.
  2. Intramode concentration: generated set keeps the full geometric footprint of a fresh draw, but only a fraction f ∈ {1, .8, .6, .4, .2} of its points are distinct (rest exact duplicates); reference = f (distinct-capacity fraction). Geometry coverage should stay deceptively high while capacity falls.
  3. Outlier pollution: fraction p ∈ {0, .15, .3, .45, .6} of generated samples replaced by far outliers (cluster at (14, 14)); reference = 1 − p.

Controls: Naeem coverage with raw (uncapped) radii, coverage on CADMA's regularized radii, and the cycle-1 Gini distance correlation ρ_g. ρ_g is a discrepancy (0 = identical), not a coverage estimate; its "MAE" below is for the proxy 1 − ρ_g and is labelled as such.

Plus: CADMA applied to real toy-GAN runs (cycle-1 style 8-Gaussian ring, plain MLP GAN, 1200 steps, 3 seeds) — a covered run (trained on all 8 modes) and a collapsed run (trained on modes {0, 1}, evaluated against the full mixture).

k choice + sensitivity

Paper: k = 70 at N = 25000 (k/N = 0.0028). Here N = 1400 and k = 5 (k/N = 0.0036), which also matches the incumbent Naeem control's k so neighbourhood orders are equal. Sensitivity grid (3 seeds, seed-mean C_cap MAE; D_collapse rise = concentration f = 1 → 0.2):

k density MAE concentration MAE outlier MAE D_collapse rise (concentration)
3 0.117 0.150 0.178 — (matching starved; C_cap loses to regularized coverage)
5 0.060 0.182 0.091 0.303
10 0.022 0.275 0.033 0.223
20 0.007 0.339 0.011 0.135

Density/outlier tracking improves monotonically in k, but concentration detection degrades monotonically — each distinct prototype can support ~k matches, so duplication hides as k grows. k = 5 is the smallest k at which C_cap beats both coverage variants on all three constructions, and where the decomposition is most informative. The test encodes the k ∈ {3, 10} subset of this trade-off.

Iteration log (gauntlet, 4 rounds — one lever per round: k)

  • Round 1 (k = 5, baseline): density MAE C_cap 0.060 vs Naeem-raw 0.189; outlier 0.091 vs 0.224; concentration 0.182 vs 0.260 — claim (i) holds comparatively on all three. Identity baseline C_cap only 0.813 (finite-N matching deficit), inflating absolute MAEs.
  • Round 2 (k = 3): worse across the board (0.117 / 0.150 / 0.178) and C_cap loses to regularized coverage — radii too small, matching starved. Rejected.
  • Round 3 (k = 10): density 0.022, outlier 0.033 (paper-like), but concentration MAE degrades to 0.275 and the D_collapse rise shrinks.
  • Round 4 (k = 20): density 0.007, outlier 0.011, concentration 0.339, D_collapse rise only 0.135. Confirms the trade-off is monotone and structural, not noise.
  • Decision: ship k = 5 (see above). No construction was tuned at any point; only k moved.

Final results (k = 5, seed-mean MAE vs reference)

construction C_cap Coverage (regularized radii) Naeem coverage (raw) 1 − ρ_g (proxy)
density shift 0.060 0.152 0.189 0.249
intramode concentration 0.182 0.207 0.260 0.400
outlier pollution 0.091 0.178 0.224 0.054

For scale, the paper's headline (their Table 1, N = 25000, CIFAR-10 + Gaussian): C_cap average MAE 0.027 vs Coverage 0.209, Recall 0.213. Our toy-scale C_cap MAEs are larger in absolute terms (finite-N matching deficit: C_cap ≈ 0.81 for two independent draws of the same mixture at N = 1400, k = 5), but the comparative pattern reproduces.

Diagnostic separation (claim ii):

  • Concentration (f = 1 → 0.2): D_collapse 0.119 → 0.422 while D_drop falls 0.103 → 0.012; at f = 0.2, D_collapse > 3 × D_drop. Raw coverage still reads 0.600 where C_cap reads 0.501 against a 0.200 reference — geometry stays deceptively high.
  • Outlier pollution (p = 0 → 0.6): D_drop 0.103 → 0.360 while D_collapse falls to 0.005; mirror image of concentration. A single coverage number cannot distinguish these two arms; the decomposition does, with opposite signs.
  • Density shift: D_collapse rises monotonically 0.119 → 0.787; D_drop rises mid-shift (0.317 at t = 0.6) then falls to 0.000 at full shift — exactly the paper's Sec. 4.1.1 behaviour (starved modes become fully uncovered rather than locally under-supported).
  • ρ_g control: ≈ 0 at every concentration stage (|ρ_g| ≤ 0.001) — duplication leaves the pairwise-distance distribution essentially unchanged, so this control is blind to the concentration failure. It does track outlier pollution well as a discrepancy (its proxy MAE 0.054 beats C_cap there — reported honestly; it remains a single undifferentiated number with no dropping/collapse attribution).

Toy-GAN runs (8-ring, 1200 steps, 3 seeds, seed-mean; trained on the pop-os GPU box, CADMA evaluated on the returned sample sets):

run C_cap Coverage D_drop D_collapse
covered 0.457 0.599 0.141 0.403
collapsed (2 of 8 modes) 0.169 0.178 0.009 0.722

Attribution is correct: the collapsed run's deficit reads as collapse (D_collapse 0.72, D_drop ≈ 0.01 — its two modes are over-populated, the other six are fully uncovered and so lower C_cap directly). The covered short-budget GAN is itself fairly concentrated (D_collapse 0.40), which the decomposition surfaces honestly instead of hiding inside a single 0.60 coverage number.

Verdict

PROVEN at toy scale, with one honest qualification. Claim (i): C_cap beats Naeem coverage (raw and regularized) on MAE on all three constructions, across 3 seeds, at the chosen k = 5. Claim (ii): the decomposition separates concentration from support loss with opposite-signed diagnostics, and attributes the real collapsed GAN run to collapse. Qualification: absolute MAE at N = 1400 is dominated by the finite-N matching deficit and degrades for the concentration arm as k grows — the paper's near-zero MAEs are an N = 25000 phenomenon; the comparative and diagnostic claims are what transfer to toy scale.

How to run

python -m pytest tests/test_cadma_metrics.py -q        # full study (GAN fixture trains 6 short runs)
python tests/test_cadma_metrics.py                     # print the study tables, no assertions

Local verification: the 8 non-GAN tests pass in this sandbox (90 s). The GAN fixture's protocol (identical seeds, steps, architecture, eval size) was verified on the pop-os GPU box with margins ≈ 2× the asserted gaps; the in-sandbox CPU training run exceeded this VM's contended-CPU window, so the GAN test itself was not re-run to completion locally.

…12243) toy study

Adds cadma_scores to lib/toy_metrics.py (C_cap / D_drop / D_collapse via
maximum bipartite matching under Eq. 10 regularized real neighbourhoods)
and a study test proving C_cap tracks constructed reference trends with
lower MAE than Naeem coverage on density shift, intramode concentration
and outlier pollution, and that the decomposition separates collapse
from dropping where a single coverage number (and rho_g) cannot.
Research note; nothing merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant