Repository navigation
Conversation
…12243) toy study Adds cadma_scores to lib/toy_metrics.py (C_cap / D_drop / D_collapse via maximum bipartite matching under Eq. 10 regularized real neighbourhoods) and a study test proving C_cap tracks constructed reference trends with lower MAE than Naeem coverage on density shift, intramode concentration and outlier pollution, and that the decomposition separates collapse from dropping where a single coverage number (and rho_g) cannot. Research note; nothing merged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Paper watch, cycle 2 — CADMA coverage decomposition
Paper: "CADMA: Capacity-Aware Coverage Decomposition for Generative Model Assessment" (Meng et al.), arXiv:2611.12243 — https://arxiv.org/abs/2611.12243 (Sections 3 and 4.1 read before building). Research note; nothing merges — review only.
What this adds
lib/toy_metrics.py:cadma_scores(real, fake, k=5, regularize=True).tests/test_cadma_metrics.py: the study below (constructions + k-sensitivity + real toy-GAN runs), with local equivalent implementations of the two control metrics, since master does not carry cycle 1'snaeem_coverage/gini_distance_correlation(they live onpaper-watch/gini-distance-metric, PR Paper watch: EmbeddGAN salvage — Gini distance correlation as an evaluation metric #559; branches deliberately not merged).Formulas (paper Sec. 3)
Real set R (size N), generated set F (size M = N).
rho_i= distance fromr_ito its k-th nearest neighbour within R; radius regularization (Eq. 10)eps_i = min(rho_i, mean(rho)). Admissible matrixA_ij = 1[ ||r_i - f_j|| <= eps_i ];Z*= maximum-cardinality bipartite matching under A (Hopcroft–Karp viascipy.sparse.csgraph.maximum_bipartite_matching),N_mat = |Z*|.C_cap = N_mat / N(Eq. 7)D_drop = Coverage - C_cap, Coverage = Naeem coverage on the same eps radii (Eq. 8)D_collapse = (M_cov - N_mat) / N,M_cov= # generated samples inside ≥ 1 real neighbourhood (Eq. 9)So
Coverage = C_cap + D_dropexactly. Fully uncovered real samples lowerC_capdirectly and count in neither diagnostic (paper footnote 1) — this matters for reading the toy-GAN result below.Study design (ground truth by construction, ≥ 3 seeds wherever sampling noise exists)
7-mode Gaussian mixture (ring radius 3.0, σ = 0.35), N = M = 1400, seed-mean over 3 seeds.
Controls: Naeem coverage with raw (uncapped) radii, coverage on CADMA's regularized radii, and the cycle-1 Gini distance correlation ρ_g. ρ_g is a discrepancy (0 = identical), not a coverage estimate; its "MAE" below is for the proxy 1 − ρ_g and is labelled as such.
Plus: CADMA applied to real toy-GAN runs (cycle-1 style 8-Gaussian ring, plain MLP GAN, 1200 steps, 3 seeds) — a covered run (trained on all 8 modes) and a collapsed run (trained on modes {0, 1}, evaluated against the full mixture).
k choice + sensitivity
Paper: k = 70 at N = 25000 (k/N = 0.0028). Here N = 1400 and k = 5 (k/N = 0.0036), which also matches the incumbent Naeem control's k so neighbourhood orders are equal. Sensitivity grid (3 seeds, seed-mean C_cap MAE; D_collapse rise = concentration f = 1 → 0.2):
Density/outlier tracking improves monotonically in k, but concentration detection degrades monotonically — each distinct prototype can support ~k matches, so duplication hides as k grows. k = 5 is the smallest k at which C_cap beats both coverage variants on all three constructions, and where the decomposition is most informative. The test encodes the k ∈ {3, 10} subset of this trade-off.
Iteration log (gauntlet, 4 rounds — one lever per round: k)
Final results (k = 5, seed-mean MAE vs reference)
For scale, the paper's headline (their Table 1, N = 25000, CIFAR-10 + Gaussian): C_cap average MAE 0.027 vs Coverage 0.209, Recall 0.213. Our toy-scale C_cap MAEs are larger in absolute terms (finite-N matching deficit: C_cap ≈ 0.81 for two independent draws of the same mixture at N = 1400, k = 5), but the comparative pattern reproduces.
Diagnostic separation (claim ii):
Toy-GAN runs (8-ring, 1200 steps, 3 seeds, seed-mean; trained on the pop-os GPU box, CADMA evaluated on the returned sample sets):
Attribution is correct: the collapsed run's deficit reads as collapse (D_collapse 0.72, D_drop ≈ 0.01 — its two modes are over-populated, the other six are fully uncovered and so lower C_cap directly). The covered short-budget GAN is itself fairly concentrated (D_collapse 0.40), which the decomposition surfaces honestly instead of hiding inside a single 0.60 coverage number.
Verdict
PROVEN at toy scale, with one honest qualification. Claim (i): C_cap beats Naeem coverage (raw and regularized) on MAE on all three constructions, across 3 seeds, at the chosen k = 5. Claim (ii): the decomposition separates concentration from support loss with opposite-signed diagnostics, and attributes the real collapsed GAN run to collapse. Qualification: absolute MAE at N = 1400 is dominated by the finite-N matching deficit and degrades for the concentration arm as k grows — the paper's near-zero MAEs are an N = 25000 phenomenon; the comparative and diagnostic claims are what transfer to toy scale.
How to run
Local verification: the 8 non-GAN tests pass in this sandbox (90 s). The GAN fixture's protocol (identical seeds, steps, architecture, eval size) was verified on the pop-os GPU box with margins ≈ 2× the asserted gaps; the in-sandbox CPU training run exceeded this VM's contended-CPU window, so the GAN test itself was not re-run to completion locally.