Skip to content

Paper watch: EmbeddGAN salvage — Gini distance correlation as an evaluation metric - #559

Open
mikkel wants to merge 1 commit into
masterfrom
paper-watch/gini-distance-metric
Open

mikkel wants to merge 1 commit into
masterfrom
paper-watch/gini-distance-metric

Conversation

@mikkel

@mikkel mikkel commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Paper-watch research note — EmbeddGAN salvage: rho_g as an evaluation metric

Paper: "EmbeddGAN: A Novel GAN Framework Using an Embedding Network and
Gini Distance Correlation" — https://arxiv.org/abs/2609.22508

Why only the metric is salvaged. EmbeddGAN's framework replaces the
discriminator with an embedding network trained against the Gini distance
correlation — a replacement for the ParticleGAN formulation, out of scope
for the paper watch. The salvage is rho_g itself, computed on raw samples,
as a distribution-level evaluation metric. Nothing in the training
formulation changes; this PR adds two metrics to lib/toy_metrics.py and
one study test.

The statistic. For equal-size real/fake sets A, B of size m:

gCov(A,B) = (1/m^2) sum_ij ||a_i - b_j||
            - (1/(2m(m-1))) sum_{i!=j} [||a_i - a_j|| + ||b_i - b_j||]
Delta(X)  = (1/(2m)^2) sum over all pairs in X = A u B of ||x_i - x_j||
rho_g     = gCov / Delta

Zero iff the two distributions agree. Note, found while implementing: the
paper states rho_g in [0, 1], but under this normalization the disjoint
point-mass limit is 2, and finite-sample values for near-identical
distributions can be marginally negative. The implementation returns the
statistic unclamped and documents this.

Study design (tests/test_gini_distance_metric.py). Plain MLP GAN
(repo SimpleMLPGenerator/SimpleMLPDiscriminator) on an 8-Gaussian ring
(std 0.25, radius 2), 3 seeds x 2 regimes, 1600 steps, checkpoints every
200 steps logging rho_g, the run's own critic loss, the critic loss
re-evaluated against the full mixture, and Naeem coverage (k=5, also added
to lib/toy_metrics.py):

  • covered — trained on all 8 modes;
  • collapsed — trained on modes {0, 1} only, evaluated against the full
    mixture. Ground truth is by construction, never by the metric: this is
    exactly the blindness being measured, since the collapsed arm's critic
    equilibrates on its own training distribution.

Calibration anchors: real-vs-real gives rho_g ~ 0.000 and coverage 0.97;
a perfect 2-mode generator against the full mixture gives rho_g 0.38 and
coverage 0.26.

Iteration log (the gauntlet).

  • Round 1 — construction invalid. std 0.12, lr 2e-3, 800 steps,
    checkpoints every 100. The covered arm itself collapsed (final
    coverage 0.02-0.16 across checkpoints, rho_g ~ 0.65): with modes that
    sharp, a plain GAN smears/collapses and Naeem balls (5th-NN radius
    ~0.02) measure sharpness more than presence. Claim untestable as built.
  • Round 2 — one refinement: toy width + training config. std 0.25,
    lr 1e-3, 1600 steps, checkpoints every 200 (calibration probes for this
    single change: lr-only and steps-only variants stayed collapsed or
    smeared; EMA-generator eval was tried and rejected — it averaged the
    orbit into the gaps, coverage 0.0). Claim proven across 3 seeds:
final checkpoint (step 1600) covered (mean) collapsed (mean)
rho_g 0.0026 0.390
Naeem coverage 0.731 0.148
critic loss (own distribution) 1.341 1.387 (= 2 ln 2 equilibrium)
  • Separation (Cohen's d, final checkpoint): rho_g 11.14 vs critic
    loss 2.95 (critic loss re-evaluated on the full mixture: 1.93).
  • Tracking (Spearman vs true coverage, all 48 checkpoints pooled):
    rho_g -0.896 vs critic loss -0.408 (full-mixture critic: -0.747).
  • The detail that makes the point: in the covered arm the critic loss
    rises over training (1.09 -> 1.35) while coverage improves, and in the
    collapsed arm it sits at the exact fooled-equilibrium value 1.386,
    indistinguishable from a healthy run. The assertions in the test encode
    exactly these margins (d_rho_g >= d_critic and d_rho_g > 2;
    Spearman rho_g < -0.7 and |rho_g| >= |critic|; construction gap > 0.3).

Verdict: PROVEN in 2 iterations. rho_g on raw outputs separates
collapsed from covered runs ~4x more strongly than the critic loss and
tracks true coverage far more closely. It earns a place as a cheap
reference-based audit metric alongside the existing toy metrics.

Seams. No repo seam required fixing for this study. One environment
note: tests/test_toy_metrics.py::test_exact_transport_dependency_and_known_translation
fails in this sandbox with ModuleNotFoundError: No module named 'ot'
(POT not installed) — pre-existing, unrelated to this diff (the other 4
tests in that file pass).

How to run.

python -m pytest tests/test_gini_distance_metric.py -q
python tests/test_gini_distance_metric.py   # prints the full study table

The study is CPU-only; ~3 min uncontended (longer on a loaded box).
Nothing here is merged by the author — PR for review only.

…uation metric

Adds gini_distance_correlation and naeem_coverage to lib/toy_metrics.py
and a study test proving rho_g separates collapsed from covered toy GAN
runs (Cohen's d 11.1 vs critic loss 2.9) and tracks true coverage
(Spearman -0.90 vs -0.41) across 3 seeds x 2 regimes. Research note;
nothing merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant