Repository navigation
Conversation
…uation metric Adds gini_distance_correlation and naeem_coverage to lib/toy_metrics.py and a study test proving rho_g separates collapsed from covered toy GAN runs (Cohen's d 11.1 vs critic loss 2.9) and tracks true coverage (Spearman -0.90 vs -0.41) across 3 seeds x 2 regimes. Research note; nothing merged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Paper-watch research note — EmbeddGAN salvage: rho_g as an evaluation metric
Paper: "EmbeddGAN: A Novel GAN Framework Using an Embedding Network and
Gini Distance Correlation" — https://arxiv.org/abs/2609.22508
Why only the metric is salvaged. EmbeddGAN's framework replaces the
discriminator with an embedding network trained against the Gini distance
correlation — a replacement for the ParticleGAN formulation, out of scope
for the paper watch. The salvage is rho_g itself, computed on raw samples,
as a distribution-level evaluation metric. Nothing in the training
formulation changes; this PR adds two metrics to
lib/toy_metrics.pyandone study test.
The statistic. For equal-size real/fake sets A, B of size m:
Zero iff the two distributions agree. Note, found while implementing: the
paper states rho_g in [0, 1], but under this normalization the disjoint
point-mass limit is 2, and finite-sample values for near-identical
distributions can be marginally negative. The implementation returns the
statistic unclamped and documents this.
Study design (
tests/test_gini_distance_metric.py). Plain MLP GAN(repo
SimpleMLPGenerator/SimpleMLPDiscriminator) on an 8-Gaussian ring(std 0.25, radius 2), 3 seeds x 2 regimes, 1600 steps, checkpoints every
200 steps logging rho_g, the run's own critic loss, the critic loss
re-evaluated against the full mixture, and Naeem coverage (k=5, also added
to
lib/toy_metrics.py):mixture. Ground truth is by construction, never by the metric: this is
exactly the blindness being measured, since the collapsed arm's critic
equilibrates on its own training distribution.
Calibration anchors: real-vs-real gives rho_g ~ 0.000 and coverage 0.97;
a perfect 2-mode generator against the full mixture gives rho_g 0.38 and
coverage 0.26.
Iteration log (the gauntlet).
checkpoints every 100. The covered arm itself collapsed (final
coverage 0.02-0.16 across checkpoints, rho_g ~ 0.65): with modes that
sharp, a plain GAN smears/collapses and Naeem balls (5th-NN radius
~0.02) measure sharpness more than presence. Claim untestable as built.
lr 1e-3, 1600 steps, checkpoints every 200 (calibration probes for this
single change: lr-only and steps-only variants stayed collapsed or
smeared; EMA-generator eval was tried and rejected — it averaged the
orbit into the gaps, coverage 0.0). Claim proven across 3 seeds:
loss 2.95 (critic loss re-evaluated on the full mixture: 1.93).
rho_g -0.896 vs critic loss -0.408 (full-mixture critic: -0.747).
rises over training (1.09 -> 1.35) while coverage improves, and in the
collapsed arm it sits at the exact fooled-equilibrium value 1.386,
indistinguishable from a healthy run. The assertions in the test encode
exactly these margins (d_rho_g >= d_critic and d_rho_g > 2;
Spearman rho_g < -0.7 and |rho_g| >= |critic|; construction gap > 0.3).
Verdict: PROVEN in 2 iterations. rho_g on raw outputs separates
collapsed from covered runs ~4x more strongly than the critic loss and
tracks true coverage far more closely. It earns a place as a cheap
reference-based audit metric alongside the existing toy metrics.
Seams. No repo seam required fixing for this study. One environment
note:
tests/test_toy_metrics.py::test_exact_transport_dependency_and_known_translationfails in this sandbox with
ModuleNotFoundError: No module named 'ot'(POT not installed) — pre-existing, unrelated to this diff (the other 4
tests in that file pass).
How to run.
The study is CPU-only; ~3 min uncontended (longer on a loaded box).
Nothing here is merged by the author — PR for review only.