Spec 006 T103a: the analysis sizes from the graph

specs/006-content-service-to-node/research.md

@@ -1219,28 +1219,33 @@ merge. The full form is graph-core's event hierarchy (node already parses it,
 `(main resource, identifier)` pairs of the physical entities its reactions
 reach, its subpathways' included; `totalEntityAndInteractors` adds each
 interaction partner's identifier under the target's resource. Computed from the
-graph, `totalEntity` matched Java for every pathway of five species (3,067
-human); `totalEntityAndInteractors` matched four species exactly and 2,988 of
-3,067 human pathways, the other 79 one or two higher. analysis-core's
+graph (`analysis-sizes.ts`, T103a), `totalEntity` matches Java for every
+pathway of all 16 species Java has a hierarchy for (3,067 human tree nodes);
+`totalEntityAndInteractors` matches 14 species exactly. The others are 79 human
+tree nodes (76 pathways), each 1 to 6 higher in node (50 by one, 14 by two, 12
+by three to six), and 8 yeast pathways, each 1 higher. analysis-core's
 InteractorsBuilder keys interactors by upper-cased identifier in one radix map
 that holds their aliases too, so a partner whose identifier is another's alias
-is merged into whichever was made first: hash-map iteration and Neo4j's order
-of equal scores, which the graph cannot reproduce.
-
-**Decision: node computes the sizes from the graph, once per release, and
-the 79 are a recorded difference.** The gap is one or two entities on pathways
-holding hundreds or thousands (Signal Transduction 10,318 against 10,316), far
-below anything a reader of the Reacfoam can see, and the alternative -- a Java
-step at each release reading the data file with analysis-core -- adds a library
-build the project is not set up to keep current. The 79 are listed as an
-expected difference, so the comparison with Java fails on any other; the cause
-is reported to analysis-core (reactome/analysis-core#13, fix in
-reactome/analysis-core#14: InteractorsBuilder.getOrCreate looks a new partner
-up among every interactor's aliases), and once fixed there, Java's
-numbers become node's. The `pathwaysOnly` tree is built from the graph by the
-analysis hierarchy's rules. Rejected: a release-time snapshot of the data file
-(exact, but a Java library step to maintain); reading the totals from an
-analysis of every identifier (`result.json`, 134 MB, a hack).
+is merged into whichever was made first: in human, Q6NSH3 (CT45A5) into P0DMU9
+(CT45A10), which lists it as a secondary identifier; in yeast, P04838 into the
+polyubiquitin P0CG63. Distinct entries in the graph, often near-identical
+paralogues that once shared one.
+
+**Decision: node computes the sizes from the graph, once per species while it
+runs, and the 84 pathways are a recorded difference.** The gap is a handful of
+entities on pathways holding hundreds or thousands (Signal Transduction 10,318
+against 10,316), far below anything a reader of the Reacfoam can see, and the
+alternative -- a Java step at each release reading the data file with
+analysis-core -- adds a library build the project is not set up to keep
+current. The 84 are listed with both values in
+`fixtures/analysis-size-differences.json`, and `analysis-sizes-check.ts` fails
+on any other difference, or on one of them gone. The cause is reported to
+analysis-core (reactome/analysis-core#13), to be fixed with a Reactome release;
+once it is, Java's numbers become node's and the record goes. The
+`pathwaysOnly` tree is built from the graph by the analysis hierarchy's rules.
+Rejected: a release-time snapshot of the data file (exact, but a Java library
+step to maintain); reading the totals from an analysis of every identifier
+(`result.json`, 134 MB, a hack).
 
 **The overlay.** With a token, Java adds `llp`, `entities` and `reactions` to
 each node the stored result names. Node reads them through the analysis

specs/006-content-service-to-node/tasks.md

@@ -107,8 +107,8 @@ These were held in Java because their substance is a Java library or Java numeri
 - [ ] T101 `GET /interactors/psicquic/molecule/{resource}/{acc}/details` and `/summary`, `POST /interactors/psicquic/molecules/{resource}/details` and `/summary`
 - [ ] T102 `POST /interactors/upload/tuple/content`, `/tuple/form`, `/tuple/url`, `POST /interactors/upload/psicquic/url`, `POST /interactors/token/{token}`
 - [ ] T103 `GET /data/eventsHierarchy/{species}`, the Reacfoam's tree (research D25). In this order, a PR each:
-  - [ ] T103a The sizes from the graph, per species, as analysis-core counts them: `totalEntity` and `totalEntityAndInteractors`, importable-only too, computed once per release and cached; checked against Java's live `pathwaysOnly=true` answer for every species, the 79 human `totalEntityAndInteractors` recorded as the expected difference (D25) and nothing else allowed to differ.
-  - [ ] T103b Node answers `pathwaysOnly=true` without a token, the tree built by the analysis hierarchy's rules, byte-identical but for the recorded 79, errors and `Content-Disposition` included; routed in nginx and the proxy, contract type and recordings.
+  - [x] T103a The sizes from the graph, per species, as analysis-core counts them: `totalEntity` and `totalEntityAndInteractors`, computed once per species and cached (`analysis-sizes.ts`); checked against Java's live `pathwaysOnly=true` answer for all 16 species (`analysis-sizes-check.ts`), the 84 pathways whose `totalEntityAndInteractors` differ recorded with both values (D25) and nothing else allowed to differ. The importable-only counts belong to the token overlay, T103c.
+  - [ ] T103b Node answers `pathwaysOnly=true` without a token, the tree built by the analysis hierarchy's rules, byte-identical but for the recorded 84, errors and `Content-Disposition` included; routed in nginx and the proxy, contract type and recordings.
   - [ ] T103c The token overlay, through the analysis gateway (`result.json`, `/token?importableOnly=true`); the diff harness compares a token answer without regard to the order below a hit node, as Java's own order changes.
   - [ ] T103d The full form: graph-core's event hierarchy (`parseHierarchy`) with the sizes by stId.
 

tools/content-node/analysis-sizes-check.ts

@@ -0,0 +1,101 @@
+/**
+ * Node's analysis sizes against Java's (spec 006 T103a, research D25): for
+ * every species with a top-level pathway in the graph -- those Java has a
+ * hierarchy for, each of which must answer -- each pathway's
+ * totalEntity and totalEntityAndInteractors in Java's
+ * /data/eventsHierarchy/{taxId}?pathwaysOnly=true, against analysisSizes.
+ *
+ *   JAVA_CONTENT_SERVICE=http://localhost:8080 node tools/content-node/analysis-sizes-check.ts
+ *
+ * Passes when the only differences are those recorded in
+ * fixtures/analysis-size-differences.json (analysis-core's alias merge,
+ * reactome/analysis-core#13), each with the values recorded; fails on any
+ * other difference, a pathway one side lacks, a species that does not answer,
+ * or a recorded difference gone -- as when analysis-core is fixed, and the
+ * record should go with it. Which interactor analysis-core's merge keeps
+ * depends on its build order, so a rebuilt data file may move a recorded Java
+ * value: the record is then measured again.
+ */
+import { readFileSync } from 'node:fs';
+import { analysisSizes } from './analysis-sizes.ts';
+import { close, read } from './graph.ts';
+
+type JavaNode = {
+  stId: string;
+  totalEntity: number;
+  totalEntityAndInteractors: number;
+  children?: JavaNode[];
+};
+type Recorded = Record<string, Record<string, { java: number; node: number }>>;
+
+const JAVA = process.env['JAVA_CONTENT_SERVICE'] ?? 'http://localhost:8080';
+const recorded = (
+  JSON.parse(
+    readFileSync(new URL('./fixtures/analysis-size-differences.json', import.meta.url), 'utf8')
+  ) as { differences: Recorded }
+).differences;
+
+const problems: string[] = [];
+const species = await read<{ taxId: string; name: string }>(
+  `MATCH (:TopLevelPathway)-[:species]->(s:Species)
+   RETURN DISTINCT s.taxId AS taxId, s.displayName AS name ORDER BY name`
+);
+let checked = 0;
+for (const { taxId, name: displayName } of species) {
+  const answer = await fetch(
+    `${JAVA}/ContentService/data/eventsHierarchy/${taxId}?pathwaysOnly=true`
+  );
+  if (answer.status !== 200) {
+    problems.push(`${taxId} ${displayName}: Java answered ${answer.status}`);
+    continue;
+  }
+  const tree = (await answer.json()) as JavaNode[];
+  const sizes = await analysisSizes(taxId);
+  const expected = recorded[taxId] ?? {};
+  const seen = new Set<string>();
+  const walk = (node: JavaNode) => {
+    const mine = sizes.get(node.stId);
+    if (!mine) problems.push(`${taxId} ${node.stId}: in Java's tree, not node's`);
+    else {
+      if (mine.totalEntity !== node.totalEntity)
+        problems.push(
+          `${taxId} ${node.stId} totalEntity: Java ${node.totalEntity}, node ${mine.totalEntity}`
+        );
+      const record = expected[node.stId];
+      const java = node.totalEntityAndInteractors;
+      const ours = mine.totalEntityAndInteractors;
+      if (record) {
+        seen.add(node.stId);
+        if (record.java !== java || record.node !== ours)
+          problems.push(
+            `${taxId} ${node.stId} totalEntityAndInteractors: recorded Java ${record.java}, node ${record.node}; now Java ${java}, node ${ours}`
+          );
+      } else if (java !== ours)
+        problems.push(
+          `${taxId} ${node.stId} totalEntityAndInteractors: Java ${java}, node ${ours}`
+        );
+    }
+    for (const child of node.children ?? []) walk(child);
+  };
+  for (const node of tree) walk(node);
+  const javas = new Set<string>();
+  const collect = (node: JavaNode) => {
+    javas.add(node.stId);
+    for (const child of node.children ?? []) collect(child);
+  };
+  for (const node of tree) collect(node);
+  for (const stId of sizes.keys())
+    if (!javas.has(stId)) problems.push(`${taxId} ${stId}: in node's tree, not Java's`);
+  for (const stId of Object.keys(expected))
+    if (!seen.has(stId)) problems.push(`${taxId} ${stId}: recorded, but not in Java's tree`);
+  checked++;
+  console.log(`${taxId} ${displayName}: ${sizes.size} pathways`);
+}
+for (const taxId of Object.keys(recorded))
+  if (!species.some((s) => s.taxId === taxId))
+    problems.push(`${taxId}: recorded, but no top-level pathway`);
+await close();
+
+if (species.length === 0) problems.push('no species with a top-level pathway: is the graph there?');
+console.log(problems.length ? problems.join('\n') : `${checked} species: as recorded.`);
+process.exitCode = problems.length ? 1 : 0;

tools/content-node/analysis-sizes.spec.ts

@@ -0,0 +1,136 @@
+// @vitest-environment node
+//
+// The analysis sizes (spec 006 T103a, research D25) over stand-in graph rows:
+// what a pathway counts, how its subpathways add in, and which resources
+// count. The sizes from the real graph are checked against Java's by
+// analysis-sizes-check.ts, every pathway of every species.
+import { describe, expect, it } from 'vitest';
+import { computeSizes, countedResource } from './analysis-sizes.ts';
+import type { SizeInputs } from './analysis-sizes.ts';
+
+/** A top-level pathway with two subpathways, one of them under both. */
+const tree: SizeInputs['tree'] = [
+  { parent: 'TOP', child: null },
+  { parent: 'TOP', child: 'A' },
+  { parent: 'TOP', child: 'B' },
+  { parent: 'A', child: 'B' },
+];
+
+const sizes = (inputs: Partial<SizeInputs>) =>
+  Object.fromEntries(computeSizes({ tree, entities: [], targets: [], partners: [], ...inputs }));
+
+describe('analysis sizes', () => {
+  it('counts each identifier once per resource, a pathway its subpathways too', () => {
+    const got = sizes({
+      entities: [
+        {
+          pathway: 'A',
+          ids: [
+            ['UniProt', 'P1'],
+            ['ChEBI', '15422'],
+          ],
+        },
+        // B is under A and under TOP: its identifiers count in both, once.
+        {
+          pathway: 'B',
+          ids: [
+            ['UniProt', 'P1'],
+            ['UniProt', 'P2'],
+          ],
+        },
+      ],
+    });
+    expect(got).toEqual({
+      TOP: { totalEntity: 3, totalEntityAndInteractors: 3 },
+      A: { totalEntity: 3, totalEntityAndInteractors: 3 },
+      B: { totalEntity: 2, totalEntityAndInteractors: 2 },
+    });
+  });
+
+  it('gives a pathway with no subpathways, and one with nothing in it, a size', () => {
+    expect(Object.keys(sizes({})).sort()).toEqual(['A', 'B', 'TOP']);
+    expect(sizes({})['B']).toEqual({ totalEntity: 0, totalEntityAndInteractors: 0 });
+  });
+
+  it("adds a target's partners under the target's resource", () => {
+    const got = sizes({
+      entities: [{ pathway: 'B', ids: [['UniProt', 'P1']] }],
+      // A target in a resource that does not count brings no partners.
+      targets: [
+        {
+          pathway: 'B',
+          ids: [
+            ['UniProtKB', 'P1'],
+            ['UCSC', 'U1'],
+          ],
+        },
+      ],
+      // P1 is its own partner, and already counted; Q9 is new.
+      partners: [
+        { acc: 'P1', partners: ['P1', 'Q9'] },
+        { acc: 'U1', partners: ['Z1'] },
+      ],
+    });
+    expect(got['B']).toEqual({ totalEntity: 1, totalEntityAndInteractors: 2 });
+    expect(got['TOP']).toEqual({ totalEntity: 1, totalEntityAndInteractors: 2 });
+  });
+
+  it('keeps the same identifier in two resources apart', () => {
+    const got = sizes({
+      entities: [
+        {
+          pathway: 'A',
+          ids: [
+            ['UniProt', 'X'],
+            ['ENSEMBL', 'X'],
+          ],
+        },
+      ],
+    });
+    expect(got['A']?.totalEntity).toBe(2);
+  });
+
+  it('counts only the resources ResourceFactory names, under its equivalences', () => {
+    expect(
+      [
+        'UniProt',
+        'UniProtKB',
+        'GeneCards',
+        'UCSC human',
+        'PRO',
+        'ChEBI',
+        'IUPHAR',
+        'NCBI Nucleotide',
+      ].map(countedResource)
+    ).toEqual([
+      'UNIPROT',
+      'UNIPROT',
+      'UNIPROT',
+      'UNIPROT',
+      'UNIPROT',
+      'CHEBI',
+      'IUPHAR',
+      'NCBI_NUCLEOTIDE',
+    ]);
+    expect(['NCBI Protein', 'EMBL', 'COMPOUND', 'PubChem Compound'].map(countedResource)).toEqual([
+      'NCBI_PROTEIN',
+      'EMBL',
+      'COMPOUND',
+      'PUBCHEM_COMPOUND',
+    ]);
+    for (const other of ['Guide to Pharmacology - Ligands', 'UCSC', 'BioGPS Gene', 'Reactome'])
+      expect(countedResource(other)).toBeUndefined();
+    const got = sizes({
+      entities: [
+        {
+          pathway: 'A',
+          ids: [
+            ['UCSC', 'P1'],
+            ['UniProt', 'P1'],
+          ],
+        },
+      ],
+    });
+    expect(got['A']?.totalEntity).toBe(1);
+  });
+});