From c8e6361d30f6ecc0b8a70edcb6c44eb3ac816f57 Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Wed, 11 Nov 2026 23:48:30 +0000 Subject: [PATCH 1/2] Spec 006 T103a: the analysis sizes from the graph analysis-sizes.ts computes each pathway's totalEntity and totalEntityAndInteractors as analysis-core does, from the graph, once per species while the service runs: the tree from each top-level pathway through hasEvent, the identifiers its reactions reach by EntitiesBuilder's relationships, the interaction partners of InteractorsBuilder's targets, counted per ResourceFactory resource, subpathways included. Against Java's live eventsHierarchy for all 16 species with a hierarchy: totalEntity matches every pathway. totalEntityAndInteractors matches but for 84 pathways (76 human, 1 to 6 higher; 8 yeast, 1 higher), where analysis-core merges a partner into another interactor whose alias it is (reactome/analysis-core#13). Those are recorded with both values, and analysis-sizes-check.ts fails on any other difference or one of them gone (seen failing with one removed). Unit tests over stand-in rows, each seen failing under a mutation of the code. Research D25 corrected: the human differences reach six, not two, and yeast has eight. Co-Authored-By: Claude Opus 5.5 --- specs/006-content-service-to-node/research.md | 45 ++- specs/006-content-service-to-node/tasks.md | 4 +- tools/content-node/analysis-sizes-check.ts | 87 +++++ tools/content-node/analysis-sizes.spec.ts | 124 +++++++ tools/content-node/analysis-sizes.ts | 195 ++++++++++ .../fixtures/analysis-size-differences.json | 345 ++++++++++++++++++ tools/load.spec.ts | 1 + 7 files changed, 779 insertions(+), 22 deletions(-) create mode 100644 tools/content-node/analysis-sizes-check.ts create mode 100644 tools/content-node/analysis-sizes.spec.ts create mode 100644 tools/content-node/analysis-sizes.ts create mode 100644 tools/content-node/fixtures/analysis-size-differences.json diff --git a/specs/006-content-service-to-node/research.md b/specs/006-content-service-to-node/research.md index d47cf18e..2cf04a30 100644 --- a/specs/006-content-service-to-node/research.md +++ b/specs/006-content-service-to-node/research.md @@ -1219,28 +1219,33 @@ merge. The full form is graph-core's event hierarchy (node already parses it, `(main resource, identifier)` pairs of the physical entities its reactions reach, its subpathways' included; `totalEntityAndInteractors` adds each interaction partner's identifier under the target's resource. Computed from the -graph, `totalEntity` matched Java for every pathway of five species (3,067 -human); `totalEntityAndInteractors` matched four species exactly and 2,988 of -3,067 human pathways, the other 79 one or two higher. analysis-core's +graph (`analysis-sizes.ts`, T103a), `totalEntity` matches Java for every +pathway of all 16 species Java has a hierarchy for (3,067 human tree nodes); +`totalEntityAndInteractors` matches 14 species exactly. The others are 79 human +tree nodes (76 pathways), each 1 to 6 higher in node (50 by one, 14 by two, 12 +by three to six), and 8 yeast pathways, each 1 higher. analysis-core's InteractorsBuilder keys interactors by upper-cased identifier in one radix map that holds their aliases too, so a partner whose identifier is another's alias -is merged into whichever was made first: hash-map iteration and Neo4j's order -of equal scores, which the graph cannot reproduce. - -**Decision: node computes the sizes from the graph, once per release, and -the 79 are a recorded difference.** The gap is one or two entities on pathways -holding hundreds or thousands (Signal Transduction 10,318 against 10,316), far -below anything a reader of the Reacfoam can see, and the alternative -- a Java -step at each release reading the data file with analysis-core -- adds a library -build the project is not set up to keep current. The 79 are listed as an -expected difference, so the comparison with Java fails on any other; the cause -is reported to analysis-core (reactome/analysis-core#13, fix in -reactome/analysis-core#14: InteractorsBuilder.getOrCreate looks a new partner -up among every interactor's aliases), and once fixed there, Java's -numbers become node's. The `pathwaysOnly` tree is built from the graph by the -analysis hierarchy's rules. Rejected: a release-time snapshot of the data file -(exact, but a Java library step to maintain); reading the totals from an -analysis of every identifier (`result.json`, 134 MB, a hack). +is merged into whichever was made first: in human, Q6NSH3 (CT45A5) into P0DMU9 +(CT45A10), which lists it as a secondary identifier; in yeast, P04838 into the +polyubiquitin P0CG63. Distinct entries in the graph, often near-identical +paralogues that once shared one. + +**Decision: node computes the sizes from the graph, once per species while it +runs, and the 84 pathways are a recorded difference.** The gap is a handful of +entities on pathways holding hundreds or thousands (Signal Transduction 10,318 +against 10,316), far below anything a reader of the Reacfoam can see, and the +alternative -- a Java step at each release reading the data file with +analysis-core -- adds a library build the project is not set up to keep +current. The 84 are listed with both values in +`fixtures/analysis-size-differences.json`, and `analysis-sizes-check.ts` fails +on any other difference, or on one of them gone. The cause is reported to +analysis-core (reactome/analysis-core#13), to be fixed with a Reactome release; +once it is, Java's numbers become node's and the record goes. The +`pathwaysOnly` tree is built from the graph by the analysis hierarchy's rules. +Rejected: a release-time snapshot of the data file (exact, but a Java library +step to maintain); reading the totals from an analysis of every identifier +(`result.json`, 134 MB, a hack). **The overlay.** With a token, Java adds `llp`, `entities` and `reactions` to each node the stored result names. Node reads them through the analysis diff --git a/specs/006-content-service-to-node/tasks.md b/specs/006-content-service-to-node/tasks.md index 0dd726fa..4af161c1 100644 --- a/specs/006-content-service-to-node/tasks.md +++ b/specs/006-content-service-to-node/tasks.md @@ -107,8 +107,8 @@ These were held in Java because their substance is a Java library or Java numeri - [ ] T101 `GET /interactors/psicquic/molecule/{resource}/{acc}/details` and `/summary`, `POST /interactors/psicquic/molecules/{resource}/details` and `/summary` - [ ] T102 `POST /interactors/upload/tuple/content`, `/tuple/form`, `/tuple/url`, `POST /interactors/upload/psicquic/url`, `POST /interactors/token/{token}` - [ ] T103 `GET /data/eventsHierarchy/{species}`, the Reacfoam's tree (research D25). In this order, a PR each: - - [ ] T103a The sizes from the graph, per species, as analysis-core counts them: `totalEntity` and `totalEntityAndInteractors`, importable-only too, computed once per release and cached; checked against Java's live `pathwaysOnly=true` answer for every species, the 79 human `totalEntityAndInteractors` recorded as the expected difference (D25) and nothing else allowed to differ. - - [ ] T103b Node answers `pathwaysOnly=true` without a token, the tree built by the analysis hierarchy's rules, byte-identical but for the recorded 79, errors and `Content-Disposition` included; routed in nginx and the proxy, contract type and recordings. + - [x] T103a The sizes from the graph, per species, as analysis-core counts them: `totalEntity` and `totalEntityAndInteractors`, computed once per species and cached (`analysis-sizes.ts`); checked against Java's live `pathwaysOnly=true` answer for all 16 species (`analysis-sizes-check.ts`), the 84 pathways whose `totalEntityAndInteractors` differ recorded with both values (D25) and nothing else allowed to differ. The importable-only counts belong to the token overlay, T103c. + - [ ] T103b Node answers `pathwaysOnly=true` without a token, the tree built by the analysis hierarchy's rules, byte-identical but for the recorded 84, errors and `Content-Disposition` included; routed in nginx and the proxy, contract type and recordings. - [ ] T103c The token overlay, through the analysis gateway (`result.json`, `/token?importableOnly=true`); the diff harness compares a token answer without regard to the order below a hit node, as Java's own order changes. - [ ] T103d The full form: graph-core's event hierarchy (`parseHierarchy`) with the sizes by stId. diff --git a/tools/content-node/analysis-sizes-check.ts b/tools/content-node/analysis-sizes-check.ts new file mode 100644 index 00000000..3437007b --- /dev/null +++ b/tools/content-node/analysis-sizes-check.ts @@ -0,0 +1,87 @@ +/** + * Node's analysis sizes against Java's (spec 006 T103a, research D25): for + * every species Java has a pathway hierarchy for, each pathway's + * totalEntity and totalEntityAndInteractors in Java's + * /data/eventsHierarchy/{taxId}?pathwaysOnly=true, against analysisSizes. + * + * JAVA_CONTENT_SERVICE=http://localhost:8080 node tools/content-node/analysis-sizes-check.ts + * + * Passes when the only differences are those recorded in + * fixtures/analysis-size-differences.json (analysis-core's alias merge, + * reactome/analysis-core#13), each with the values recorded; fails on any + * other difference, a pathway one side lacks, or a recorded difference gone + * -- as when analysis-core is fixed, and the record should go with it. + */ +import { readFileSync } from 'node:fs'; +import { analysisSizes } from './analysis-sizes.ts'; +import { close } from './graph.ts'; + +type JavaNode = { + stId: string; + totalEntity: number; + totalEntityAndInteractors: number; + children?: JavaNode[]; +}; +type Recorded = Record>; + +const JAVA = process.env['JAVA_CONTENT_SERVICE'] ?? 'http://localhost:8080'; +const recorded = ( + JSON.parse( + readFileSync(new URL('./fixtures/analysis-size-differences.json', import.meta.url), 'utf8') + ) as { differences: Recorded } +).differences; + +const problems: string[] = []; +const species = (await (await fetch(`${JAVA}/ContentService/data/species/all`)).json()) as { + taxId: string | null; + displayName: string; +}[]; +let checked = 0; +for (const { taxId, displayName } of species) { + if (!taxId) continue; + const answer = await fetch( + `${JAVA}/ContentService/data/eventsHierarchy/${taxId}?pathwaysOnly=true` + ); + // Java has no hierarchy for most species, and answers those with a 500. + if (answer.status !== 200) continue; + const tree = (await answer.json()) as JavaNode[]; + const sizes = await analysisSizes(taxId); + const expected = recorded[taxId] ?? {}; + const seen = new Set(); + const walk = (node: JavaNode) => { + const mine = sizes.get(node.stId); + if (!mine) problems.push(`${taxId} ${node.stId}: in Java's tree, not node's`); + else { + if (mine.totalEntity !== node.totalEntity) + problems.push( + `${taxId} ${node.stId} totalEntity: Java ${node.totalEntity}, node ${mine.totalEntity}` + ); + const record = expected[node.stId]; + const java = node.totalEntityAndInteractors; + const ours = mine.totalEntityAndInteractors; + if (record) { + seen.add(node.stId); + if (record.java !== java || record.node !== ours) + problems.push( + `${taxId} ${node.stId} totalEntityAndInteractors: recorded Java ${record.java}, node ${record.node}; now Java ${java}, node ${ours}` + ); + } else if (java !== ours) + problems.push( + `${taxId} ${node.stId} totalEntityAndInteractors: Java ${java}, node ${ours}` + ); + } + for (const child of node.children ?? []) walk(child); + }; + for (const node of tree) walk(node); + for (const stId of Object.keys(expected)) + if (!seen.has(stId)) problems.push(`${taxId} ${stId}: recorded, but not in Java's tree`); + checked++; + console.log(`${taxId} ${displayName}: ${sizes.size} pathways`); +} +for (const taxId of Object.keys(recorded)) + if (!species.some((s) => s.taxId === taxId)) problems.push(`${taxId}: recorded, no such species`); +await close(); + +if (checked === 0) problems.push('no species answered: is Java running?'); +console.log(problems.length ? problems.join('\n') : `${checked} species: as recorded.`); +process.exitCode = problems.length ? 1 : 0; diff --git a/tools/content-node/analysis-sizes.spec.ts b/tools/content-node/analysis-sizes.spec.ts new file mode 100644 index 00000000..d73d0355 --- /dev/null +++ b/tools/content-node/analysis-sizes.spec.ts @@ -0,0 +1,124 @@ +// @vitest-environment node +// +// The analysis sizes (spec 006 T103a, research D25) over stand-in graph rows: +// what a pathway counts, how its subpathways add in, and which resources +// count. The sizes from the real graph are checked against Java's by +// analysis-sizes-check.ts, every pathway of every species. +import { describe, expect, it } from 'vitest'; +import { computeSizes, countedResource } from './analysis-sizes.ts'; +import type { SizeInputs } from './analysis-sizes.ts'; + +/** A top-level pathway with two subpathways, one of them under both. */ +const tree: SizeInputs['tree'] = [ + { parent: 'TOP', child: null }, + { parent: 'TOP', child: 'A' }, + { parent: 'TOP', child: 'B' }, + { parent: 'A', child: 'B' }, +]; + +const sizes = (inputs: Partial) => + Object.fromEntries(computeSizes({ tree, entities: [], targets: [], partners: [], ...inputs })); + +describe('analysis sizes', () => { + it('counts each identifier once per resource, a pathway its subpathways too', () => { + const got = sizes({ + entities: [ + { + pathway: 'A', + ids: [ + ['UniProt', 'P1'], + ['ChEBI', '15422'], + ], + }, + // B is under A and under TOP: its identifiers count in both, once. + { + pathway: 'B', + ids: [ + ['UniProt', 'P1'], + ['UniProt', 'P2'], + ], + }, + ], + }); + expect(got).toEqual({ + TOP: { totalEntity: 3, totalEntityAndInteractors: 3 }, + A: { totalEntity: 3, totalEntityAndInteractors: 3 }, + B: { totalEntity: 2, totalEntityAndInteractors: 2 }, + }); + }); + + it('gives a pathway with no subpathways, and one with nothing in it, a size', () => { + expect(Object.keys(sizes({})).sort()).toEqual(['A', 'B', 'TOP']); + expect(sizes({})['B']).toEqual({ totalEntity: 0, totalEntityAndInteractors: 0 }); + }); + + it("adds a target's partners under the target's resource", () => { + const got = sizes({ + entities: [{ pathway: 'B', ids: [['UniProt', 'P1']] }], + targets: [{ pathway: 'B', ids: [['UniProtKB', 'P1']] }], + // P1 is its own partner, and already counted; Q9 is new. + partners: [{ acc: 'P1', partners: ['P1', 'Q9'] }], + }); + expect(got['B']).toEqual({ totalEntity: 1, totalEntityAndInteractors: 2 }); + expect(got['TOP']).toEqual({ totalEntity: 1, totalEntityAndInteractors: 2 }); + }); + + it('keeps the same identifier in two resources apart', () => { + const got = sizes({ + entities: [ + { + pathway: 'A', + ids: [ + ['UniProt', 'X'], + ['ENSEMBL', 'X'], + ], + }, + ], + }); + expect(got['A']?.totalEntity).toBe(2); + }); + + it('counts only the resources ResourceFactory names, under its equivalences', () => { + expect( + [ + 'UniProt', + 'UniProtKB', + 'GeneCards', + 'UCSC human', + 'PRO', + 'ChEBI', + 'IUPHAR', + 'NCBI Nucleotide', + ].map(countedResource) + ).toEqual([ + 'UNIPROT', + 'UNIPROT', + 'UNIPROT', + 'UNIPROT', + 'UNIPROT', + 'CHEBI', + 'IUPHAR', + 'NCBI_NUCLEOTIDE', + ]); + expect(['NCBI Protein', 'EMBL', 'COMPOUND', 'PubChem Compound'].map(countedResource)).toEqual([ + 'NCBI_PROTEIN', + 'EMBL', + 'COMPOUND', + 'PUBCHEM_COMPOUND', + ]); + for (const other of ['Guide to Pharmacology - Ligands', 'UCSC', 'BioGPS Gene', 'Reactome']) + expect(countedResource(other)).toBeUndefined(); + const got = sizes({ + entities: [ + { + pathway: 'A', + ids: [ + ['UCSC', 'P1'], + ['UniProt', 'P1'], + ], + }, + ], + }); + expect(got['A']?.totalEntity).toBe(1); + }); +}); diff --git a/tools/content-node/analysis-sizes.ts b/tools/content-node/analysis-sizes.ts new file mode 100644 index 00000000..ac184359 --- /dev/null +++ b/tools/content-node/analysis-sizes.ts @@ -0,0 +1,195 @@ +/** + * Each pathway's size as the AnalysisService's hierarchy holds it (spec 006 + * T103a, research D25): `totalEntity`, the distinct identifiers its reactions + * reach, and `totalEntityAndInteractors`, those with their interaction + * partners, each counted per resource and its subpathways' included. + * /data/eventsHierarchy writes them; Java reads them from the AnalysisService's + * data file, which analysis-core builds from the graph as this does: + * + * - The tree: each top-level pathway of the species, then every Pathway below + * it through hasEvent (HierarchyBuilder.fillBranch). A pathway's size is its + * own reactions' identifiers and its subpathways', so it is the same under + * any parent. + * - Entities: from each reaction a species pathway holds directly, the + * physical entities it reaches with a reference entity, by the relationships + * EntitiesBuilder follows; each one's (resource, variant identifier or + * identifier). + * - Interactors: from the narrower set InteractorsBuilder follows, pathways + * under a top-level pathway only, each target's interaction partners, filed + * under the target's resource. + * - Resources: ResourceFactory's main and auxiliary ones, under its + * equivalences; any other is not counted. + * + * One known difference (research D25): analysis-core merges a partner whose + * identifier is another interactor's alias into that interactor + * (reactome/analysis-core#13), so 79 human pathways' totalEntityAndInteractors + * are one or two lower in Java than here. Computed once per species while the + * process lives: the graph is a release's, and changes only with it. + */ +import { read } from './graph.ts'; + +export interface AnalysisSize { + totalEntity: number; + totalEntityAndInteractors: number; +} + +/** ResourceFactory.MAIN and AUX: the resources a size counts. */ +const COUNTED = new Set([ + 'UNIPROT', + 'ENSEMBL', + 'MIRBASE', + 'CHEBI', + 'IUPHAR', + 'NCBI_NUCLEOTIDE', + 'NCBI_PROTEIN', + 'EMBL', + 'COMPOUND', + 'PUBCHEM_COMPOUND', +]); + +/** ResourceFactory's equivalences that lead to a counted resource. */ +const EQUIVALENT: Record = { + UNIPROTKB: 'UNIPROT', + GENECARDS: 'UNIPROT', + UCSC_HUMAN: 'UNIPROT', + PRO: 'UNIPROT', +}; + +/** + * ResourceFactory.getResource's name for a reference entity's databaseName, + * if it is one a size counts: upper-cased, whitespace to underscores, then + * its equivalence. + */ +export function countedResource(databaseName: string): string | undefined { + const name = databaseName.toUpperCase().replace(/\s/g, '_').trim(); + const resource = EQUIVALENT[name] ?? name; + return COUNTED.has(resource) ? resource : undefined; +} + +/** The species' pathway tree, as parent → child pairs from its top-level pathways. */ +const TREE_QUERY = ` +MATCH (t:TopLevelPathway)-[:species]->(:Species {taxId: $taxId}) +MATCH (t)-[:hasEvent*0..]->(p:Pathway)-[:hasEvent]->(c:Pathway) +RETURN DISTINCT p.stId AS parent, c.stId AS child +UNION +MATCH (t:TopLevelPathway)-[:species]->(:Species {taxId: $taxId}) +RETURN t.stId AS parent, null AS child`; + +/** EntitiesBuilder: each species pathway's own reactions' identifiers. */ +const ENTITIES_QUERY = ` +MATCH (:Species {taxId: $taxId})<-[:species]-(p:Pathway)-[:hasEvent]->(rle:ReactionLikeEvent), + (rle)-[:input|output|catalystActivity|physicalEntity|entityFunctionalStatus|diseaseEntity|regulatedBy|regulator|hasComponent|hasMember|hasCandidate|repeatedUnit|proteinMarker|RNAMarker*]->(:PhysicalEntity)-[:referenceEntity]->(re:ReferenceEntity) +WITH DISTINCT p, re +RETURN p.stId AS pathway, collect(DISTINCT [re.databaseName, coalesce(re.variantIdentifier, re.identifier)]) AS ids`; + +/** InteractorsBuilder's targets: what a pathway's reactions reach that may interact. */ +const TARGETS_QUERY = ` +MATCH (:Species {taxId: $taxId})<-[:species]-(p:Pathway)-[:hasEvent]->(rle:ReactionLikeEvent), + (rle)-[:input|output|catalystActivity|physicalEntity|entityFunctionalStatus|diseaseEntity|regulatedBy|regulator*]->(:PhysicalEntity)-[:referenceEntity]->(re:ReferenceEntity) +WHERE (p:TopLevelPathway) OR (:TopLevelPathway)-[:hasEvent*]->(p) +WITH DISTINCT p, [re.databaseName, coalesce(re.variantIdentifier, re.identifier)] AS target +RETURN p.stId AS pathway, collect(target) AS ids`; + +/** InteractionsService.getInteractions(acc): each target's partners, by its identifier. */ +const PARTNERS_QUERY = ` +UNWIND $accs AS acc +MATCH (t:ReferenceEntity)<-[:interactor]-(:Interaction)-[:interactor]->(o:ReferenceEntity) +WHERE t.variantIdentifier = acc OR (t.variantIdentifier IS NULL AND t.identifier = acc) +RETURN acc, collect(DISTINCT coalesce(o.variantIdentifier, o.identifier)) AS partners`; + +type IdRow = { pathway: string; ids: [string, string][] }; + +/** The tree's rows and the identifier rows, as computeSizes reads them. */ +export interface SizeInputs { + tree: { parent: string; child: string | null }[]; + entities: IdRow[]; + targets: IdRow[]; + partners: { acc: string; partners: string[] }[]; +} + +/** Every pathway's size in the tree the inputs describe, by stable id. */ +export function computeSizes({ tree, entities, targets, partners }: SizeInputs) { + const children = new Map(); + for (const { parent, child } of tree) { + const list = children.get(parent) ?? []; + if (child !== null) { + list.push(child); + // A pathway with no subpathways is only ever a child here. + if (!children.has(child)) children.set(child, []); + } + children.set(parent, list); + } + const partnersOf = new Map(partners.map((row) => [row.acc, row.partners])); + const own = new Map; all: Set }>(); + const ownOf = (pathway: string) => { + let sets = own.get(pathway); + if (!sets) own.set(pathway, (sets = { entities: new Set(), all: new Set() })); + return sets; + }; + for (const { pathway, ids } of entities) { + const sets = ownOf(pathway); + for (const [databaseName, id] of ids) { + const resource = countedResource(databaseName); + if (resource === undefined) continue; + sets.entities.add(`${resource}|${id}`); + sets.all.add(`${resource}|${id}`); + } + } + for (const { pathway, ids } of targets) { + const sets = ownOf(pathway); + for (const [databaseName, id] of ids) { + const resource = countedResource(databaseName); + if (resource === undefined) continue; + for (const partner of partnersOf.get(id) ?? []) sets.all.add(`${resource}|${partner}`); + } + } + + const done = new Map; all: Set }>(); + const total = (pathway: string): { entities: Set; all: Set } => { + const known = done.get(pathway); + if (known) return known; + const mine = own.get(pathway); + const sets = { entities: new Set(mine?.entities), all: new Set(mine?.all) }; + for (const child of children.get(pathway) ?? []) { + const below = total(child); + for (const id of below.entities) sets.entities.add(id); + for (const id of below.all) sets.all.add(id); + } + done.set(pathway, sets); + return sets; + }; + const sizes = new Map(); + for (const pathway of children.keys()) { + const { entities: e, all } = total(pathway); + sizes.set(pathway, { totalEntity: e.size, totalEntityAndInteractors: all.size }); + } + return sizes; +} + +/** The graph's inputs for one species. */ +async function sizeInputs(taxId: string): Promise { + const [tree, entities, targets] = await Promise.all([ + read(TREE_QUERY, { taxId }), + read(ENTITIES_QUERY, { taxId }), + read(TARGETS_QUERY, { taxId }), + ]); + const accs = [...new Set(targets.flatMap((row) => row.ids.map(([, id]) => id)))]; + const partners = await read(PARTNERS_QUERY, { accs }); + return { tree, entities, targets, partners }; +} + +const bySpecies = new Map>>(); + +/** + * Every pathway's size for a species, by taxId, computed once while the + * process lives; a failed computation is tried again on the next call. + */ +export function analysisSizes(taxId: string): Promise> { + let sizes = bySpecies.get(taxId); + if (!sizes) { + sizes = sizeInputs(taxId).then(computeSizes); + sizes.catch(() => bySpecies.delete(taxId)); + bySpecies.set(taxId, sizes); + } + return sizes; +} diff --git a/tools/content-node/fixtures/analysis-size-differences.json b/tools/content-node/fixtures/analysis-size-differences.json new file mode 100644 index 00000000..76e57640 --- /dev/null +++ b/tools/content-node/fixtures/analysis-size-differences.json @@ -0,0 +1,345 @@ +{ + "_about": "Pathways whose totalEntityAndInteractors node computes higher than Java, release 98: analysis-core merges an interaction partner whose identifier is another interactor's alias into it (reactome/analysis-core#13; spec 006 research D25). By taxId, then stId. analysis-sizes-check.ts fails on any other difference, or on one of these gone.", + "differences": { + "4932": { + "R-SCE-199991": { + "java": 214, + "node": 215 + }, + "R-SCE-5653656": { + "java": 214, + "node": 215 + }, + "R-SCE-6791226": { + "java": 204, + "node": 205 + }, + "R-SCE-72312": { + "java": 204, + "node": 205 + }, + "R-SCE-8856825": { + "java": 26, + "node": 27 + }, + "R-SCE-8856828": { + "java": 33, + "node": 34 + }, + "R-SCE-8868773": { + "java": 204, + "node": 205 + }, + "R-SCE-8953854": { + "java": 441, + "node": 442 + } + }, + "9606": { + "R-HSA-1236974": { + "java": 318, + "node": 319 + }, + "R-HSA-1236975": { + "java": 340, + "node": 341 + }, + "R-HSA-1266738": { + "java": 6862, + "node": 6864 + }, + "R-HSA-1280215": { + "java": 5735, + "node": 5736 + }, + "R-HSA-1280218": { + "java": 3712, + "node": 3714 + }, + "R-HSA-1430728": { + "java": 7165, + "node": 7167 + }, + "R-HSA-1502540": { + "java": 169, + "node": 170 + }, + "R-HSA-159236": { + "java": 448, + "node": 451 + }, + "R-HSA-162582": { + "java": 10316, + "node": 10318 + }, + "R-HSA-1643685": { + "java": 9773, + "node": 9776 + }, + "R-HSA-168249": { + "java": 3937, + "node": 3939 + }, + "R-HSA-168256": { + "java": 8802, + "node": 8806 + }, + "R-HSA-170834": { + "java": 1394, + "node": 1395 + }, + "R-HSA-1989781": { + "java": 691, + "node": 692 + }, + "R-HSA-201451": { + "java": 134, + "node": 135 + }, + "R-HSA-212436": { + "java": 6977, + "node": 6980 + }, + "R-HSA-2173789": { + "java": 554, + "node": 555 + }, + "R-HSA-2173793": { + "java": 1026, + "node": 1027 + }, + "R-HSA-2173795": { + "java": 582, + "node": 583 + }, + "R-HSA-2173796": { + "java": 628, + "node": 629 + }, + "R-HSA-2262752": { + "java": 4756, + "node": 4758 + }, + "R-HSA-2682334": { + "java": 816, + "node": 817 + }, + "R-HSA-2990846": { + "java": 2553, + "node": 2554 + }, + "R-HSA-3108214": { + "java": 539, + "node": 540 + }, + "R-HSA-3108232": { + "java": 2515, + "node": 2516 + }, + "R-HSA-373760": { + "java": 1147, + "node": 1148 + }, + "R-HSA-392499": { + "java": 7623, + "node": 7625 + }, + "R-HSA-3928664": { + "java": 548, + "node": 549 + }, + "R-HSA-400206": { + "java": 728, + "node": 729 + }, + "R-HSA-4090294": { + "java": 719, + "node": 720 + }, + "R-HSA-422475": { + "java": 2503, + "node": 2504 + }, + "R-HSA-447043": { + "java": 336, + "node": 337 + }, + "R-HSA-5213460": { + "java": 517, + "node": 518 + }, + "R-HSA-5218859": { + "java": 609, + "node": 610 + }, + "R-HSA-5357801": { + "java": 1948, + "node": 1949 + }, + "R-HSA-556833": { + "java": 3670, + "node": 3671 + }, + "R-HSA-5621481": { + "java": 674, + "node": 676 + }, + "R-HSA-5621575": { + "java": 236, + "node": 237 + }, + "R-HSA-5663202": { + "java": 2660, + "node": 2661 + }, + "R-HSA-5663205": { + "java": 7132, + "node": 7134 + }, + "R-HSA-5675482": { + "java": 511, + "node": 512 + }, + "R-HSA-5688426": { + "java": 1152, + "node": 1153 + }, + "R-HSA-5689880": { + "java": 453, + "node": 454 + }, + "R-HSA-597592": { + "java": 5891, + "node": 5893 + }, + "R-HSA-72165": { + "java": 224, + "node": 225 + }, + "R-HSA-72172": { + "java": 1319, + "node": 1322 + }, + "R-HSA-72202": { + "java": 594, + "node": 597 + }, + "R-HSA-72203": { + "java": 1654, + "node": 1658 + }, + "R-HSA-73857": { + "java": 7187, + "node": 7190 + }, + "R-HSA-74160": { + "java": 7990, + "node": 7993 + }, + "R-HSA-877300": { + "java": 632, + "node": 633 + }, + "R-HSA-8862803": { + "java": 1513, + "node": 1514 + }, + "R-HSA-8863678": { + "java": 1513, + "node": 1514 + }, + "R-HSA-8864260": { + "java": 1524, + "node": 1525 + }, + "R-HSA-8866910": { + "java": 756, + "node": 757 + }, + "R-HSA-8953854": { + "java": 3817, + "node": 3823 + }, + "R-HSA-8953897": { + "java": 5180, + "node": 5182 + }, + "R-HSA-9006936": { + "java": 1570, + "node": 1571 + }, + "R-HSA-913531": { + "java": 2626, + "node": 2627 + }, + "R-HSA-9635486": { + "java": 982, + "node": 983 + }, + "R-HSA-9637628": { + "java": 62, + "node": 63 + }, + "R-HSA-9645723": { + "java": 1619, + "node": 1620 + }, + "R-HSA-9675108": { + "java": 2759, + "node": 2760 + }, + "R-HSA-9679506": { + "java": 2877, + "node": 2879 + }, + "R-HSA-9694516": { + "java": 1932, + "node": 1933 + }, + "R-HSA-9707564": { + "java": 321, + "node": 322 + }, + "R-HSA-9711123": { + "java": 1585, + "node": 1586 + }, + "R-HSA-9734009": { + "java": 1515, + "node": 1516 + }, + "R-HSA-9772573": { + "java": 985, + "node": 986 + }, + "R-HSA-9824439": { + "java": 1796, + "node": 1797 + }, + "R-HSA-9824446": { + "java": 5747, + "node": 5752 + }, + "R-HSA-983169": { + "java": 719, + "node": 720 + }, + "R-HSA-983170": { + "java": 235, + "node": 236 + }, + "R-HSA-9839923": { + "java": 2672, + "node": 2675 + }, + "R-HSA-9918481": { + "java": 1558, + "node": 1560 + }, + "R-HSA-9920588": { + "java": 344, + "node": 346 + } + } + } +} diff --git a/tools/load.spec.ts b/tools/load.spec.ts index 7ce7bf3e..4f1daefb 100644 --- a/tools/load.spec.ts +++ b/tools/load.spec.ts @@ -34,6 +34,7 @@ const SERVICES = [ const RUN_ON_LOAD = new Set([ 'tools/analysis-types/record', 'tools/content-node/diff', + 'tools/content-node/analysis-sizes-check', 'tools/content-node/people-name-check', 'tools/content-node/reaction-layout-check', 'tools/content-node/snapshot-case', From 1249db072792166ab6f52fbb637765467fc52d64 Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Wed, 11 Nov 2026 23:58:02 +0000 Subject: [PATCH 2/2] Spec 006 T103a: review points on the analysis sizes - The check required no answer from Java: a species answering an error was skipped, its records unchecked, and a pathway only node has was never reported. It now takes the species from the graph's top-level pathways, fails on any that does not answer (seen failing with Java answering 404s), and compares both sides' pathways. - Entities and targets are read for the tree's pathways whatever their own species, as analysis-core attaches a pathway's counts wherever it sits. None differs today; all 16 species still as recorded. - The record notes that a rebuilt data file may move a Java value, since which interactor analysis-core's merge keeps depends on build order. - A test for a target in a resource that does not count (seen failing with the skip removed); the module comment's figures corrected. Co-Authored-By: Claude Opus 5.5 --- tools/content-node/analysis-sizes-check.ts | 42 ++++++++++++------- tools/content-node/analysis-sizes.spec.ts | 16 ++++++- tools/content-node/analysis-sizes.ts | 38 ++++++++++------- .../fixtures/analysis-size-differences.json | 2 +- 4 files changed, 67 insertions(+), 31 deletions(-) diff --git a/tools/content-node/analysis-sizes-check.ts b/tools/content-node/analysis-sizes-check.ts index 3437007b..775e78b0 100644 --- a/tools/content-node/analysis-sizes-check.ts +++ b/tools/content-node/analysis-sizes-check.ts @@ -1,6 +1,7 @@ /** * Node's analysis sizes against Java's (spec 006 T103a, research D25): for - * every species Java has a pathway hierarchy for, each pathway's + * every species with a top-level pathway in the graph -- those Java has a + * hierarchy for, each of which must answer -- each pathway's * totalEntity and totalEntityAndInteractors in Java's * /data/eventsHierarchy/{taxId}?pathwaysOnly=true, against analysisSizes. * @@ -9,12 +10,15 @@ * Passes when the only differences are those recorded in * fixtures/analysis-size-differences.json (analysis-core's alias merge, * reactome/analysis-core#13), each with the values recorded; fails on any - * other difference, a pathway one side lacks, or a recorded difference gone - * -- as when analysis-core is fixed, and the record should go with it. + * other difference, a pathway one side lacks, a species that does not answer, + * or a recorded difference gone -- as when analysis-core is fixed, and the + * record should go with it. Which interactor analysis-core's merge keeps + * depends on its build order, so a rebuilt data file may move a recorded Java + * value: the record is then measured again. */ import { readFileSync } from 'node:fs'; import { analysisSizes } from './analysis-sizes.ts'; -import { close } from './graph.ts'; +import { close, read } from './graph.ts'; type JavaNode = { stId: string; @@ -32,18 +36,19 @@ const recorded = ( ).differences; const problems: string[] = []; -const species = (await (await fetch(`${JAVA}/ContentService/data/species/all`)).json()) as { - taxId: string | null; - displayName: string; -}[]; +const species = await read<{ taxId: string; name: string }>( + `MATCH (:TopLevelPathway)-[:species]->(s:Species) + RETURN DISTINCT s.taxId AS taxId, s.displayName AS name ORDER BY name` +); let checked = 0; -for (const { taxId, displayName } of species) { - if (!taxId) continue; +for (const { taxId, name: displayName } of species) { const answer = await fetch( `${JAVA}/ContentService/data/eventsHierarchy/${taxId}?pathwaysOnly=true` ); - // Java has no hierarchy for most species, and answers those with a 500. - if (answer.status !== 200) continue; + if (answer.status !== 200) { + problems.push(`${taxId} ${displayName}: Java answered ${answer.status}`); + continue; + } const tree = (await answer.json()) as JavaNode[]; const sizes = await analysisSizes(taxId); const expected = recorded[taxId] ?? {}; @@ -73,15 +78,24 @@ for (const { taxId, displayName } of species) { for (const child of node.children ?? []) walk(child); }; for (const node of tree) walk(node); + const javas = new Set(); + const collect = (node: JavaNode) => { + javas.add(node.stId); + for (const child of node.children ?? []) collect(child); + }; + for (const node of tree) collect(node); + for (const stId of sizes.keys()) + if (!javas.has(stId)) problems.push(`${taxId} ${stId}: in node's tree, not Java's`); for (const stId of Object.keys(expected)) if (!seen.has(stId)) problems.push(`${taxId} ${stId}: recorded, but not in Java's tree`); checked++; console.log(`${taxId} ${displayName}: ${sizes.size} pathways`); } for (const taxId of Object.keys(recorded)) - if (!species.some((s) => s.taxId === taxId)) problems.push(`${taxId}: recorded, no such species`); + if (!species.some((s) => s.taxId === taxId)) + problems.push(`${taxId}: recorded, but no top-level pathway`); await close(); -if (checked === 0) problems.push('no species answered: is Java running?'); +if (species.length === 0) problems.push('no species with a top-level pathway: is the graph there?'); console.log(problems.length ? problems.join('\n') : `${checked} species: as recorded.`); process.exitCode = problems.length ? 1 : 0; diff --git a/tools/content-node/analysis-sizes.spec.ts b/tools/content-node/analysis-sizes.spec.ts index d73d0355..5375572f 100644 --- a/tools/content-node/analysis-sizes.spec.ts +++ b/tools/content-node/analysis-sizes.spec.ts @@ -55,9 +55,21 @@ describe('analysis sizes', () => { it("adds a target's partners under the target's resource", () => { const got = sizes({ entities: [{ pathway: 'B', ids: [['UniProt', 'P1']] }], - targets: [{ pathway: 'B', ids: [['UniProtKB', 'P1']] }], + // A target in a resource that does not count brings no partners. + targets: [ + { + pathway: 'B', + ids: [ + ['UniProtKB', 'P1'], + ['UCSC', 'U1'], + ], + }, + ], // P1 is its own partner, and already counted; Q9 is new. - partners: [{ acc: 'P1', partners: ['P1', 'Q9'] }], + partners: [ + { acc: 'P1', partners: ['P1', 'Q9'] }, + { acc: 'U1', partners: ['Z1'] }, + ], }); expect(got['B']).toEqual({ totalEntity: 1, totalEntityAndInteractors: 2 }); expect(got['TOP']).toEqual({ totalEntity: 1, totalEntityAndInteractors: 2 }); diff --git a/tools/content-node/analysis-sizes.ts b/tools/content-node/analysis-sizes.ts index ac184359..8d602059 100644 --- a/tools/content-node/analysis-sizes.ts +++ b/tools/content-node/analysis-sizes.ts @@ -20,11 +20,17 @@ * - Resources: ResourceFactory's main and auxiliary ones, under its * equivalences; any other is not counted. * + * Entities and targets are read for the pathways in the tree, whatever their + * own species: analysis-core attaches each pathway's counts to every place it + * holds in any hierarchy (pathwayLocation, by dbId). + * * One known difference (research D25): analysis-core merges a partner whose * identifier is another interactor's alias into that interactor - * (reactome/analysis-core#13), so 79 human pathways' totalEntityAndInteractors - * are one or two lower in Java than here. Computed once per species while the - * process lives: the graph is a release's, and changes only with it. + * (reactome/analysis-core#13), so in release 98 76 human pathways' + * totalEntityAndInteractors are 1 to 6 lower in Java than here, and 8 yeast + * pathways' 1 lower; fixtures/analysis-size-differences.json records them. + * Computed once per species while the process lives: the graph is a + * release's, and changes only with it. */ import { read } from './graph.ts'; @@ -75,18 +81,19 @@ UNION MATCH (t:TopLevelPathway)-[:species]->(:Species {taxId: $taxId}) RETURN t.stId AS parent, null AS child`; -/** EntitiesBuilder: each species pathway's own reactions' identifiers. */ +/** EntitiesBuilder: each tree pathway's own reactions' identifiers. */ const ENTITIES_QUERY = ` -MATCH (:Species {taxId: $taxId})<-[:species]-(p:Pathway)-[:hasEvent]->(rle:ReactionLikeEvent), - (rle)-[:input|output|catalystActivity|physicalEntity|entityFunctionalStatus|diseaseEntity|regulatedBy|regulator|hasComponent|hasMember|hasCandidate|repeatedUnit|proteinMarker|RNAMarker*]->(:PhysicalEntity)-[:referenceEntity]->(re:ReferenceEntity) +MATCH (p:Pathway)-[:hasEvent]->(rle:ReactionLikeEvent) +WHERE p.stId IN $pathways +MATCH (rle)-[:input|output|catalystActivity|physicalEntity|entityFunctionalStatus|diseaseEntity|regulatedBy|regulator|hasComponent|hasMember|hasCandidate|repeatedUnit|proteinMarker|RNAMarker*]->(:PhysicalEntity)-[:referenceEntity]->(re:ReferenceEntity) WITH DISTINCT p, re RETURN p.stId AS pathway, collect(DISTINCT [re.databaseName, coalesce(re.variantIdentifier, re.identifier)]) AS ids`; -/** InteractorsBuilder's targets: what a pathway's reactions reach that may interact. */ +/** InteractorsBuilder's targets: what a tree pathway's reactions reach that may interact. */ const TARGETS_QUERY = ` -MATCH (:Species {taxId: $taxId})<-[:species]-(p:Pathway)-[:hasEvent]->(rle:ReactionLikeEvent), - (rle)-[:input|output|catalystActivity|physicalEntity|entityFunctionalStatus|diseaseEntity|regulatedBy|regulator*]->(:PhysicalEntity)-[:referenceEntity]->(re:ReferenceEntity) -WHERE (p:TopLevelPathway) OR (:TopLevelPathway)-[:hasEvent*]->(p) +MATCH (p:Pathway)-[:hasEvent]->(rle:ReactionLikeEvent) +WHERE p.stId IN $pathways +MATCH (rle)-[:input|output|catalystActivity|physicalEntity|entityFunctionalStatus|diseaseEntity|regulatedBy|regulator*]->(:PhysicalEntity)-[:referenceEntity]->(re:ReferenceEntity) WITH DISTINCT p, [re.databaseName, coalesce(re.variantIdentifier, re.identifier)] AS target RETURN p.stId AS pathway, collect(target) AS ids`; @@ -168,10 +175,13 @@ export function computeSizes({ tree, entities, targets, partners }: SizeInputs) /** The graph's inputs for one species. */ async function sizeInputs(taxId: string): Promise { - const [tree, entities, targets] = await Promise.all([ - read(TREE_QUERY, { taxId }), - read(ENTITIES_QUERY, { taxId }), - read(TARGETS_QUERY, { taxId }), + const tree = await read(TREE_QUERY, { taxId }); + const pathways = [ + ...new Set(tree.flatMap(({ parent, child }) => (child === null ? [parent] : [parent, child]))), + ]; + const [entities, targets] = await Promise.all([ + read(ENTITIES_QUERY, { pathways }), + read(TARGETS_QUERY, { pathways }), ]); const accs = [...new Set(targets.flatMap((row) => row.ids.map(([, id]) => id)))]; const partners = await read(PARTNERS_QUERY, { accs }); diff --git a/tools/content-node/fixtures/analysis-size-differences.json b/tools/content-node/fixtures/analysis-size-differences.json index 76e57640..5d3fc16d 100644 --- a/tools/content-node/fixtures/analysis-size-differences.json +++ b/tools/content-node/fixtures/analysis-size-differences.json @@ -1,5 +1,5 @@ { - "_about": "Pathways whose totalEntityAndInteractors node computes higher than Java, release 98: analysis-core merges an interaction partner whose identifier is another interactor's alias into it (reactome/analysis-core#13; spec 006 research D25). By taxId, then stId. analysis-sizes-check.ts fails on any other difference, or on one of these gone.", + "_about": "Pathways whose totalEntityAndInteractors node computes higher than Java, release 98: analysis-core merges an interaction partner whose identifier is another interactor's alias into it (reactome/analysis-core#13; spec 006 research D25). By taxId, then stId. analysis-sizes-check.ts fails on any other difference, or on one of these gone. Which interactor the merge keeps depends on analysis-core's build order, so a rebuilt data file may move a Java value; measure the record again then.", "differences": { "4932": { "R-SCE-199991": {