Spec 006 T103: how the events hierarchy is ported (research D25, tasks)

specs/006-content-service-to-node/research.md

@@ -1198,6 +1198,65 @@ diagrams and reactions do, instead of refusing them (#613's 400s): its token
 is drawn, the rest are accepted. A `.gif` of it is still refused, a format
 the genome-wide view cannot capture.
 
+## D25. The events hierarchy: the AnalysisService's sizes, snapshotted at release (T103)
+
+`GET /data/eventsHierarchy/{species}` is what the Pathway Browser's Reacfoam
+draws (`reacfoam.service.ts`): always `pathwaysOnly=true`, by taxId, with an
+analysis `token`, `resource` and `interactors` when one is shown. Measured
+11 Nov 2026 against release 98.
+
+**Two trees.** With `pathwaysOnly`, Java answers from analysis-core's
+in-memory `PathwayHierarchy`, which the AnalysisService's data file holds: its
+names and types come from that file, and siblings sit in a `TreeSet` ordered by
+species, `hasEvent` order (a top-level pathway's is its place among
+`toLower(displayName)`), then name by `String.compareTo`, so equal siblings
+merge. The full form is graph-core's event hierarchy (node already parses it,
+`parseHierarchy`), each pathway's sizes looked up by stId and every other event
+`0/0`. Without a token, both are deterministic; `interactors`, `resource` and
+`importableOnly` change nothing.
+
+**The sizes.** `totalEntity` is, per pathway, the distinct
+`(main resource, identifier)` pairs of the physical entities its reactions
+reach, its subpathways' included; `totalEntityAndInteractors` adds each
+interaction partner's identifier under the target's resource. Computed from the
+graph, `totalEntity` matched Java for every pathway of five species (3,067
+human); `totalEntityAndInteractors` matched four species exactly and 2,988 of
+3,067 human pathways, the other 79 one or two higher. analysis-core's
+InteractorsBuilder keys interactors by upper-cased identifier in one radix map
+that holds their aliases too, so a partner whose identifier is another's alias
+is merged into whichever was made first: hash-map iteration and Neo4j's order
+of equal scores, which the graph cannot reproduce.
+
+**Decision: node computes the sizes from the graph, once per release, and
+the 79 are a recorded difference.** The gap is one or two entities on pathways
+holding hundreds or thousands (Signal Transduction 10,318 against 10,316), far
+below anything a reader of the Reacfoam can see, and the alternative -- a Java
+step at each release reading the data file with analysis-core -- adds a library
+build the project is not set up to keep current. The 79 are listed as an
+expected difference, so the comparison with Java fails on any other; the cause
+is reported to analysis-core (reactome/analysis-core#13, fix in
+reactome/analysis-core#14: InteractorsBuilder.getOrCreate looks a new partner
+up among every interactor's aliases), and once fixed there, Java's
+numbers become node's. The `pathwaysOnly` tree is built from the graph by the
+analysis hierarchy's rules. Rejected: a release-time snapshot of the data file
+(exact, but a Java library step to maintain); reading the totals from an
+analysis of every identifier (`result.json`, 134 MB, a hack).
+
+**The overlay.** With a token, Java adds `llp`, `entities` and `reactions` to
+each node the stored result names. Node reads them through the analysis
+gateway: `/download/{token}/result.json` carries per-pathway TOTAL and
+per-resource statistics, and `/token/{token}?importableOnly=true` the
+importable ones. Java collects a hit node's children into a HashSet of identity
+hashes, so its own order below a hit changes from one request to the next:
+token answers are compared without regard to that order.
+
+**Errors to keep:** an unknown species, or one with no hierarchy, is a 500
+("Something unexpected happened…"); `pathwaysOnly` reads a taxId, dbId or name,
+the full form treats a number as a taxId and an exact name ends in a
+StackOverflowError 500; a bad boolean is Spring's 400; a bad token a 500 "Not
+Found"; every 200 carries `Content-Disposition: inline;
+swaggerDownload="attachment"; filename="{species}.json"`.
+
 ## What exists already
 
 Branch `content-node-spike`, deliberately off main:

specs/006-content-service-to-node/tasks.md

@@ -106,7 +106,11 @@ These were held in Java because their substance is a Java library or Java numeri
 - [x] T100 `GET /exporter/event/{identifier}.sbml`, `GET /exporter/event/{identifier}.sbgn`, `GET /exporter/document/event/{identifier}.pdf`. Not ported one by one: each is replaced by an exporter Reactome keeps for both the bulk file and the website. **SBML** is done that way by [spec 015](../015-graph-exports/tasks.md) (node answers it through `sbml-exporter`, T065, #666; release and production wait for its T068). **PDF** is answered by `pdf-export`'s single-event service (its spec 001, Story 2 and FR-014): done, #882. Node answers `/exporter/document/event/{id}.pdf` (`eventPdfAnswer`), its parameters converted as Spring converts them under Java's own names (`level [0 - 1]`, the analysis options), its 404 and 400, then the PDF from the service (`deploy/pdf-export`: pdf-export pinned to a commit of its `main` in `deploy/java/PINS.md`, `PDF_SERVICE`; unset, a 503), passed on as it arrives with Java's type and the stable id as its file name (`"{stId}.pdf"`, the extension inside the quotes where Java's is outside: declared, #884), two at a time; an analysis token the AnalysisService does not know is a 404 and one whose result has gone a 410, where Java answers an empty 200 (declared). Routed on beta alone (dev.conf), locally and under ng serve. Java's errors compared live on 7 Nov 2026, spaced: the 404s (an unknown id, a versioned one, an old `REACT_` one), the 400 for a non-event and the three conversion 400s are Java's but for `url`. The service run against beta's graph, render service and AnalysisService (the same commit and entrypoint, outside its image): a reaction 6.3 s, a pathway of 13 reactions 27.7 s, an illustrated pathway 23.2 s, the mouse pathway 30.7 s, the pathway with a token 37.6 s, each within SC-005's 8 s and 3 s per figure, every figure drawn by the render service (its counters rose by each PDF's figures); `event-pdf` took 0.7 s, 5.4 s, 19.1 s, 4.4 s and 6.4 s (declared: no figures' folder yet). **SBGN** follows SBML's pattern, one exporter for the download page's bulk file and the website (decided 5 Nov 2026): done, node answers it through `sbgn-exporter` (its spec 001 T018), from the release's diagram files (`DIAGRAM_FILES`) or node's own reaction layout, routed on beta alone, #862. Compared on 7 Nov 2026 with Java's answers on a local deployment of release 98, spaced, by sbgn-exporter's `ci/compare.ts` with the layouts, for 62 events: 31 pathways across the bulk file's sizes (its largest among them, 8 with drugs, one by dbId and one by versioned stable id), a subpathway drawn in its parent's diagram, and 30 reactions by dbId. Node's documents were sbgn-exporter's from the layouts the endpoint reads: the release's diagram files, and node's own reaction layouts as beta answers them. Java's type, download name and CORS header were the same on every answer. The 32 pathway answers differ only in declared kinds. 24 reactions differ in every layout id, by Java's reaction-layout counter (offsets 225458 to 225691); with it removed, only in declared kinds (all 56: 99 D1, 74 D2, 213 D3, 815 D5, 283 D6, nothing undeclared). The other 6 reactions are drawn with a compartment that has no database object (the nuclear envelope around the nucleoplasm, say), which the library first pinned refused: fixed in [sbgn-exporter#34](https://github.com/reactome/sbgn-exporter/pull/34), and the pin moved to its dist commit `2144a05f`; with it, the six compare with only declared differences against Java's own converter run on the same layouts (the reference harness, offline: 9 D1, 7 D5). Java's 400 and 404 bodies are node's but for `url`. Java's 500 for a missing diagram file was read from its code, not provoked: it needs a file missing from Java's folder, and Java mails its error report for it.
 - [ ] T101 `GET /interactors/psicquic/molecule/{resource}/{acc}/details` and `/summary`, `POST /interactors/psicquic/molecules/{resource}/details` and `/summary`
 - [ ] T102 `POST /interactors/upload/tuple/content`, `/tuple/form`, `/tuple/url`, `POST /interactors/upload/psicquic/url`, `POST /interactors/token/{token}`
-- [ ] T103 `GET /data/eventsHierarchy/{species}` — portable. Java reads two things from analysis-core (EventsController): every node's `totalEntity`/`totalEntityAndInteractors`, from the in-memory pathway hierarchy the AnalysisService's data file holds, on every request; and, with a `token`, the stored analysis result it overlays. Node can read the result through the AnalysisService's own `token` API (behind the gateway). The research question is the sizes: computed from the graph as analysis-core counts them, byte for byte, or read from the AnalysisService.
+- [ ] T103 `GET /data/eventsHierarchy/{species}`, the Reacfoam's tree (research D25). In this order, a PR each:
+  - [ ] T103a The sizes from the graph, per species, as analysis-core counts them: `totalEntity` and `totalEntityAndInteractors`, importable-only too, computed once per release and cached; checked against Java's live `pathwaysOnly=true` answer for every species, the 79 human `totalEntityAndInteractors` recorded as the expected difference (D25) and nothing else allowed to differ.
+  - [ ] T103b Node answers `pathwaysOnly=true` without a token, the tree built by the analysis hierarchy's rules, byte-identical but for the recorded 79, errors and `Content-Disposition` included; routed in nginx and the proxy, contract type and recordings.
+  - [ ] T103c The token overlay, through the analysis gateway (`result.json`, `/token?importableOnly=true`); the diff harness compares a token answer without regard to the order below a hit node, as Java's own order changes.
+  - [ ] T103d The full form: graph-core's event hierarchy (`parseHierarchy`) with the sizes by stId.
 
 ## Phase 9b: Endpoints Java does not publish