Repository navigation
feat(graph-explorer): explore the graph's model, start a mapping sheet and check one - #144
Merged
Merged
Conversation
…t and check one A section at /graph-explorer for people modelling a new dataset. It reads the Graph API's ontology routes through the existing proxy and shows the model the graph serves. Explore opens on the datasets and the kinds of thing their figures are about. Any term can be put at the centre: its broader and narrower terms, what it is used with, what it connects to, and beside it the definition, the dcid and the StatVar count per dataset. A dataset at the centre shows what it measures, between whom and split by what, and how its StatVars are formed, as sentences. A dataset filter marks what one dataset uses while the reader moves. Lists are scoped to one kind of thing or to the properties the instance declares, and compare datasets side by side. Start a sheet builds the columns of a mapping sheet from a kind of thing and the properties that split its figures, names the closest existing dataset pattern, and exports CSV or a Google Sheet in the reader's Drive with a dropdown per value list. Check a sheet takes a range pasted from a spreadsheet, sends it to the check route in chunks and marks each cell the graph does not accept, with the closest values. The section admits signed-in readers whose verified email domain is in GRAPH_EXPLORER_EMAIL_DOMAINS; an unset list admits nobody. The proxy passes the requested path to the gate so sign-in returns to it.
safeNext resolved `next` against a placeholder origin and kept the resolved path. Dot segments can resolve `/a/..//host` to `//host`, which a redirect reads as another host. safeNext now answers `/studio` for a resolved path that begins with `//`. graphExplorerSignIn judges the requested path after URL resolution, so a path with dot segments that leaves /graph-explorer returns to the Graph Explorer's first screen.
…e and tell Google failures apart The base switch is `?base=1`, and every link keeps `show` and `base`. ExplorerDataProvider sits in the layout, so the ontology, provenances and terms documents load once per visit. An unknown `?show=` gets a line naming the dataset with a Clear control. A link to the view already showing adds no history entry. readGoogleAccessWith returns one of three reasons to ask for access (no Google account, scope missing, token expired). A failed auth call answers 502 `auth_failed` and is logged without the token. The page marks the return from Google, and says so when access is still refused. checkRows halves a refused chunk and doubles after each chunk the route answers, up to the starting size. The wire guards check every element of every list. Flagged rows show 300 at a time. Kind words, paths, sheet links and refusal copy each have one definition. Sheet drafting, list filtering, member search and the focus headings moved from components into src/lib/graphExplorer with tests. The marks are off on a dataset's own view, every box orders by use, and requireReader logs once when the allowlist is empty.
…k the right cell parsePasted reads comma-separated text when the header line holds no tab, strips a leading byte-order mark, and reads a quoted cell left open at the end of the text as plain text. When two columns map to the same property or definition field, the first keeps the mapping, each later one is set to Ignored, and step 2 names them. requestGraph in load.ts is the one request loop for loadDocument and checkRows. It waits out a 503 or a 429 that carries Retry-After for up to a minute, and a 400 whose body stalls until the request bound fires is a timeout. findingLines computes the closest members once per field and value. Findings and refused rows show 300 at a time, and focus moves to the first newly shown line when the last step is shown.
…ailed renewal apart Better Auth's /get-access-token, /refresh-token and /account-info are closed over HTTP (TOKEN_PATHS in src/lib/auth/closedPaths.ts). The server's own token read still works. A failed token read in Better Auth answers 502 google_renewal_failed and the page offers Try Again and Continue with Google. A return from Google with an error counts as refused and keeps the consent button. A failed Google call logs its cause, and a refusal logs Google's status and reasons. resolveSheet names the class, measured property and split ids from the URL that the model lacks. The more-properties list keeps its own open state, and a created sheet's link goes when the draft changes. emailDomain refuses a domain with characters outside ASCII letters, digits, dots and hyphens. resolveOnSite in nextPath.ts is the one on-site resolver for safeNext and graphExplorerSignIn, and `//` gives the fallback. The proxy sets the request-path header through withRequestPath, which replaces a value the client sent.
…he base switch on every screen The ontology document marks each enumeration `complete` when its list is every member the graph holds. wire.ts guards it, the model carries it, and a screen that counts members says "partial list" for an incomplete one. The wire guards refuse a document that lists one dcid twice. TopBar reads the loaded documents and the current screen itself and holds the "Search base Data Commons" switch, so every screen, the error screen included, can search and turn the switch off. A base-term view shows the loading state until the terms list is in. A wait names its cause, so a 429 wait names the site's limit. Check a sheet works out closest members only for the lines on screen. byUse replaces topByUse. periodText reads an ISO 8601 period as a word. A scoped-list row carries `dimmed`, listScopeExists judges a `list` scope, and resolveSheet names a dataset pattern the model lacks. queryValue, SheetSection, the search box's Enter and Escape rules and the Google access reasons each have one definition.
… of shared links and mark partial lists in exports parsePasted chooses its delimiter from the header record with quotes honoured, and a closing quote counts only before a delimiter, a line end or the end of the text. A column that repeats an earlier one names it beside its select. Start a sheet reads the consent trip once on mount and removes the parameter from the address, so a copied link never carries it. The create state lives with the draft it was made for, and one create runs at a time. A partial value list carries one line on the Google Sheet's value-list tab and a -partial suffix on its CSV file name. The member-count wording lives in src/lib/graphExplorer/counts.ts. A terms list that is loading or failed shows once.
…ddress takeConsentTrip and consentReturns remove error_description with the other parameters a consent trip leaves.
…d sentences in screen copy Comments and docstrings across the Graph Explorer state what the code does in the present tense. Nine on-screen strings that joined two sentences with a semicolon or colon read as two sentences.
…nance readTerms accepts a null declared_by on a term, as readOntology does on a node. Base search lists such a term when the model does not hold it.
… sheet flows - Sheet exports carry the measurement denominator and observation properties, so CSV and Google Sheet match the chosen pattern. - A consent trip judges only the first answer after the return, a 403 scope refusal asks for consent, and a create that times out says the sheet may already be in Drive. - Sheet requests fit the handler's member and body limits by construction, cut lists marked partial. - One class rule for measured options and the closest pattern, and an unusable pattern from the URL is named. - safeNext reads the first of a repeated next parameter. - An aborted terms load restarts, keyboard search scrolls the active result into view, and the error screen resets on a new query. - A value list's dataset use comes from its members. - The check page lists sheet problems before sending, waits past the server's deadline, and ranks suggestions with a banded distance. - The model indexes property use once.
…t_row - The explorer loads /api/graph/schema, /schema/terms and /schema/usage, and the check posts to /api/graph/schema/check. - A check sends up to 2,000 rows a request and resumes at next_row. A 400 naming one row marks that row refused and the check goes on. - Findings are placed by row, property and subject, served values come from served, and origin_not_served has its own message. - The page blocks only what the route refuses as a whole: too many properties in a row, or a value too long. Every other fault comes back as a finding on its cell. A leading dcid: is stripped from node references. - A 503 on the check reads as busy or still building.
Each node's other_properties appear in the definition panel beside the named fields, are searchable by value and by property name, add columns to a value list's members in the panel, the value-list CSV and the Google Sheet tab, and feed the check page's suggestions, so a source code such as a wbCode points to the member that carries it.
- declared_by reads as an origin id (dfo, base, un_system), and the definition panel names the declaring origin. - The check resumes at rows_checked, sends no ref, and has its own message for not_a_node. - Problem bodies are read for cause.code and retryable: a schema still building and a busy check wait with their own words, and a 500 or a refusal marked not retryable stops at once. - An unknown value of a fixed member shows a plain fallback in place of failing the whole document.
…pe DFO terms as dfo
Schema nodes, terms and usage entries are read by ref and label, as on
the node route, and a node value in other_properties is {ref}. The
DFO-declared scope is kind dfo, served at ?list=dfo.
The CSS copy of each word (::after with attr(data-word)) counted toward the accessible name, so screen readers heard every word twice. The word spans are aria-hidden and one visually hidden span carries the label.
A class's instances {refs, complete} make it a value list, so
currencies, groupings and the other DFO range classes get lists on
Explore, in Start a sheet's dropdowns, value-list tab and CSV, and in
Check a sheet's suggestions. A code column appears only when some
member has a code, in the tree, the CSV and the Google Sheet tab, and
the value tree draws no list markers.
… bound sheet exports - A class's kind and whether it lists values are separate facts, so Grouping, CurrencyUnit and the other value classes keep their class picture and also show their list. Enumerations focus as value lists. - A property's value list is the union of its range classes' lists, partial when any range class has none. - The page cuts labels, codes, column names and the title to the handler's bounds, ids are bounded at the graph's dcid length, and the handler refuses a sheet whose Google create body passes 2 MB.
The value tree's header and rows share one region that scrolls sideways inside the panel, behind a sticky name column with a Name header. Each code column is as wide as its content up to 16 characters, and codes and dcids break only after "/". A reference cell shows the node's plain name in the tree, the CSV and the sheet tab, with the full label where plain names collide, and the member filter matches those names.
…at every width - Ids break only at "/", ".", "_" and camelCase boundaries, through one BreakableId component; prose wraps with break-word. - The focus picture keeps 240 px side columns, stacks below a 720 px picture, joins its connectors to the centre card, and the base tag and dimmed items reach at least 3.4:1. - The explorer's focus ring is 1 px ink at a 2 px offset, the search clear glyph uses ink, and scroll regions show a thin scrollbar. - A failed load says search is off and explains the failure once. - The check summary counts rows with findings, a clean check shows one result line, phone column rows share one layout, and sheet export messages carry their tone. - The definition panel names held references by plain name, and an unlabelled dataset reads as words.
…y their defining fields The check page lists the API's range_not_judged warning with its own message, so a check holding one never reads "No problems". A sheet-template link names its pattern by a hash of the fields that define it (population type, measured property, stat type, qualifier, denominator, constraint and observation properties), so reordering patterns by usage cannot switch the template a link opens. A link to a pattern the dataset no longer has shows a notice and starts from the class's most-used pattern.
jm-rivera
marked this pull request as ready for review
November 14, 2026 21:17
…s limit Start a sheet takes at most 16 split properties, the most constraint properties the check accepts per row. Once 16 are ticked the other boxes are disabled and a note states the limit. A link naming more keeps the first 16 in link order and names the rest in a notice.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the Graph Explorer at
/graph-explorer, a tool for the people who map new datasets onto the graph. Explore shows the model's classes, properties and value lists, every property each node carries, and which datasets use each. Start a sheet drafts a mapping sheet as a CSV or a Google Sheet, with dropdowns and a value-list tab. Check a sheet tests a pasted sheet against the graph before any pipeline code is written and suggests the value a wrong cell probably meant. The section is open to signed-in readers whose verified email domain is listed inGRAPH_EXPLORER_EMAIL_DOMAINS.Changes
src/app/graph-explorer/: the layout with the reader gate, the three pages, an error screen, and theapi/sheetsroute handler.src/components/organisms/GraphExplorer/,NewSheet/,CheckSheet/,ReaderRefused/: the screens. The view (nodeorlist), the dataset filter (show) and the base Data Commons switch (base) live in the URL.src/lib/graphExplorer/: the logic, with tests.wire.tsguards the four Graph API bodies and shows a plain fallback for an unknown value of a fixed member.model.tsbuilds the model, and any class withinstancesis a value list.search.tsmatches names, codes and the values and names ofother_properties, soDODfindsAmountOutstanding.pasted.tsreads a pasted TSV or CSV.check.tsshows each finding code,range_not_judgedamong them, with its own message, sends up to 2,000 rows a request, resumes atrows_checked, marks a row the route refuses by its/rows/<i>pointer and goes on, and strips a leadingdcid:from node cells.load.tsholds the one request loop, which waits out a 503 or 429 withRetry-Afterand stops at once on a 500 or a problem marked not retryable.googleSheet.ts,sheetsExport.tsandgoogleConsent.tscreate the Google Sheet.newSheet.tsnames a dataset pattern in a link by a hash of the fields that define it, so reordering patterns by usage cannot change the template a link opens, and a link to a pattern the dataset no longer has shows a notice. A sheet splits by at most 16 properties, the most constraint properties the check accepts per row; past that the other boxes are disabled, and a link naming more keeps the first 16 and names the rest.src/lib/auth/reader.ts,requireReader.ts: the gate. An unsetGRAPH_EXPLORER_EMAIL_DOMAINSadmits nobody.src/lib/auth/closedPaths.ts,auth.ts: Better Auth's/get-access-token,/refresh-tokenand/account-infoanswer 404 over HTTP, so a provider token cannot reach the browser.src/lib/auth/nextPath.ts:safeNextrefuses anextpath that resolves to//host. This is its own commit and affects every sign-in.src/components/atoms/Button/: each button's accessible name says its label once, where screen readers heard every word twice. This is its own commit and affects every button with a word label.src/proxy.ts: setsx-dfo-request-pathon/graph-explorerrequests, replacing any value the client sent, so a signed-out reader returns to the page they asked for.src/styles/_mixins.scss:scrollbar-visibletakesbothfor a box that scrolls either way.src/lib/crawl.ts,.env.example,deploy/README.md: the section isnoindex, and the new variable and Google settings are documented.Notes for reviewers
GET /api/graph/schema,/schema/terms,/schema/usageandPOST /api/graph/schema/checkthrough the/api/graph/*proxy, which feat(graph): serve the graph's schema, its use by each dataset and a StatVar definition check apis#301 adds. Deploy dfo-apis first: the screens load nothing until it serves those routes.wire.tsholds hand-written types for them, which the vendored client can replace once it is regenerated from the deployed API.GRAPH_EXPLORER_EMAIL_DOMAINS, without which nobody can open the section. Create Google Sheet also needs thedrive.filescope on the Google sign-in OAuth client's consent screen and the Google Sheets API enabled in its project. Without the Sheets API, creating a sheet answers 502 and logsPERMISSION_DENIED SERVICE_DISABLED. The CSV downloads work without either.Testing:
pnpm test(2,626 of 2,626 pass),pnpm typecheck,pnpm lint,pnpm lint:cssandpnpm buildpass at dec465b. At a8c597e the graph explorer tests (334 of 334),tsc --noEmitandeslinton the changed files pass; the fullpnpm test,lint:cssandbuildhave not run there. A three-repo end-to-end run against a local Graph API on the dev graph checked the explorer in a browser at 1280 and 400 px: search byDOD,CHF,756andwbCode, the definition panel's extra properties, the 272-member currency value list and its CSV, a sheet with a currency dropdown and value-list tab, and a check of an 11-row paste whose cells got the expected findings and suggestions. A visual QA pass at 1280, 768 and 400 px followed, and its confirmed faults are fixed.