Skip to content

store: keep postponed attribute indexes pending when a graft copies from an unsynced base - #6756

Open
lx3133584 wants to merge 1 commit into
graphprotocol:masterfrom
lx3133584:fix/postponed-indexes-on-graft
Open

lx3133584 wants to merge 1 commit into
graphprotocol:masterfrom
lx3133584:fix/postponed-indexes-on-graft

Conversation

@lx3133584

Copy link
Copy Markdown

Problem

With GRAPH_POSTPONE_ATTRIBUTE_INDEX_CREATION=true, a graft (or graphman copy create) taken while the source has not reached the chain head yet produces a destination with postponed_indexes_created = true but without the postponed attribute indexes. Because that flag short-circuits create_postponed_indexes, the destination never builds them — not even once it reaches the head — and queries filtering on those attributes run without an index.

This mostly hits grafted subgraphs: an indexer picking one up from scratch indexes the base only up to the graft block, so the base never gets near the head and never builds its postponed indexes; the graft then copies a base without them and marks them as created. Nothing in the logs says anything.

Reported in #6749 with a production case: deployment B copied from A while A was syncing ended up with 343 indexes against A's 709, the 366 missing ones exactly the BTree attribute indexes (~8.4 GB on A).

Root Cause

store/postgres/src/copy.rs, create_indexes (l. 302-344): the destination only gets the indexes in index_list, i.e. those that exist on the source, plus indexes for new columns — and then set_postponed_indexes_created(dst) is called unconditionally.

The flag was added to the copy path in #6583 so that a copy does not recreate indexes that were intentionally dropped on the source. It cannot tell "dropped on the source" from "not built yet on the source", and the copy path treats both as the former.

Fix

Read postponed_indexes_created for the source and set the flag on the destination only when the source has it set. A source that finished syncing and then had indexes dropped on purpose still has the flag, so the behaviour #6583 introduced is preserved; a source that is still syncing does not, so the destination is left pending and builds the indexes through the normal create_postponed_indexes path once it gets near the head.

Testing

Added graft_before_postponed_indexes to store/test-store/tests/postgres/graft.rs. It grafts at block 1 while the base has no postponed index and the flag is false, then runs create_postponed_indexes on both deployments and checks the destination ends up with the attribute indexes for the columns it shares with the base.

On master the test fails on the last assertion:

dst after graft: flag=true postponed=["attr_0_5_user_added"]
dst after create_postponed_indexes: ["attr_0_5_user_added"]
graft marked postponed indexes as created but they are missing: ["attr_0_5_user_added"]

With the fix the destination gets them:

dst after graft: flag=false postponed=["attr_0_5_user_added"]
dst after create_postponed_indexes: ["attr_0_1_user_name", "attr_0_2_user_email", "attr_0_3_user_age", "attr_0_4_user_favorite_color", "attr_0_5_user_added"]

Full suite against PostgreSQL 14, with and without GRAPH_POSTPONE_ATTRIBUTE_INDEX_CREATION=true:

cargo test -p test-store --test postgres -- --test-threads=1
test result: ok. 99 passed; 0 failed; 0 ignored

Fixes #6749

…rom an unsynced base

A graft or `graphman copy` made while the source has not reached the chain
head yet produced a destination marked `postponed_indexes_created = true`
without the postponed attribute indexes. `copy_data` recreates on the
destination only the indexes that exist on the source, so a source that has
not built them yet hands over none of them, and `create_indexes` then set the
flag unconditionally. `create_postponed_indexes` returns early once that flag
is set, so the destination never built them, not even at the chain head, and
queries filtering on those attributes ran without an index.

Set the flag on the destination only when the source has already created its
own postponed indexes. That keeps the reason the flag was added to the copy
path -- not recreating indexes that were dropped on the source on purpose --
while leaving it unset for a source that is simply still syncing, so the
destination builds the indexes when it gets near the head.

The regression test grafts at block 1 before the base builds anything, then
lets both deployments run `create_postponed_indexes` and checks that the
destination ends up with the attribute indexes for the columns it shares with
the base.

Signed-off-by: 梁旭 <liangxu.5@bytedance.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Grafted deployments (and graphman copy) never build their postponed attribute indexes when the base had not built them yet (v0.45.0 regression)

1 participant