Skip to content

DataLakeCatalog: metadata.json is downloaded on every query, even when already in metadata cache #2925

Open

Description

@xieandrew

Describe the bug

When a query reads an Iceberg table through a DataLakeCatalog database, the first read of metadata.json during table setup always goes to S3. This happens on the initiator and again on every swarm node, and on every query, even though the same file was put into the Iceberg metadata cache by the previous query. The cache is keyed by the table UUID and the first read happens before the UUID is known.

The catalog already gives the UUID in its GetTable response, so the cache lookup could happen before the download. Each of these uncached reads is a HEAD request followed by a GET, and a test cluster the pair takes about 45 ms. This latency happens twice per table on the initiator (see #2721) and once on every fragment, which adds up to a large part of the setup time for short queries.

To Reproduce

This needs system.blob_storage_log to be enabled in the server config, since that is where the S3 reads show up.

  1. Run the same query against a REST catalog table a few times, with a swarm cluster set and blob storage logging turned on for reads. The query needs a filter so that it actually runs on the swarm.
SELECT count() FROM lake.`ns.tbl` WHERE id > 0
SETTINGS object_storage_cluster = 'swarm',
         enable_blob_storage_log_for_read_operations = 1,
         log_comment = 'metadata-json-1';

The later runs are the same except for log_comment (metadata-json-2, metadata-json-3 and so on). The fragments inherit log_comment from the initial query, so it can be used to find their rows too.

  1. Flush the logs with SYSTEM FLUSH LOGS, or wait for the next automatic flush.

  2. List the reads of metadata.json on every node for each run:

SELECT q.log_comment, q.is_initial_query, b.hostname, b.remote_path,
       count() AS gets, round(avg(b.elapsed_microseconds) / 1000, 1) AS avg_get_ms
FROM clusterAllReplicas('swarm', system.blob_storage_log) AS b
JOIN clusterAllReplicas('swarm', system.query_log) AS q ON b.query_id = q.query_id
WHERE b.event_time > now() - INTERVAL 10 MINUTE
  AND b.remote_path LIKE '%metadata.json'
  AND q.type = 'QueryFinish'
  AND q.log_comment LIKE 'metadata-json-%'
GROUP BY ALL
ORDER BY q.log_comment, q.is_initial_query DESC, b.hostname;

Every run, including the warm ones, has two GETs of the same metadata.json on the initiator and one on each fragment. The other reads of the file in the same build don't show up there, because they are served from the cache. The HEAD request in front of each GET isn't in blob_storage_log, which only records the reads.

  1. Optionally, look at the profile events, which tell the same story but are harder to read. IcebergJsonFileParsing counts every call, cache hits included (7 on the initiator and 3 per fragment for a single-table query). IcebergJsonFileParsingMicroseconds includes the HEAD and the GET, and with a small metadata file nearly all of its time is those two requests:
SELECT log_comment, is_initial_query, hostName() AS host, query_duration_ms,
       ProfileEvents['IcebergJsonFileParsing'] AS json_calls,
       ProfileEvents['IcebergJsonFileParsingMicroseconds'] / 1000 AS json_ms
FROM clusterAllReplicas('swarm', system.query_log)
WHERE type = 'QueryFinish'
  AND event_time > now() - INTERVAL 10 MINUTE
  AND log_comment LIKE 'metadata-json-%'
ORDER BY log_comment, is_initial_query DESC, host;

Expected behavior
The UUID from the catalog could be passed down to the first read, so that it checks the cache before going to S3.

Screenshots
N/A

Key information
Provide relevant runtime details.

  • 26.6.8.20004.altinityantalya (The code path is the same in antalya-26.8)
  • AWS
  • S3
  • Ice REST Catalog

Additional context

Upstream ClickHouse#126267 (backported to antalya-26.8 in #2873) caches the parsed metadata.json, but only for reads that already know the UUID. It does not change the first read.

Upstream ClickHouse#99981 is open and makes the same change but is larger and would need rework before it could be backported.

#2721 is similar but a separate problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions