Repository navigation
Description
Describe the bug
When a query reads an Iceberg table through a DataLakeCatalog database, the first read of metadata.json during table setup always goes to S3. This happens on the initiator and again on every swarm node, and on every query, even though the same file was put into the Iceberg metadata cache by the previous query. The cache is keyed by the table UUID and the first read happens before the UUID is known.
The catalog already gives the UUID in its GetTable response, so the cache lookup could happen before the download. Each of these uncached reads is a HEAD request followed by a GET, and a test cluster the pair takes about 45 ms. This latency happens twice per table on the initiator (see #2721) and once on every fragment, which adds up to a large part of the setup time for short queries.
To Reproduce
This needs system.blob_storage_log to be enabled in the server config, since that is where the S3 reads show up.
- Run the same query against a REST catalog table a few times, with a swarm cluster set and blob storage logging turned on for reads. The query needs a filter so that it actually runs on the swarm.
SELECT count() FROM lake.`ns.tbl` WHERE id > 0
SETTINGS object_storage_cluster = 'swarm',
enable_blob_storage_log_for_read_operations = 1,
log_comment = 'metadata-json-1';The later runs are the same except for log_comment (metadata-json-2, metadata-json-3 and so on). The fragments inherit log_comment from the initial query, so it can be used to find their rows too.
-
Flush the logs with
SYSTEM FLUSH LOGS, or wait for the next automatic flush. -
List the reads of
metadata.jsonon every node for each run:
SELECT q.log_comment, q.is_initial_query, b.hostname, b.remote_path,
count() AS gets, round(avg(b.elapsed_microseconds) / 1000, 1) AS avg_get_ms
FROM clusterAllReplicas('swarm', system.blob_storage_log) AS b
JOIN clusterAllReplicas('swarm', system.query_log) AS q ON b.query_id = q.query_id
WHERE b.event_time > now() - INTERVAL 10 MINUTE
AND b.remote_path LIKE '%metadata.json'
AND q.type = 'QueryFinish'
AND q.log_comment LIKE 'metadata-json-%'
GROUP BY ALL
ORDER BY q.log_comment, q.is_initial_query DESC, b.hostname;Every run, including the warm ones, has two GETs of the same metadata.json on the initiator and one on each fragment. The other reads of the file in the same build don't show up there, because they are served from the cache. The HEAD request in front of each GET isn't in blob_storage_log, which only records the reads.
- Optionally, look at the profile events, which tell the same story but are harder to read.
IcebergJsonFileParsingcounts every call, cache hits included (7 on the initiator and 3 per fragment for a single-table query).IcebergJsonFileParsingMicrosecondsincludes the HEAD and the GET, and with a small metadata file nearly all of its time is those two requests:
SELECT log_comment, is_initial_query, hostName() AS host, query_duration_ms,
ProfileEvents['IcebergJsonFileParsing'] AS json_calls,
ProfileEvents['IcebergJsonFileParsingMicroseconds'] / 1000 AS json_ms
FROM clusterAllReplicas('swarm', system.query_log)
WHERE type = 'QueryFinish'
AND event_time > now() - INTERVAL 10 MINUTE
AND log_comment LIKE 'metadata-json-%'
ORDER BY log_comment, is_initial_query DESC, host;Expected behavior
The UUID from the catalog could be passed down to the first read, so that it checks the cache before going to S3.
Screenshots
N/A
Key information
Provide relevant runtime details.
- 26.6.8.20004.altinityantalya (The code path is the same in
antalya-26.8) - AWS
- S3
- Ice REST Catalog
Additional context
Upstream ClickHouse#126267 (backported to antalya-26.8 in #2873) caches the parsed metadata.json, but only for reads that already know the UUID. It does not change the first read.
Upstream ClickHouse#99981 is open and makes the same change but is larger and would need rework before it could be backported.
#2721 is similar but a separate problem.
Activity