Skip to content

fix(rpc): cap Caddy's container log size - #13

Merged
nol4lej merged 2 commits into
mainfrom
fix/caddy-log-limit
Nov 2, 2026
Merged

nol4lej merged 2 commits into
mainfrom
fix/caddy-log-limit

Conversation

@nol4lej

@nol4lej nol4lej commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Problem

rpc-1 was down for ~36 h. The disk filled to 100%, the node left db_version empty, and on every start it failed with:

Error: Service(Client(Backend("Database version cannot be read from existing db_version file")))

What was on the disk (75 GB):

Item Size
Archive node DB 52 GB
Docker build cache 6.4 GB
journald 2.5 GB
Caddy container log 1.2 GB

The Caddy log is the only item caused by this repo's config: the caddy service had no logging: block, so Docker never rotated it. During the outage, every request was logged as a 502 with its full headers.

Change

Add json-file log rotation to the caddy service in testnet/rpc/docker-compose.yml and mainnet/rpc/docker-compose.yml: 50 MiB per file, 3 files. This matches the node service.

Deploy

On each RPC host:

git pull && docker compose up -d caddy

…), document the sentry role of RPC nodes and never --reserved-only without a sentry
@nol4lej
nol4lej merged commit c619686 into main Nov 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant