solana: RPC server with full-rpc-api & incremental-snaps, creating incremental too large

Problem

find this issue, node A: is one RPC node for wormhole v1, no voting full-rpc-api

115G    incremental-snapshot-130818676-130838232-G2mRKq1AXZdD1rViCc2wojmRPZVWetcviwsp8iJxwfMm.tar
104G    incremental-snapshot-130818676-130837922-FEQUpQsfYmDA8iSmgNPi1HbJayTP1bx7MakkhkCY2YmS.tar
93G    incremental-snapshot-130818676-130837603-7ktJj9aLSUcxE2Ltmyo3Y1DoTwPEWxuZvvKZi1UXBUQs.tar
83G    incremental-snapshot-130818676-130837294-H3pyoXsToJWQQQvk9RQ1pBL2zHREUtc9rgmrdFZbZtiw.tar
70G    snapshot-130818676-HX9cDJSYKbJsHVQGpCN6JbKw7UG9cPh54WvWgt1yBcxW.tar

Node B: no voting server, and no RPC use case running without full-rpc

74G    snapshot-130818676-HX9cDJSYKbJsHVQGpCN6JbKw7UG9cPh54WvWgt1yBcxW.tar
73G    snapshot-130792644-DEbLaquqHS1rfj96gsa9CTVUKYwiCtSYuDN7oykCJbBg.tar
23G    snapshot-130341166-DQjsqaTXutLRuEQBqZ73CEuU51hcDYzskFmAKL1hoLu4.tar.zst
4.8G    incremental-snapshot-130818676-130839371-9T8T4wFesoXCCp2dVnvqiJT2hixUp2vVw11v43cbmrtb.tar

not understand can you give some light here? of course, the node crashes after reaching the free space on the disk.

Proposed Solution

Fixed it

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 16 (8 by maintainers)

Most upvoted comments

issue closed, not observing this behavior on last versions

iammelea on May 27, 2022

These look to all be uncompressed snapshots, I wonder if that’s related.

@iammelea - can you reproduce this when you use normal snapshot compression? Generally these days we don’t recommend uncompressed snapshots actually, as the archive sizes can be quite large and generally the IOPS increase isn’t worth the CPU load decrease from not having to compress the tarball

mvines on Apr 22, 2022

https://github.com/solana-labs/solana/commit/bc35e1c5f57e653fe75ca06a2913ff1a37f58b9b This recent commit looks relevant for the incremental snapshot size. In the above change, we limits the incremental snapshots to only slots that is greater than the base slots.

HaoranYi on Apr 22, 2022

Thanks for reporting this, @iammelea. I haven’t done testing on an RPC node, so we’ll get on investigating this.

brooksprumo on Apr 22, 2022

cc/ @brooksprumo

t-nelson on Apr 22, 2022