risingwave: Persistent Storage (S3) grows indefinitely
Describe the bug
We are observing that the file storage in S3 continues to grow indefinitely, even on an idle database.
Error message/log
There are no errors associated with this issue.
To Reproduce
Graph your S3 BucketSizeBytes CloudWatch metric for idle databases to observe this.
Expected behavior
We kind of expected the “compactor” to actually compact the data and the size to decrease when there aren’t many changes happening. At the least we’d expect the size to remain relatively constant on a database that is mostly idle. In our load testing environment, which is not idle, the rate of growth seems to be much faster and larger than the rate of data change. It never recovers, even when changes are undone and left to “rest” for a bit.
How did you deploy RisingWave?
We are deployed in ECS.
The version of RisingWave
PostgreSQL 9.5.0-RisingWave-1.7.2 (6ecc083c72b83a96e7e3b58ca3a19b28cd5f74bf)
Additional context
S3 data in & out can be pricey. What is it doing that it just keeps writing “stuff” into S3 and burning money?
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 25
@gshmu Is the issue resolved? If not, please upgrade to our latest version (v2.0) and try.
We don’t have enough information for debugging. When the issue happens next time, please provide the information according to my comment above.
My guess is that either compaction is severely lagging or there is a long running batch query pinning the stale objects. Which version are you using? Please kindly provide the above information when the issue occurs next time.