risingwave: Persistent Storage (S3) grows indefinitely

Describe the bug

We are observing that the file storage in S3 continues to grow indefinitely, even on an idle database.

Error message/log

There are no errors associated with this issue.

To Reproduce

Graph your S3 BucketSizeBytes CloudWatch metric for idle databases to observe this.

Expected behavior

We kind of expected the “compactor” to actually compact the data and the size to decrease when there aren’t many changes happening. At the least we’d expect the size to remain relatively constant on a database that is mostly idle. In our load testing environment, which is not idle, the rate of growth seems to be much faster and larger than the rate of data change. It never recovers, even when changes are undone and left to “rest” for a bit.

How did you deploy RisingWave?

We are deployed in ECS.

The version of RisingWave

PostgreSQL 9.5.0-RisingWave-1.7.2 (6ecc083c72b83a96e7e3b58ca3a19b28cd5f74bf)

Additional context

S3 data in & out can be pricey. What is it doing that it just keeps writing “stuff” into S3 and burning money?

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 2
  • Comments: 25

Most upvoted comments

@gshmu Is the issue resolved? If not, please upgrade to our latest version (v2.0) and try.

We don’t have enough information for debugging. When the issue happens next time, please provide the information according to my comment above.

@hzxa21 we using docker-compose run the risingwave. because the disk continues to grow, so we down the containers.

more info: minio is using a mounted nfs disk.

My guess is that either compaction is severely lagging or there is a long running batch query pinning the stale objects. Which version are you using? Please kindly provide the above information when the issue occurs next time.