neon: No such file or directory errors reading layer files

This happened on the CI with the new test from PR #1735. I think we’ve seen this a few times before, but couldn’t find an existing issue for it

https://app.circleci.com/pipelines/github/neondatabase/neon/6512/workflows/cb7bf6ab-11db-440c-88fa-3c704e9f6fca/jobs/65793/artifacts

2022-05-19T08:03:47.313535Z ERROR compact loop terminated with error: Failed to load delta layer /tmp/test_output/test_pageserver_chaos/repo/tenants/775a6bca9dcc4d5baf1db289176bbd7c/timelines/e3aa3049715941f858d26f6c42c375f6/000000067F000032AE000040000000003707-000000067F000032AE000040000000003792__000000001E7EFFE9-0000000021371F61
...
2022-05-19T08:03:49.189238Z  WARN could not remove ephemeral file '/tmp/test_output/test_pageserver_chaos/repo/tenants/775a6bca9dcc4d5baf1db289176bbd7c/timelines/e3aa3049715941f858d26f6c42c375f6/ephemeral-6': No such file or directory (os error 2)
2022-05-19T08:03:49.190033Z ERROR Thread 'layer flush thread' tenant_id: Some(775a6bca9dcc4d5baf1db289176bbd7c), timeline_id: Some(e3aa3049715941f858d26f6c42c375f6) exited with error: No such file or directory (os error 2)

_Originally posted by @hlinnaka in https://github.com/neondatabase/neon/issues/1735#issuecomment-1131463161_

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 16 (16 by maintainers)

Commits related to this issue

Add test that repeatedly kills and restarts the pageserver. This caught or reproduced several bugs when I originally wrote this test back in May, including #1731, #1740, #1751, and #707. I believe al... — committed to neondatabase/neon by hlinnaka 2 years ago

Most upvoted comments

The new error seems to be because of WAL receiver disconnection:

It seems to be not the source of the problem, but result of test termination because of timeout expiration.

knizhnik on May 20, 2022