woodpecker: Server crashes and after a restart SQL lite db is locked

Component

server

Describe the bug

During the execution of a pipeline, often times the server crashes. After crash, when the server is restarted the server is in a very slow state, often times reporting that “db is locked”

When the server crashes the following info is outputted to the logs:

goroutine 33930 [running]: net/http.(http2responseWriter).write(0xc0017c6780?, 0xc00111fea8?, {0x0?, 0x12634a0?, 0x1362a80?}, {0x141f1bf?, 0xc000074d00?}) /usr/local/go/src/net/http/h2_bundle.go:6797 +0x13f net/http.(http2responseWriter).WriteString(0xc00111fee0?, {0x141f1bf?, 0xc000074d00?}) /usr/local/go/src/net/http/h2_bundle.go:6790 +0x28 io.WriteString({0x7f428c48e2b8, 0xc000074d00}, {0x141f1bf, 0x18}) /usr/local/go/src/io/io.go:316 +0x58 github.com/gin-gonic/gin.(*responseWriter).WriteString(0xc00154a000, {0x141f1bf, 0x18}) /woodpecker/src/github.com/woodpecker-ci/woodpecker/vendor/github.com/gin-gonic/gin/response_writer.go:90 +0x73 io.WriteString({0x7f428c48e278, 0xc00154a000}, {0x141f1bf, 0x18}) /usr/local/go/src/io/io.go:316 +0x58 go.woodpecker-ci.org/woodpecker/server/api.LogStreamSSE.func2() /woodpecker/src/github.com/woodpecker-ci/woodpecker/server/api/stream.go:224 +0x134 created by go.woodpecker-ci.org/woodpecker/server/api.LogStreamSSE in goroutine 33971 /woodpecker/src/github.com/woodpecker-ci/woodpecker/server/api/stream.go:208 +0x7a5

System Info

{"source":"https://github.com/woodpecker-ci/woodpecker","version":"2.0.0-rc.0"}

Additional context

No response

Validations

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

looks like an issue in cron handler

I’ll create a pull that add the option to disables cron, if it go away we know it’s this code part and can fix it

Just another update, I captured the logs when db locking presents itself. You can see that the db seems infinetly locked and so everything stops working

image

hmm the easyest way would be to use a framework like https://codeberg.org/6543/xorm_context but mentioned framework needs bit of testing …