spkrepo: Unable to publish package (500 Internal Server Error)

When attempting to publish a package the following is reported:

HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 141
Content-Type: text/html
Date: Sun, 23 Apr 2023 12:02:35 GMT
Server: nginx/1.23.2

<html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

This appears to be happening consistently for moderately sized SPK files (50-75 MB) and does not appear to be based on time of day (server load).

About this issue

Original URL
State: closed
Created a year ago
Comments: 37 (37 by maintainers)

Most upvoted comments

I think you may be correct. I did the new run and got this in one of the logs:

Package published successfully
{"package": "prowlarr", "version": "20230427-6", "firmware": "6.1-15047", "architectures": ["apollolake", "avoton", "braswell", "broadwell", "broadwellnk", "broadwellntbap", "bromolow", "cedarview", "denverton", "dockerx64", "geminilake", "grantley", "purley", "kvmx64", "v1000", "x86", "x86_64"]}

But then right after, I get the following in all the other archs:

ERROR: Failed to publish package - HTTP response code 500

We may have the same situation again. If you can, please remove prowlarr v6 again because I suspect we have re-created the same situation. I will try a manual upload one file at a time once you have.

Edit: sorry I’m not sure, but I deleted v6 again, and it is now gone.

Okay, thanks. I’ll retry, one at a time now.

mreid-tt on Apr 27, 2023

For some history. The first Ci used travis and CircleCI processed one arch at a time. Also it was expected that every dev build on their own devices before publishing so that they would test the build locally as well (that hasn’t changed) but with GitHub actions it became easier and we implemented the publishing part as well. So if you build locally and test the one or two architectures you don’t need to build all of them to publish. Lowering the barrier and time spend building a bit.

publicarray on Apr 30, 2023

@publicarray, looking back at the application logs you kindly provided, I believe this new theory is correct. In the logs we first see this error:

[2023-04-16 12:32:53 +0000] [7] [ERROR] Error handling request /packages
Traceback (most recent call last):
  File "/usr/src/app/spkrepo/views/api.py", line 261, in post
    os.mkdir(os.path.join(data_path, package.name, str(version.version)))
FileExistsError: [Errno 17] File exists: '/data/lidarr/11'

Immediately after that we have these errors and similar:

[2023-04-16 12:33:40 +0000] [10] [ERROR] Error handling request /packages
Traceback (most recent call last):
  File "/usr/src/app/spkrepo/views/api.py", line 264, in post
    build.save(spk.stream)
  File "/usr/src/app/spkrepo/models.py", line 357, in save
    with io.open(
FileNotFoundError: [Errno 2] No such file or directory: '/data/lidarr/11/lidarr.v11.f42661[comcerto2k].spk'

I believe this supports the theory that:

Two workers attempt to create a directory at ‘/data/lidarr/11’
One succeeds and the other fails
The failing one deletes the directory path
Other workers saving builds fail because the path does not exist

mreid-tt on Apr 30, 2023

We had problems uploading before (due to load) but never like this (missing folders/files)…

Hmm, okay. Perhaps a possible fix would be to adjust the code to verify that a folder is created (for both the package and version functions) before committing a new entry to the database.

mreid-tt on Apr 27, 2023