postgres-operator: [UI] Backups tab: Error loading stored clusters. Please try again or start over.

When opening the “Backup tab” in the operator ui, I get a message “Error loading stored clusters. Please try again or start over.” Retrying doesn’t help.

The UI makes a request to http://localhost:8081/stored_clusters, which returns 500 Internal Server Error.

Output from kubectl logs deployment/postgres-operator-ui:

--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/operator_ui/main.py", line 190, in wrapper
    return f(*args, **kwargs)
  File "/operator_ui/main.py", line 741, in get_stored_clusters
    prefix=SPILO_S3_BACKUP_PREFIX,
  File "/operator_ui/spiloutils.py", line 252, in read_stored_clusters
    Prefix=prefix,
  File "/usr/lib/python3.6/site-packages/botocore/client.py", line 276, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3.6/site-packages/botocore/client.py", line 559, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/usr/lib/python3.6/site-packages/botocore/client.py", line 605, in _convert_to_request_dict
    api_params, operation_model, context)
  File "/usr/lib/python3.6/site-packages/botocore/client.py", line 637, in _emit_api_params
    params=api_params, model=operation_model, context=context)
  File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/lib/python3.6/site-packages/botocore/handlers.py", line 223, in validate_bucket_name
    if not VALID_BUCKET.search(bucket) and not VALID_S3_ARN.search(bucket):
TypeError: expected string or bytes-like object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/logging/handlers.py", line 936, in emit
    self.socket.send(msg)
  File "/usr/lib/python3.6/site-packages/gevent/_socket3.py", line 390, in send
    return _socket.socket.send(self._sock, data, flags)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/logging/handlers.py", line 857, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib/python3.6/site-packages/gevent/_socket3.py", line 307, in connect
    raise error(result, strerror(result))
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/logging/handlers.py", line 939, in emit
    self._connect_unixsocket(self.address)
  File "/usr/lib/python3.6/logging/handlers.py", line 868, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib/python3.6/site-packages/gevent/_socket3.py", line 307, in connect
    raise error(result, strerror(result))
FileNotFoundError: [Errno 2] No such file or directory
Call stack:
  File "/usr/lib/python3.6/site-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python3.6/site-packages/gevent/baseserver.py", line 26, in _handle_and_close_when_done
    return handle(*args_tuple)
  File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 1482, in handle
    handler.handle()
  File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 454, in handle
    result = self.handle_one_request()
  File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 671, in handle_one_request
    self.handle_one_response()
  File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 935, in handle_one_response
    self.run_application()
  File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 908, in run_application
    self.result = self.application(self.environ, self.start_response)
  File "/usr/lib/python3.6/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/operator_ui/main.py", line 160, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python3.6/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/lib/python3.6/site-packages/flask/app.py", line 1870, in handle_exception
    self.log_exception((exc_type, exc_value, tb))
  File "/usr/lib/python3.6/site-packages/flask/app.py", line 1891, in log_exception
    "Exception on %s [%s]" % (request.path, request.method), exc_info=exc_info
Message: 'Exception on /stored_clusters [GET]'
Arguments: ()
127.0.0.1 - - [2020-04-27 16:55:29] "GET /stored_clusters HTTP/1.1" 500 411 2.097737
10.1.6.1 - - [2020-04-27 16:55:32] "GET /health HTTP/1.1" 200 117 0.001161
10.1.6.1 - - [2020-04-27 16:55:42] "GET /health HTTP/1.1" 200 117 0.002137
10.1.6.1 - - [2020-04-27 16:55:52] "GET /health HTTP/1.1" 200 117 0.001463
10.1.6.1 - - [2020-04-27 16:56:02] "GET /health HTTP/1.1" 200 117 0.000974
10.1.6.1 - - [2020-04-27 16:56:12] "GET /health HTTP/1.1" 200 117 0.001322
10.1.6.1 - - [2020-04-27 16:56:22] "GET /health HTTP/1.1" 200 117 0.001323
10.1.6.1 - - [2020-04-27 16:56:32] "GET /health HTTP/1.1" 200 117 0.001397
10.1.6.1 - - [2020-04-27 16:56:42] "GET /health HTTP/1.1" 200 117 0.001307
10.1.6.1 - - [2020-04-27 16:56:52] "GET /health HTTP/1.1" 200 117 0.001155
10.1.6.1 - - [2020-04-27 16:57:02] "GET /health HTTP/1.1" 200 117 0.001214
10.1.6.1 - - [2020-04-27 16:57:12] "GET /health HTTP/1.1" 200 117 0.002490
10.1.6.1 - - [2020-04-27 16:57:22] "GET /health HTTP/1.1" 200 117 0.001992
10.1.6.1 - - [2020-04-27 16:57:32] "GET /health HTTP/1.1" 200 117 0.001392
10.1.6.1 - - [2020-04-27 16:57:42] "GET /health HTTP/1.1" 200 117 0.002429
10.1.6.1 - - [2020-04-27 16:57:52] "GET /health HTTP/1.1" 200 117 0.001851
10.1.6.1 - - [2020-04-27 16:58:02] "GET /health HTTP/1.1" 200 117 0.003601
10.1.6.1 - - [2020-04-27 16:58:12] "GET /health HTTP/1.1" 200 117 0.001118
10.1.6.1 - - [2020-04-27 16:58:22] "GET /health HTTP/1.1" 200 117 0.001185
10.1.6.1 - - [2020-04-27 16:58:32] "GET /health HTTP/1.1" 200 117 0.001606
10.1.6.1 - - [2020-04-27 16:58:42] "GET /health HTTP/1.1" 200 117 0.001739
operator_ui.main ERROR    Exception on /stored_clusters [GET]
        Traceback (most recent call last):
          File "/usr/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
            response = self.full_dispatch_request()
          File "/usr/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
            rv = self.handle_user_exception(e)
          File "/usr/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
            reraise(exc_type, exc_value, tb)
          File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
            raise value
          File "/usr/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
            rv = self.dispatch_request()
          File "/usr/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
            return self.view_functions[rule.endpoint](**req.view_args)
          File "/operator_ui/main.py", line 190, in wrapper
            return f(*args, **kwargs)
          File "/operator_ui/main.py", line 741, in get_stored_clusters
            prefix=SPILO_S3_BACKUP_PREFIX,
          File "/operator_ui/spiloutils.py", line 252, in read_stored_clusters
            Prefix=prefix,
          File "/usr/lib/python3.6/site-packages/botocore/client.py", line 276, in _api_call
            return self._make_api_call(operation_name, kwargs)
          File "/usr/lib/python3.6/site-packages/botocore/client.py", line 559, in _make_api_call
            api_params, operation_model, context=request_context)
          File "/usr/lib/python3.6/site-packages/botocore/client.py", line 605, in _convert_to_request_dict
            api_params, operation_model, context)
          File "/usr/lib/python3.6/site-packages/botocore/client.py", line 637, in _emit_api_params
            params=api_params, model=operation_model, context=context)
          File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
            return self._emitter.emit(aliased_event_name, **kwargs)
          File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
            return self._emit(event_name, kwargs)
          File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
            response = handler(**kwargs)
          File "/usr/lib/python3.6/site-packages/botocore/handlers.py", line 223, in validate_bucket_name
            if not VALID_BUCKET.search(bucket) and not VALID_S3_ARN.search(bucket):
        TypeError: expected string or bytes-like object

My environment:

# How I created the postgres-operator
kubectl apply -f manifests/configman.yaml
kubectl apply -f manifests/api-service.yaml
kubectl apply -f manifests/operator-service-account-rbac.yaml
kubectl apply -f manifests/postgres-operator.yaml

# How I created the UI:
kubectl apply -f ui/manifests

Everything except backups seem to work.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 3
  • Comments: 23 (3 by maintainers)

Most upvoted comments

@FxKu I just tested it with the extra _backup_list with the prefix and it brings the snapshots “back” https://github.com/zalando/postgres-operator/compare/master...aranair:aranair.test-prefix

Screenshot 2020-12-29 at 6 56 36 PM

Without it, it would always just show “No snapshots found”. ~Do you want me to make a PR of this or are there other concerns with that removal?~

Yet another edit: It seems like after that change, wal.e “sort-of” finds the files but I would see multiple instances of each snapshot shown above^

  • I also rolled it back to the previous implementation in v1.5 and it would still display duplicates, so I suspect the issue may have been around(?)

Separately, the frontend is also having issues querying for DOM elements with ID with / in them.

Does anyone have this all set-up, outside of the main team, who knows what’s the correct format they should be in?

+4
aranair on Dec 30, 2020

The error in the UI could be more user friendly.

It’d be nice if UI would check if corresponding env vars are empty & to allow to set them in the UI. pguibackuperror

+3
vainkop on Sep 11, 2021

I’m using self hosted s3 storage (Minio), where can I enter my custom endpoint? The env var WAL_S3_ENDPOINT doesn’t work.

+1
DrissiReda on Oct 7, 2020

any update on this issue

+1
sachinyadav354 on Jun 16, 2020

I’m also facing the same issue and would really appreciate if someone can help with a workaround. thanks 😃

+1
slayerjain on Apr 29, 2020
Read more comments on GitHub
← postgres-operator: The operator fails over every 30m when sidecar is specified
postgres-operator: While deploying Zalando getting issue- could not create cluster: could not create pod disruption budget →