prefect: prefect server start failes to start graphql container on 0.14.10 (but 0.13.19 works)

Description

I am a first time user of prefect - switching over from airflow.

Just downloaded the latest version 0.14.10

I tried running

Pulling postgres ... done                                                                                                                                                                                  
Pulling hasura   ... done                                                                                                                                                                                  
Pulling graphql  ... done                                                                                                                                                                                  
Pulling apollo   ... done                                                                                                                                                                                  
Pulling towel    ... done                                                                                                                                                                                  
Pulling ui       ... done                                                                                                                                                                                  
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/                                                                                     
Creating network "prefect-server" with the default driver                                                                                                                                                  
Creating tmp_postgres_1 ... done                                                                                                                                                                           
Creating tmp_hasura_1   ... done                                                                                                                                                                           
                                                                                                                                                                                                           
ERROR: for graphql  Container "9c870d315bbc" is unhealthy.  

With this error. I retried a bunch with same error.

After switching to version 13.19 it worked immediately.

Expected Behavior

I expected the graphql container to start successfully.

Reproduction

conda create -n testenv python=3.8
pip install prefect
prefect backend server
prefect server start

Since I already changed to 0.13.19 I can’t really run ‘prefect diagnsostics’ anymore… I used a clean conda env with only Kedro and Prefect installed (via pip) on Ubuntu 20.04.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20 (9 by maintainers)

Most upvoted comments

I have the same issue with Hasura apparently not being able to start:

towel_1     | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n    yield\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n    timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n    request, auth=auth, timeout=timeout, history=history\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n    response = await self.send_single_request(request, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n    timeout=timeout.as_dict(),\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n    method, url, headers=headers, stream=stream, timeout=timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n    self.socket = await self._open_socket(timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n    hostname, port, ssl_context, timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n    return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n    stream_reader=stream_reader, stream_writer=stream_writer\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n    raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.18.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n    return await self.reschedule_flow_runs()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n    limit=5000,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n    await self.gen.athrow(typ, value, traceback)\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n    raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
{
  "config_overrides": {},
  "env_vars": [],
  "system_information": {
    "platform": "macOS-10.16-x86_64-i386-64bit",
    "prefect_backend": "cloud",
    "prefect_version": "0.15.6",
    "python_version": "3.10.0"
  }
}
docker version
Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:20 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:53:48 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
docker-compose version
Docker Compose version v2.0.0

I’ve also tried the docker-compose disable-v2 to no avail. I’m on Apple M1 and I’ve given 4GB RAM to Docker Desktop (I cannot allocate more)

I’d recommend setting a higher memory limit. I’m glad it was as simple as resetting your config though!