clipper: Prometheus Related Error When Starting Connection
I’m running clipper on python 3.6 in linux 18.04. I’m running clipper from a container and I share the current docker.sock via a volume. When starting a connection to clipper I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.6/dist-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.35/containers/08610f91d0e81e3ac6b106114adb48ad0d8ba1f49195c59bb4d0e0cf3a0565ed/start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/sabnock/deploy_worker.py", line 83, in work
clipper_conn.start_clipper()
File "/usr/local/lib/python3.6/dist-packages/clipper_admin/clipper_admin.py", line 139, in start_clipper
qf_http_timeout_content, num_frontend_replicas)
File "/usr/local/lib/python3.6/dist-packages/clipper_admin/docker/docker_container_manager.py", line 290, in start_clipper
self.extra_container_kwargs)
File "/usr/local/lib/python3.6/dist-packages/clipper_admin/docker/docker_metric_utils.py", line 111, in run_metric_image
**extra_container_kwargs)
File "/usr/local/lib/python3.6/dist-packages/docker/models/containers.py", line 809, in run
container.start()
File "/usr/local/lib/python3.6/dist-packages/docker/models/containers.py", line 400, in start
return self.client.api.start(self.id, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/docker/api/container.py", line 1093, in start
self._raise_for_status(res)
File "/usr/local/lib/python3.6/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.6/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error: Bad Request ("OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/tmp712ott38.yml\\\" to rootfs \\\"/var/lib/docker/overlay2/cf27191a9d3eb2471e2da0d575afba45d72bf6e13cd021dfe0562f943b6ebcd3/merged\\\" at \\\"/var/lib/docker/overlay2/cf27191a9d3eb2471e2da0d575afba45d72bf6e13cd021dfe0562f943b6ebcd3/merged/etc/prometheus/prometheus.yml\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type")
Do you have some idea of what may be the cause of this error? I looked up related issues but all I found was the same error being caused due to clipper being ran in Windows 10. #688
Any help would be highly appreciated.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 26 (6 by maintainers)
@withsmilo My bad. I will be more careful next time.
@DNCoelho Would you mind creating a new issue about this?
Hi, I also solved the communication problem installing clipper inside a docker image starting from
docker:dind
instead of connectingdocker:latest
todocker:dind
! However, we are planning to useKubernetesContainerManager
for production.Thanks for the support!
sure thing: I created a new issue (#705), you find all the relevant logs there