clipper: Prometheus Related Error When Starting Connection

I’m running clipper on python 3.6 in linux 18.04. I’m running clipper from a container and I share the current docker.sock via a volume. When starting a connection to clipper I get the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.6/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.35/containers/08610f91d0e81e3ac6b106114adb48ad0d8ba1f49195c59bb4d0e0cf3a0565ed/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/sabnock/deploy_worker.py", line 83, in work
    clipper_conn.start_clipper()
  File "/usr/local/lib/python3.6/dist-packages/clipper_admin/clipper_admin.py", line 139, in start_clipper
    qf_http_timeout_content, num_frontend_replicas)
  File "/usr/local/lib/python3.6/dist-packages/clipper_admin/docker/docker_container_manager.py", line 290, in start_clipper
    self.extra_container_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/clipper_admin/docker/docker_metric_utils.py", line 111, in run_metric_image
    **extra_container_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/docker/models/containers.py", line 809, in run
    container.start()
  File "/usr/local/lib/python3.6/dist-packages/docker/models/containers.py", line 400, in start
    return self.client.api.start(self.id, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/docker/api/container.py", line 1093, in start
    self._raise_for_status(res)
  File "/usr/local/lib/python3.6/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error: Bad Request ("OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/tmp712ott38.yml\\\" to rootfs \\\"/var/lib/docker/overlay2/cf27191a9d3eb2471e2da0d575afba45d72bf6e13cd021dfe0562f943b6ebcd3/merged\\\" at \\\"/var/lib/docker/overlay2/cf27191a9d3eb2471e2da0d575afba45d72bf6e13cd021dfe0562f943b6ebcd3/merged/etc/prometheus/prometheus.yml\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type")

Do you have some idea of what may be the cause of this error? I looked up related issues but all I found was the same error being caused due to clipper being ran in Windows 10. #688

Any help would be highly appreciated.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 26 (6 by maintainers)

Most upvoted comments

@withsmilo My bad. I will be more careful next time.

@DNCoelho Would you mind creating a new issue about this?

Hi, I also solved the communication problem installing clipper inside a docker image starting from docker:dind instead of connecting docker:latest to docker:dind! However, we are planning to use KubernetesContainerManager for production.

Thanks for the support!

@ZaninAndrea Could you tell me error logs? Also, would you be able to create a new issue?

sure thing: I created a new issue (#705), you find all the relevant logs there