crawl4ai: [Bug]: Docker compose is not working
crawl4ai version
latest
Expected Behavior
it should work?
Current Behavior
When I follow the document to the last step and try to run it with local profile, I get an error.
document: https://github.com/unclecode/crawl4ai/blob/main/docs/deprecated/docker-deployment.md
Steps; I filled the .env file. I ran the following command and the answer;
root@abc123:~/crawl4ai# VERSION=all docker compose --profile local-amd64 up -d
service "base-config" has neither an image nor a build context specified: invalid compose project
my docker version;
root@abc123:~/crawl4ai# docker version
Client: Docker Engine - Community
Version: 27.0.3
API version: 1.46
Go version: go1.21.11
Git commit: 7d4bcd8
Built: Sat Jun 29 00:02:33 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.0.3
API version: 1.46 (minimum version 1.24)
Go version: go1.21.11
Git commit: 662f78c
Built: Sat Jun 29 00:02:33 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.18
GitCommit: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
runc:
Version: 1.7.18
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
root@abc123:~/crawl4ai# docker compose version
Docker Compose version v2.28.1
Is this reproducible?
Yes
Inputs Causing the Bug
Steps to Reproduce
Code snippets
OS
Linux
Python version
3.10
Browser
No response
Browser version
No response
Error logs & Screenshots (if applicable)
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 22 (10 by maintainers)
I edited the docker-compose.yaml file as it should be. while the base-config approach is good, the “x-” prefix should be used for such sub-parts, otherwise docker-compose will throw an error because it considers this part as a service.
here is the edited version; https://github.com/unclecode/crawl4ai/pull/618/
I am having the same issue using VERSION=all docker compose --profile local-arm64 up -d
Seeking help of docker experts in root causing this issue.
@FractalMind Could you lend us a hand here? 😁
@unclecode I will be happy to help, of course 😃 I checked the dockerfile and docker-compose.yaml file in the next branch. I test it on macOS (15.3) and the result is still the same.
@unclecode I was using the “main” branch. I try out the “next” version a little later.
@unclecode For some reason Chromium wasn’t installed so I rebuilt the container after updating the Dockerfile with Install base requirements RUN pip install --no-cache-dir -r requirements.txt
&& playwright install chromium
Once that was installed it seems to be working well on the RPi 5. I’ve tested several of the Crawl4AI functions and haven’t had any issues so far.
I will probably move it over to my Orange Pi 5+ simply because it has more RAM and compute. I hope there’s no issues on the OPi5 because to OS support is not good.
Here’s the formatted file. I can’t attach a .yml file so you’ll need to change the extension.
docker-compose.md
@unclecode if you want I can build the project on a docker and docker-compose.yaml infrastructure? is there any active work you are doing on this?
@aravindkarnam @unclecode @Natgho
Deployed on a Raspberry Pi 5 using “docker-compose --profile local-arm64 up -d”. Container was built, deployed, and shows Healthy. I have not tested it any further than this.
Courtesy of ChatGPT-01:
"Why it fails
In older versions of Docker Compose (particularly the v1 Python-based version), you could sometimes get away with “dummy” services that existed only to provide config via extends. The new (v2) Compose engine is more strict: if something is listed under services:, it must be an actual, runnable service with a valid image or build context."
"How to fix it
Use an x- config reference (preferred for “base” config) Instead of listing base-config under services:, move those shared settings into a named extension field. Then you can “merge” them into each actual service. For example:
docker-compose.yml:
version: ‘3.9’ # Or omit if you’re using Docker Compose v2+ standalone
x-base-config: &base-config ports: - “11235:11235” - “8000:8000” - “9222:9222” - “8080:8080” environment: - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-} - OPENAI_API_KEY=${OPENAI_API_KEY:-} - CLAUDE_API_KEY=${CLAUDE_API_KEY:-} volumes: - /dev/shm:/dev/shm deploy: resources: limits: memory: 4G reservations: memory: 1G restart: unless-stopped healthcheck: test: [“CMD”, “curl”, “-f”, “http://localhost:11235/health”] interval: 30s timeout: 10s retries: 3 start_period: 40s
services:
Local build services for different platforms
crawl4ai-amd64: <<: *base-config build: context: . dockerfile: Dockerfile args: PYTHON_VERSION: “3.10” INSTALL_TYPE: ${INSTALL_TYPE:-basic} ENABLE_GPU: “false” platforms: - linux/amd64 profiles: [“local-amd64”]
crawl4ai-arm64: <<: *base-config build: context: . dockerfile: Dockerfile args: PYTHON_VERSION: “3.10” INSTALL_TYPE: ${INSTALL_TYPE:-basic} ENABLE_GPU: “false” platforms: - linux/arm64 profiles: [“local-arm64”]
Hub services for different platforms and versions
crawl4ai-hub-amd64: <<: *base-config image: unclecode/crawl4ai:${VERSION:-basic}-amd64 profiles: [“hub-amd64”]
crawl4ai-hub-arm64: <<: *base-config image: unclecode/crawl4ai:${VERSION:-basic}-arm64 profiles: [“hub-arm64”]
Notice: