elastic-ci-stack-for-aws: Intermittent docker error "could not find an available predefined network"

I see these on builds occasionally:

$ docker-compose -f docker-compose.yml -p buildkite9c0e505a3f5844e4bde33ee36c928627 run node pug-lint app/assets/javascripts
Creating network "buildkite9c0e505a3f5844e4bde33ee36c928627_default" with the default driver
ERROR: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network

Here’s the Dockerfile I’m using:

FROM node:4.5.0

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY package.json /usr/src/app/

RUN npm install
RUN npm install -g eslint babel-eslint pug-lint

COPY . /usr/src/app

and the pipeline.yml:

steps:
  - name: ":docker:"
    plugins:
      docker-compose:
        build: node

  - wait

  - name: ":eslint:"
    command: "eslint --cache app/assets/javascripts spec/javascripts"
    plugins:
      docker-compose:
        run: node

  - name: ":dog:"
    command: "pug-lint app/assets/javascripts"
    plugins:
      docker-compose:
        run: node

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 30 (18 by maintainers)

Most upvoted comments

To give a bit of feedback on my experience with this exact error…

I’ve come across it both in buildkite and in my local development environments. It is indeed related to too many networks. We had 1-2 custom buildkite jobs which were using our own scripts for startup/shutdown and so we were leaving behind a few networks. As soon as we fixed those custom jobs, these errors went away 100%.

The limit seems to be ~30 (at least on my local machine) + bridge/host networks as can be seen in:

$ for count in `seq 1 35` ; do echo -n "$count: " ; docker network create test$count ; done
1: 02b8f09541f28f1a0d5487d649de07a32a01524d20632f967ce2a928369db497
2: ca4a53df632735e7703b5e33c5ffc84d3873bfe883199509d7ef65bd59c52a98
3: 607219dc2ddc3c69d606137d3a0054b6ad56c7d6b4b3c82c514bc4bf91668d84
4: 679d79c5aee9a75f595130f7a8c017df88874475c84a3b486a54c26a9c3f1dbf
5: 96cbac056a95c8ee3009cd5daaecf634f6f3c93d8e1bc8c2e4c0010ecae617dc
6: 0c83931ae3557e52f23d93026b545653b6feeac9b3537148bcc29249ed040887
7: c15651836c7580e266c3b2903f1d786a1ccdb7a2cbc9f2327a90d331835d9920
8: 65947d9cf7f81393c4f235d080211d9954cefeac402fd217be4ce142de110dc3
9: 66314a00fbcdb2517f7a86fa0fe3b1cba381a4e8dbcd1315d34f18f585cd4a18
10: 70ae74d7c00efe3cf29ad3ce2813b9df81e7c7064262c3fa41362047abacb297
11: a677989da5731d31b660e8831f4f5682db160742ae850d7f2e5ce7e77b65d932
12: 274c41ea932bd40eb92572b48912c1522570eb21a45e5090af2381471fad5e9a
13: fbb1f756b1a92bd8e673d12db595897170baf08a898036498d93830eda3a4a31
14: 0a67a72a2bc4460c1c9db16f5d27efe21c714b1a0f8fc2a1c26fd0e9207553e0
15: 0049f09c3de8ba48d435dc7fab71adb5a39cd8d9cd6d5ee7bb7bd3f6e6b72691
16: 557aea91dda8f5988c9e3661f5377ac861c9f5e7862bb4e5bd02aa120885bcaa
17: 04e09bea616d09a0f082797d8783924d759d36c29cd411c48bbe5ed1aa242dc0
18: 8a151874a9002f1f9add48056f5974d2a5867bf16219a8b4a4b4ce4c3b679ff6
19: 61b5cd5632c04e3951a2d22e000db67b36e25ea8683d9d6f74cd111f9888927c
20: 6cea8e771f3699ae232d3f04ed40078e0d8c0e2196715df3e1ff645dc35ac433
21: 0046e52998c0d2ef213cb400bb0a44899c779782848b49e91f8719f0aacccc48
22: d58ec78a1731b482f90b3c467c220c2b40cbe8c9d068ed98686ee32f39fde1f2
23: 7542c46e1e4f99e68278188a7a50aaceb764dc65301393b93189349f9462fbf4
24: 6889b21c4e8a5776652fe54812b00dd36a48445bdcddc15a5e1284944a5b4c11
25: 04bdf9e51e2d373aa98fe11ac3cb57722a0fcc5a1a17322f8b8d28218d2ad3e7
26: 63e6c1637da62f5d4fbc34f824b5e978bb24dbccb78b1d57cca1078b7bf4f3d6
27: b65828bca18d06e1428985360a617d37294e5427b806bdce248c86f99cbe9a76
28: d950c8933a8ca48ed33773cfe08c901a04030f2f77e9c919fcda7cac05e47e46
29: 677cc64a291f934a7fd637162c46f2955d07b5bd9e508f185ad312026b7b2808
30: b4c25ea1c6eadeff3031092e10e0cdefc76481b0a411f38aa34e44e7c7390ffc
31: Error response from daemon: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network
32: Error response from daemon: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network
33: Error response from daemon: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network
34: Error response from daemon: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network
35: Error response from daemon: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network

The fix for this is not related to buildkite at all, but if you wanted to fix this on the buildkite side you could periodically check for all networks, see if any jobs are currently running, and if not cleanup those orphaned networks.

deppy on Mar 8, 2017