compose: Define services which are not started by default

1. Add an option to services to make them not start by default

I vote for Option 2. e.g. something like a start: false directive. The advantages are that we avoid the need for multiple compose.yml files or extra config files, you can just read one compose.yml to get the feel of the whole app stack.

Completely agree with @campadrenalin.

This whole discussion, as I understand it, is basically a large number of people providing a large number of very good reasons to implement Option 2, and the developers refusing because somebody hypothetically could potentially misinterpret the existence of this feature and request some other unrelated features.

I clearly pointed out in my earlier post why the team’s official position was that you should not use your Compose file as a Makefile. If you believe we’re wrong on this and still want to do so, that is your prerogative, but it is a bit preposterous to insist that we add features to support it.

👍 I routinely find myself wanting to create a tests container alongside my other services to encapsulate running unit tests. My “workaround” is to set the command to “/bin/true” and run the container with the unit test command specified on the CLI. Being able to specify which containers should start at up time would be great!

(btw, nice work all around, folks)

cc @jameydeorio

@shin-

In your example, just having a docker-compose.phpmyadmin.yml that can optionally be included is no more burdensome or complex (in terms of instructions “developer guides”) than setting an environment variable / modifying a service definition inside the main file.

Actually I didn’t think about involving environment variables here yet. My desired workflow “optional services” would finally allow is as follows:

• I copy/clone a new project
• I see there is a (single) docker-compose.yml file in it, therefore I know, all I have to do is run docker-compose up and the application is built and run
• Now I want to start one of the optional services for development purposes (e.g. additional debugging, phpMyAdmin). All I have to do is docker-compose up phpmyadmin.

For me this is much more convenient than using multiple docker-compose.yml files, and as I understand for most other people here too. The benefit of having “optional services” here is

• for the developer not having to duplicate anything between multiple files or having to carefully consider which definitions to put in which file, but only having to set one boolean flag (auto_up: false) on a service
• for the user not having to know/remember which files to include for which services: do I need to run
  • docker-compose -f docker-compose.phpmyadmin.yml up
  • docker-compose -f docker-compose.yml -f docker-compose.phpmyadmin.yml up or even
  • docker-compose -f docker-compose.phpmyadmin.yml -f docker-compose.yml up?
• instead the only thing the user needs to know is that there is a service called phpmyadmin and the user knows immediately how to start it.

See above – as soon as this feature becomes available, people will start using it for things other than its intended purpose, and expect us to implement a new featureset to support it

Don’t get me wrong, I not only understand your concern here but agree with it: there will always be users who bend this - or any other (!) - feature to fit their desired workflows even though the features were never meant to be used like that. BUT for any request on “optional services” missing features of a “task” you can then easily argue that services are not tasks and tasks need to be introduced as a new concept first and are therefore not supported (yet). I.e. anything more than just marking a service as optional is a step towards the concept of tasks (e.g. option 4) and there is a clear cut between.

With the feature of marking a service as optional we’re still on the right side of that cut. “The feature per se is not too complex to implement but it might tempt users to use it wrongly” currently being the only real (although valid!) argument standing against this feature is quite unsatisfactory as a reason not to implement it - at least for the users who could get much of convenience out of it while using it the correct way™.

I am up to rebasing my pull request #3047 to current master and do any required polishing for this to get accepted, please just let me know 😉

Hehe, I remember the fierce discussions we had when we were lobbying for allowing “browsers” to display images, with the main argument of opponents being that it’s called hyper text transfer protocol. Sorry for the OT reminiscings of an old fart.

On docker-compose version 2, every container is declared inside the services: top-level-item. Declaring a service, with start: never to run a script sounds wrong. So considering the new format, shouldn’t we declare an extra top-level item apart from services, volumes and networks?

My proposal:

• add a new scripts:, transients: (or think a better name) top level item to v2.
• inside transients you can define a container as you would do with any service
• the only difference is that they are not going to start by default, since they are intended for transient use only.

Example:

version: "2"

services:
  web:
    image: myapp
    networks:
      - front
      - back
    volumes:
      - /usr/src/app/
  redis:
    image: redis
    volumes:
      - redis-data:/var/lib/redis
    networks:
      - back

scripts:
  bower:
    image: my_image_with_bower
    volumes_from: web
    working_dir: /usr/src/app/static
    command: "bower"
// maybe would be great to place something like "bower $@"
// to define where you want the cli arguments to be placed, by default at the end.

volumes:
  redis-data:
    driver: flocker

networks:
  front:
    driver: overlay
  back:
    driver: overlay

And then you could run:

docker-compose script bower <EXTRA_CMD_ARGUMENTS |  default nothing>

docker-compose script bower install

Concerns:

• It sounds OverPowered to create another top level item just for this purpose of managing start, but it’s derived by the fact that the new format is declaring the behaviour in the top level.
• I’m trying to imagine extra flexibility (and problems) this may add.

Lastly, on the groups feature, sounds nice but if you have that many grouping I don’t see the problem on creating docker-compose files for each one of them. Probably the feature that you want is docker-compose file inheritance, that would be awesome!

PD: @bfirsh maybe if you like the idea you can add it to the “Possible Suggestions”

On second though this is a revamp of the 4th suggestion because of the imminent new format declaring services.

Someone once told me: “If several smart people cannot decide on what approach is the best, usually neither is pretty bad. The worst is not doing anything.”

This card seems a like a prime example of this 😃

The start: true|false approach is too limited. What if some services are used for testing, some others for admin and the rest for normal operation?

I would prefer adding the notion of grouping to services.

    myservice:
       group: `admin`

If the group attribute is not defined, default is assumed. That way we could start the default and admin services using docker-compose up -g admin -d.

Better yet, make groups an array.

@cpuguy83 That would seem to imply the whole service is disabled, even for run. I’d find it confusing.

@qcho hmm, now that I’ve refamiliarized myself with Docker since 1.6, I can see what you and @gittycat are talking about. In that sense, I really like @gittycat 's approach. I picture (blue sky) an interface like:

groups: # if no groups, all services are in the "default" group by…default
    - foo
    - bar
services:
  foo:
    image: imagine
    groups: foo
  bar:
    image: energy
    groups: bar
  baz:
    image: money
    # no groups, so "default"
   quux:
    image: word
    groups:
      - bar
      - default

Meanwhile, in the shell…

docker-compose up -d # Up all services in default group, so 'baz' and quux
docker-compose up -d --groups foo # Up all services in the foo group
docker-compose up -d --groups foo,default # You get the idea
docker-compose run --rm bar somecommand # We never started 'bar', so it can be a one-off command

An approach like this would be awesome and obviate the need for this ticket, but does go beyond its scope.

As docker-compose seems to be dropped in long term in favor of docker stack

So, if docker-compose is going to be deprecated in the long run

wait, wat? docker-compose is central to a lot of people for their development workflow, which hardly needs a swarm manager. Could you point me at some official statement by Docker on this regard?

+1 for option 1

I think adding services that aren’t actually part of a composition, but just happen to need to link/attach to it, is a bad design choice. There are a few other use cases that would be solved by allowing some form of includes syntax. Solving multiple problems with a single feature is always nice.

Some of these issues (the ones that deal with data-only containers, #942, the last comment from @cpuguy83) are actually already fixed by #1754, I don’t think we need to consider them a problem any more (after 1.5).

@paulodiovani That looks more like a hack than a solution.

I think the discussion here, the many example use cases as well as the fact that there are frequently duplicate issues to this opened makes quite clear that there is a real demand by many user for a possibility to “Define services which are not started by default”.

Therefor I’d really like to move the discussion here from if this should be implemented more to how this should be implemented. Could we maybe all agree on this?

Regarding on how this could be achieved @bfirsh listed some options in the opening comment:

1. Recommend users to use a separate Compose file

This is the only currently available solution for this use case. But as said above it is in this case more of a workaround since it requires manually typing long command lines and remembering which Compose files to include for which command to run. Of course, in a CI setup or automated deployment to production this is not an issue, since the command line has only to be defined once and never be typed out manually. So this is solely about convenience in non-automated environments, i.e. development, which is according to the docker-compose manual one of the main use cases of Compose itself:

Together, these features provide a convenient way for developers to get started on a project. Compose can reduce a multi-page “developer getting started guide” to a single machine readable Compose file and a few commands.

Having to type something like docker-compose -f docker-compose.yml -f docker-compose-utils.yml run init just to initialize a local setup does not sound convenient to me but more like something to be found in a multi-page “developer getting started guide”.

And to consider option 1b) here:

Recommend users to use a wrapper script/tool to use separate Compose files

If I have to pull in additional tools and dependencies just for this, why use docker-compose in the first place? Since everything docker-compose does could - in theory - also be done with just a custom shell script and the native docker client directly. I see the main advantage of docker-compose in it’s standardized user interface (I clone a project, see a docker-compose.yml in there and know I just have to run docker-compose up to get started). Always having to know which compose file to use for what weakens this advantage drastically.

1. Add an option to services to make them not start by default
2. Add a top-level configuration option to define the default services

These two options do basically the same: define which services should be (not) started by default in one compose file, option 2) by blacklisting individual services, option 3) by globally whitelisting a set of services. Considering that most of the time there are more default services than “one-off services” in a single compose file it’s probably easier to blacklist one service than having to whitelist all the other ones and having to update the whitelist every time a default service gets added or removed. Also with option 2) merging multiple compose files is probably easier.

1. Add a concept of a thing like a service, but is just for one-off commands (“scripts”, “tasks”, etc…)

I think option 4) is just a specialization of option 2) or 3) since in most of the examples above the desired one-off containers are basically defined just like any other service and do nothing more or less like normal services - except that they’re not started by default. (Or did I miss some significant difference to a “service”).

So in my opinion option 2) is the best pick here, since it requires the least change but at the same time is flexible enough to satisfy most (or all?) use cases mentioned here and be very convenient to use at the same time. Nevertheless I would be equally happy to see anything like option 3) or 4) merged into upstream. So after over a year of discussion in here, is there any timeline when or if this will happen??

I give up docker on production, there a lot of problems. Then i migrate to vagrant.

@briceburg see the discussion above: the issue here is not about a missing functionality which couldn’t be done any other way. There are workarounds for almost everything people here want to achieve. But they’re workarounds and dirty hacks instead of a normale use case of docker-compose.

It’s about convenience, about not having to type out docker-compose -f docker-compose.yml -f docker-compose-tools.yml upby hand on the cli, it’s about not having to remember whether it was docker-compose run tools tests or docker-compose run utils tests or even have to rely on a multi-page “developer getting started guide” but instead having a unified and simple interface for the developer.

Again, CCing @bfirsh, @dnephin: could you please provide any feedback to this issue? Is there any hope we could see something like this on the coming roadmap for docker-compose??

@wangwenpei well, #3047, there’s the pull request… waiting to get at least some feedback on it for over two years now. It was closed recently

for the reasons highlighted in https://github.com/docker/compose/issues/1896#issuecomment-322285976

which quite angers me to be honest because the arguments given in https://github.com/docker/compose/issues/1896#issuecomment-322285976 do not relate in any way with the pull request:

1. What is a service? By design, a service is a clearly defined, long-running process that interacts with other services to form an application. That is the basic assumption that Compose operates under and it guides our design for most, if not all features we implement.
2. By that definition, a “service that is not started by default” is not really a service [1] - it’s more of a pre-defined task, running in a container and acting on the application, rather than being part of it.

I don’t see the assertion By that definition, a "service that is not started by default" is not really a service to be true. A service that is not started by default is just a service that is not started by default, nothing more, nothing less at this point. Being started by default or not doesn’t say anything about the service being long-running or not, being part of the application or not, it only says that is not to be started by default.

And that’s also exactly what the change in the pull request does: add the ability to mark a service to be started by default or not. There is no need to introduce a new concept such as tasks to do this. In fact all the other arguments in https://github.com/docker/compose/issues/1896#issuecomment-322285976 follow from the false premise of 2. and are thus not applicable to the pull request which does not try to do anything else.

Just for completeness:

[1] Some people do mention wanting to have “optional services”, but that’s not the most common use case mentioned in this thread. Most people want the “task” feature, as far as I can tell.

Yes, there were examples of one-off tasks, but there also were examples for real services according to your definition, i.e. legitimate use cases for this feature. Being able to use this to also run one-off tasks I can not really accept as a valid argument against this. It is already possible to define a service which is not long-running, not exposing any ports and is not an essential part of the application in a totally valid docker-compose.yml.

In fact this is also one of the suggested workarounds! Just split the tasks into a separate docker-compose.yml. The other workarounds mentioned are to use your own home-made (platform-dependent) scripts, which totally contradicts the self-stated purpose of docker-compose, see my comment above, or to simply switch to another tool, i.e. throw away your perfectly well working toolchain just to get this one tiny feature in convenience which could easily be implemented in your current toolchain (docker-compose).

The only real argument brought up against the most basic implementation of option 2 - i.e. having some way to tell docker-compose to not start a particular service by default, and nothing more, no new concepts or anything - is that this could be_somehow_ be misused by some users. But this argument is quite weak compared to the demand for it and the legitimate use cases brought up here…

@shin-

but also because we [3] simply don’t have the time or resources to implement or maintain what this feature would need to become to be truly useful to people. [3] And at this point I mean “I”, as I’m the only maintainer actively working on docker-compose at this time.

On the one hand highlighting that there is only one active maintainer while on the other hand just ignoring pull requests and therefore wasting any opportunity to maybe get other developers involed in the project isn’t something I could comprehend. The code changes provided in #3047 already do implement a feature which would be truly useful to [many] people - maybe not all the people with all of their use cases, but most of the people here.

All the additional requirements - having to introduce completely new concepts, new dependency management, up to a cmake-like complexity - simply follow from a false premise!

@shin- I strongly agree with @ekkis: not being started by default does not violate the definition of being a service, therefore your point 2. is an assumption which is not necessarily true and so are the consecutive points since they’re based on the assumption of 2.

For example: when I develop an application using a mysql DB I usually define an additional service phpmyadmin. This is a true service: long running, exposed ports, dependent on other services. But I don’t want do start this service by default but only on demand/in development. Currently I realize this by putting this service into an additional docker-compose.yml and starting it by providing both files to docker-compose - just as you suggested above.

So, yes, there already are possible ways to achieve this; be it by splitting the services into multiple files or simply using a tool other than docker-compose. But as I wrote already above: this issue is not about an entirely new feature/functionality or a new concept (i.e. tasks) but about convenience, especially in development environments. Having to type out docker-compose -f docker-compose.yml -f docker-compose-tools.yml up is anything but convenient and so docker-compose does not deliver on its own promises:

Together, these features provide a convenient way for developers to get started on a project. Compose can reduce a multi-page “developer getting started guide” to a single machine readable Compose file and a few commands.

Which services to start or which yml files to use in which environment is something typically to be found in multi-page “developer getting started guide[s]” - if to be found at all…

So in my opinion and as @jimzucker wrote most +1s here are really for “optional services” rather than a new concept of tasks. At least all of them are better off with a simple boolean to flag a service to be started by default or not.

As my PoC shows the required code change - for “optional services” - is quite manageable and provides for most people here a huge benefit in convenience. Option 2 - read as “possibility to mark a service optional” - is a reasonable compromise to solve this issue.

@shin- wouldn’t you agree on that giving users the possibility to mark services as optional - but them still being services - would greatly help them? When reduced to “marking services as optional/not started by default” would this issue/a pull request for this be accepted and merged?

It’s been over a year of discussion and we seem no further forward making a decision. @bfirsh should just pick a style and go with it 😄

I don’t think it’s necessary to introduce a new concept for tasks. These read very well in my opinion and are possible without many modifications:

// 'task' being a service that manages all my tasks, probably with some custom entrypoint script
docker-compose run task tests

// 'tests' being a service specifically for testing with the default command set to run my tests
docker-compose run tests

The current “solution” with the separate compose file is not really a solution in my opinion. As @xaka said, nobody wants to type all this:

docker-compose -f default-file.yml -f additional-tasks-file.yml  run task myTask

What you’ll end up with is a ./run-task script that adds all the boilerplate before myTask for you. But now you have added another entrypoint and interface to your multi-container-app. Developers see docker-compose.yml and think: “Oh great, a compose app. I know how to handle this thing!” And now you have to tell them: “Right, you can manage it with docker-compose like every other compose app… oh, but wait… there is also this additional script you need to know…”

start: false / up: false / some-additional-flag-to-a-service: false is probably the simplest thing to implement, but most likely also the clearest and easiest to understand. And it would improve usability so much.

This is an old thread, and I agree that docker-compose shall not care about services that won’t run from start.

But I also commonly add a test service, so here is what I did:

1. Override the entrypoint of the service.

2. In the new entrypoint, check if there is a tty available (docker-compose run uses --tty by default while up doesn’t).

   # Check if there is a tty
   if [ ! -t 1 ] ; then
     echo "No tty available."
     exit 0
   fi
   

   This will always make the service exit when run without a tty (with docker-compose up, for example) while still runs as expected with docker-compose run

@shin- Though I think we all understand your point and sympathise with it, I don’t think “it might be misused” is a good argument for something with - clearly - many viable use cases. Of course it will be misused. That’s exactly how software and new ideas grow; out of using technology not entirely as it should be. The web wouldn’t be what it is today if we hadn’t been misusing all provided technology for the past 25 years.

Anyway. According to you - simplified - the difference between a service and a task is the life time. Makes sense, and I understand you view them as two different concepts and feel it’s a potential can of worms. But this distinction is nowhere clarified or expounded upon in the documentation. Nor is it anywhere pointed out that docker is not meant to be used for spinning up temporal processes. What I mean to say is that if the docker team has a strong stance about this, it should be made clear from the get-go. Also. Task or service, there’s clearly an overlap. And marking a service as optional is definitely not something that belongs uniquely to the task-side. From a practical point of view I definitely agree with @acran - having a single docker-compose file is far better DX, simplifies a lot and is less error prone than having multiple docker-compose files.

Is there any way we in the open source community can help contribute this feature which seems heavily requested?

There are so many +1 on option x, that I can’t even find the original statement. Please use the reaction buttons under posts instead.

To add a new idea:

Can’t we just set a default scale to 0?

web:
  image: busybox:latest
  command: echo 'scaled'
  scale: 0

I think a lot of people that are requesting this feature do not want to run tasks but optional services. For example, if you have a set of microservices running as Docker containers and you deploy all of them as a whole service, maybe some day your requirements change and you do not need to run all the containers in all your servers because for that case you do not need a specific functionality that is included in a non mandatory container.

Additionally, i think this is a nice feature for development environment; if you are fixing a bug in a specific container you do not need to start all the containers for it.

At this point it seems apparent option 2 is what users want. What are the next steps for making this happen?

@gittycat compose still can have the option of whether to tell engine to start a service or not. This is a very valid request still.

Another vote for option 2. There’s even a PR for it (https://github.com/docker/compose/pull/3047) that @acran was kind enough to create, so the effort from the compose devs should be fairly small, as they only need to review it (and it’s pretty short and simple) and not write any code.

If the compose devs don’t want to do this, then they should state it once and for all and then close the ticket and lock it to prevent comments.

+1 for a feature request. hacks like having commands that do nothing means even if the container doesn’t run, it has to still be built, initialized etc…

My use-case is that we don’t push all our images to docker hub or quay, nor do we have our own registry, and being able to mark a service as not-to-be-run-by-default means we can have a base image defined in docker-compose.yml that doesn’t get started, and then other containers using that or building upon it.

I would be inclined to use a docker-compose fork that implements this feature just for this singular feature.

@jimzucker No - because the problem here is not the implementation ; a basic implementation of this would take half a day for your average developer. What we’re concerned about here is a fundamental design issue.

1. What is a service? By design, a service is a clearly defined, long-running process that interacts with other services to form an application. That is the basic assumption that Compose operates under and it guides our design for most, if not all features we implement.
2. By that definition, a “service that is not started by default” is not really a service [1] - it’s more of a pre-defined task, running in a container and acting on the application, rather than being part of it.
3. Hence it is wrong (fundamentally by design) to call it a service. At that point we need to consider creating a new type of definition in your Compose file (I’ll call it “task” for brevity, the exact terminology does not matter here).
4. Since tasks are inherently short running processes, they use a different set of options compared to services. For example, we can intuit that tasks do not need to expose ports [2]. On the other hand, an option like CID file which is mostly pointless for services could be really useful for tasks.
5. Now, in most projects, tasks tend to have dependency chains (that’s why in traditional, non-container world, we have Makefiles) - so we need to have at least a basic implementation of dependency chains, with parallelism, and being able to tell if that one JAR build task that always takes 10 minutes really needs to run again.
6. Then, some services actually depend on tasks having run before being able to start, adding another level of complexity to how we calculate and handle dependencies.
7. And some tasks need some services to be up in order to run (e.g. database migration).

At this point, this has become an entirely new piece of code that probably matches or surpasses cmake in terms of scope and complexity. This is why we’ve resisted implementing this for so long - partly because this is not at all what Compose is designed to do, but also because we [3] simply don’t have the time or resources to implement or maintain what this feature would need to become to be truly useful to people.

Hopefully this clarifies things a bit more. I know people have been upset and disappointed with the response (or lack thereof) on this thread, and I’m not sure this will alleviate that in any way, but there it is.

[1] Some people do mention wanting to have “optional services”, but that’s not the most common use case mentioned in this thread. Most people want the “task” feature, as far as I can tell. [2] I’m sure someone will contradict me with an example of a task that does need to expose ports, which will only go to show that the exercise of deciding which options are applicable to each set is not trivial. [3] And at this point I mean “I”, as I’m the only maintainer actively working on docker-compose at this time.

It’s so simple. scale: 0 https://github.com/docker/compose/issues/1661

pinging @bfirsh @dnephin: Could you give an update regarding this issue? Since most comments here are in favor of option 2, are there currently any plans to implement something like that (or option 3/4)? I could polish up my pull request (#3047) and complete it with tests and documentation if you’d consider merging it then.

Thank you.

Thats the point, why should I put a container that will never run a service under a “service…disabled” category. Sounds like an ugly patch someone made and It’s not intuitive.

Maybe the “version 2” of the format should have taken this issue into account. For example another spec could be

version: 3?
containers:
  webserver:
    ...
  database:
    ...
  cache:
    ...
  some_script_container:

services: (they are groups):
  production:
    webserver:
      ...
    database:
      ...
    cache:
      ...
  development:
    webserver:
      ... DEFINE SOME CUSTOM DEV STUFF ABOVE basic container definition
    database:
      ...
    cache:
      ...     

Ok now we have proper services, with groups definition. I can start production or development service group. Or just run a script in the some_script_container. since it’s not defined in any services no one will be started

This doesn’t necessarily make it the right choice, but I think learning curve/ease of understanding should be considered.

I feel like (2) is the easiest to comprehend. I don’t really have a way to confirm this, but my gut says most people who aren’t intimately familiar with all of docker-compose’s options that run into the problem we’re trying to solve here say, “I wish there was some way to get that container to not start when I run docker-compose up,” and they see start: false and bam, we’re done and happy.

They don’t say, “If only there was a way I could create a 2nd file with an awkward linking story to solve this…” (tho I realize that https://github.com/docker/compose/issues/1987#issuecomment-139632238 helps with the “awkward linking story,” ya?).

(4) was kind of vague, but a dedicated avenue for the way for scripts and one-offs like this fits this “makes sense” bill.

@kogli I think the problem here is to pick an option. All have Pros and Cons. As I said before:

Every solution will be better for some use cases, and worse (Or even useless) for others. If the something is implemented, the mantainers it forever, so I believe they’re wise in taking their time with it.

@shin- I guess you’re the one to ask: Does it even make sense to consider this? Considering the current position of the team, how hard it is to design it, and the many tools out there, wouldn’t it be easier to simply tell people this won’t happen? Most people if not everyone who commented here found a way to deal with it.

Closing this seems pretty good, plus you can focus on the stuff that matters, and no one wonders if they should make a pull request and how to meet all those needs at the same time. They can just go build something else, and maybe even publish it to help others.

Just for the record: My solution was not perfect, but way better than the 2 days it took me to install the dependencies and fix a shitload of stuff and make the project run (and it already used Docker). Just 3 commands, with Docker and Docker Compose as the only dependencies. Yes there was more code than I wanted. Yes there’s a shitty order to run commands. And yes there’s a tiny bit more to keep in mind. A Wiki page covered those. Preety good, if I do say so myself, which means Docker Compose did it’s job, I just also needed somethig else.

Point is: If it’s out of the scope, it’s a waste of time to keep discussing it. Many people just stop here to ask about this and give their opinion, before trying to decide what to do. They will still find or build something else if this is closed, just faster. And you get one less thread to babysit.

Edit: Just thought of a good way to put it: If there’s no clear criteria to accept PR and there is no one in the core team either involved with it or planning to take a look at it in the foreseeable future, it’s not an “issue”, just an idea.

I see no reason to clutter the issue list, I believe many ideas here are good, but I think they belong somewhere else, for good or for worse. It’s almost 3 years, and on the off chance that one day the situation changes, you can always reopen this, or open a new issue about it.

@shin- I appreciate you taking the time to have this dialog with us. I agree there are several ways to do it, as you suggest multiple yml files, you can also pass a list of services but these are not convenient and lead to user error creating operational risk. Especially in the case of a commercial product we want to distribute one compose file and by default get the desired common behavior and give instructions for advanced options, one file make all of this much neater for both dev, CI/CD and distribution. The addition of ENV variables with defaults was a huge help to us and has eliminated the need to have many yml files, this will be one more step towards managing a single file describing the deployment.

Is there any form that core team would be willing to accept a contribution on this LMK.

@shin- Before we lock this can I as that we get clarity if the community is will to development would the core team merge it if it meets the standards of the team? In the threads there is an implementation and community member willing to complete it, they were just looking for some confirmation that it would not be a waste of their investment. If we can get some feedback I will track down those threads to move to see if we can move this to closure.

We can handle this fairly easily in a “tools” service with an initial entrypoint or command that simply returns. Then we run this service with an alternative command to do what we want (e.g. tests or seed the data). E.g.

docker-compuse up -d
docker-compose run tools tests

Yes, the tools service will show as “stopped” in docker-compose ps. If this is unacceptable, then I the multi-file option from https://github.com/docker/compose/pull/2051 works. e.g.

docker-compose -f docker-compose.yml -f docker-compose-tools.yml up -d
docker-compose -f docker-compose-tools.yml logs -f tools

Although it looks like a lot of orchestration helpers are being written for this need and I’m guilty of working on one as well 😃 .

Personally I believe there’s enough workarounds (such as the two above; getting creative with your entrypoints/containers and/or using the multifile approach with a shell alias/helper), and compose can be kept simple.

I think that option 1 could work, but we need a better way to reduce the verboseness of:

docker-compose -f docker-compose.yml -f docker-compose.tests.yml up

Perhaps would could add a simplified / augment flag? docker-compose --augment tests up and automatically resolve if it’s a variation of docker-compose.yml - otherwise the user would need to be explicit.

I really like the idea of a new top level keyword though, perhaps suites and commands ?

The following configuration would allow for:

docker-compose run -s application docker-compose run -c cache-clear

suites:
    application:
        services:
            - database
            - memcached
            - application
    tests:
        extends: application
        services:
            - application

commands:
    cache-clear:
        service: application
        workdir: /var/www
        command/entrypoint: php app/console ca:cl

@rubycut Yeah, that’s what happens when so called “higher principles” take over from usability and user-friendliness. I completely understand the devs don’t want to clog the product with illogical features. However, this particular highly requested feature actually seem to make sense. It’s sad when the developers think they know better than their users. (Again, they usually do. But not always. Not this time.)

@rysiekpl This is a problem we’re looking at solving in a different way, see https://github.com/docker/cli/pull/452 (once syntax is deliberated, it’ll make its way into Compose and v2 as well)

@creynders I think that’s a bit of a mischaracterization - nobody is saying that optional services shouldn’t exist (and in fact, as I pointed out, there are ways to declare them by using separate files). This is more akin to debating whether not explicitly closing an <img> tag is valid and whether our specific browser should support it or not.

Also, to your point:

But this distinction is nowhere clarified or expounded upon in the documentation. Nor is it anywhere pointed out that docker is not meant to be used for spinning up temporal processes.

This is actually the first paragraph of the https://docs.docker.com/compose/overview/ page (emphasis mine):

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a Compose file to configure your application’s services. Then, using a single command, you create and start all the services from your configuration.

@ekkis I’m really glad that Compose is in fact helpful for your projects! I hope I don’t come off as insincere in all this, it’s absolutely okay if that’s the way you want to use the software. What I am saying, however, is that taking it further in that direction would probably make it worse in the long term. We all know what happens to applications that try to do too many things (although I guess Emacs pulled it off 😛 )

I don’t think I was being “authoritarian”, but maybe I didn’t express myself properly. My intention was to encourage people to remain civil and positive in their interactions. The thread is still unlocked and will continue to be as long as people refrain from being mean or impolite.

I’ve stumbled on this looking for something else, just wanted to chip in a use case… We found that our developers were the most capable of defining what data should be backed-up from their services. As the docker-compose.yml file is in their hands and named volumes are used, we decided to use it to define the back-up strategies… Here’s a sumarized example of our old docker-compose.yml we were using.

version: '2'
services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    restart: always
    ports:
      - "5555:5555"
    volumes:
      - gitlab_config:/etc/gitlab
      - gitlab_logs:/var/log/gitlab
      - gitlab_data:/var/opt/gitlab
      - certificates:/etc/gitlab/ssl
      - registry_data:/var/opt/gitlab/gitlab-rails/shared/registry
  gitlab-runner:
    image: gitlab/gitlab-runner:latest
    restart: always
    volumes:
      - certificates:/etc/gitlab-runner/certs
      - /var/run/docker.sock:/var/run/docker.sock
volumes:
    gitlab_config:
      driver: local
    gitlab_logs:
      driver: local
    gitlab_data:
      driver: local
    certificates:
      driver: local
    registry_data:
      driver: local

To be able to back-up the volumes data, we decided to go with a busybox container to tar the data needed, but it has to be flexible and do only the volumes the developers want backed-up. Finally, we should be able to back-up every volume separately. To achieve this, we added 3 services to all our docker-compose.yml

  boot:
    image: busybox
    depends_on:
      - gitlab
      - gitlab-runner

  backup:
    image: busybox
    volumes:
      - gitlab_config:/data/gitlab_config
      - gitlab_data:/data/gitlab_data
      - registry_data:/data/gitlab_data
      - /backups/latest:/backup
    command: find . -type d -maxdepth 1 -mindepth 1 -exec tar cvf /backup/{}.tar {}  \;
    working_dir: /backup

  restore:
    image: busybox
    volumes:
      - gitlab_config:/data/gitlab_config
      - gitlab_data:/data/gitlab_data
      - registry_data:/data/gitlab_data
      - /backups/latest:/backup
    command: find . -type f -iname "*.tar" -maxdepth 1 -mindepth 1 -exec tar xvf {} \;
    working_dir: /backup

The backup and restore services just require to be configured using the volumes, the commands will do the rest. We are planning on adding a little bit of configuration to be able to choose which volumes to back-up or restore but for now, we do them all… Since we don’t want those 2 services to start on every docker-compose up command, we needed to specify all the services we wanted, which can be tedious… So we added a dummy service called boot, all it does is depend on all the services that need to run and we just call this one when we docker-compose up.

This here is full of small hacks, but allows us to easily run docker-compose up backup to have our back up stored on the host’s /backups/latest and from there we run our versionning/pruning logic.

I hope it helps someone who’s trying to achieve something similar… But in the end, being able to just tell docker-compose to not boot those 2 services, we wouldn’t need for a 3rd one and complicate our docker-compose up commands.

In my setup, I have services in the docker-compose that are required to run my infrastructure (nginx, database, webserver, message queues…). I also have define additional services that I only need for debugging reasons (for example a database web gui).

I would like the “debugging” services to NOT start automatically, but if I want to add them I can do so with a simple docker-compose up -d database-gui and it just gets added.

Also: Sometimes I change my mind and want one of these services to always start… => With option 2) I can just change that one flag

=> Vote for option two, because it is simple and it seems to satisfy everyone’s requirements here. Every other solution feels like over-engineering to me… Adding complexity to the configuration based on rare or only imagined use cases, not on practical need.

Given that docker now supports the concept of service natively for long running commands I think the best approach would be to add a new section called commands which allows for the same functionality than services except for restart policies, which should be disallowed.

Regarding the “use multiple config files” option (which is indeed the best currently available option), the usability problem with that is that it looks like this in practice:

$ docker-compose up one-shot-command
ERROR: No such service: one-shot-command
$ docker-compose up -f docker-compose.yml -f docker-compose-utils.yml one-shot-command
# Actually goes & does the requested thing

The discrepancy then infects all of your project documentation and automation: you have to ensure two compose files are available to anyone that wants to run utility commands, you have to ensure the full invocation is given for commands in the utility file, and if any command or service is ever demoted from “always running” to “only run when explicitly requested”, you need to go find any invocation that doesn’t supply the config file names and add them.

That is all doable, it’s just annoying compared to being able to say in the compose file “Only run this when I explicitly ask you to, never run it by default”.

Re-reading the thread and the discussion in #2496 about declaring scaling constraints in the compose file, I’m coming around to preferring Option 4 (i.e. a new top-level section) over Option 2.

Specifically, a section name like utilities would be a good fit: components that aren’t normally needed, but when you do need them, you want to have preconfigured which other components they’re linked to rather than having to get people to construct a suitable docker run incantation themselves.

That approach would then cover not only one-off admin commands, but also auxiliary debugging and introspection services like pgweb and Celery Flower (which are very handy to have running in development, but you typically wouldn’t want in production due to the additional security threat surface area they create).

Defining the semantics of these new “not services” also gets a lot easier: they’re exactly the same as services (including all future changes to that format), with the sole exception that unqualified commands like docker-compose build, docker-compose pull, and docker-compose up ignore them entirely - if you want any component listed in “utilities” rather than “services” to be processed, you need to name it specifically (although there could perhaps be an --all option, similar to the current one for docker-compose rm). However, the CLI for interacting with these utilities by name would be identical to that for interacting with normal services (unlike the status quo, where you need to specify the extra utility config file), and docker-compose would take care of preventing name conflicts between service and utility definitions.

While the scaling proposal in #2496 could potentially be (ab)used to get this behaviour, it still feels like a workaround rather than a properly designed feature.

This feature takes like one day to implement because it’s really simple. However, after 2 years of discussions, still nothing. Sad.

You should just do option 2 (but maybe nostart instead or something) because people have been waiting years now, then wrapping docker-compose with their own configuration managers because the tool is too limited, which defeats the point as it’s doing what docker compose should be doing. In two years someone could create their own more powerful docker-compose as well. That’s a lot of time this issue has been open for.

In the long run there should be some consideration of true options grouping. Some are relevant only to build, some only to run and some are mutual (you really want a layer of nesting at least for run or just separate out and have a build section, problem solved). Currently though a lot of things are problematic in the configuration, I’m all for just remake it all in version 5 but getting it right that time (primarily by representing the real on a technical level domain rather than trying to make it map a usage based domain or overly simple, instead create a config layer for those goals on top of the right one, you can’t do it in reverse). I think because of that state of affairs the quick fix is justified.

Another use case for which Option 2 is great:

Let’s say you have a team of people working on the same project. Some of them have access to AWS managed services, but some of them need to run the corresponding resources as local containers. So depending on my workflow, I’ll either always or never need Redis and MySQL containers in my compose constellation.

I’m going to flat-out state that managing multiple compose files is stupid for this use case. The amount of overlap is comical. Maintaining multiple files is a footgun - they will get out of sync and cause confusion. I’m frankly a bit shocked that anyone took that option seriously for any use case.

Being able to quickly enable and disable services, without commenting out large blocks of configuration (awkward in YAML, very messy if accidentally committed), is great on its own. But if compose someday gets the ability to interpolate variables into the config, this feature automatically becomes exponentially more powerful. Until then, it’s merely damned useful.

Option 2 is the one which most users want. Is the decision regarding this still to be made or development on this has started and being tracked somewhere else?

I vote for: “JUST DO SOMETHING FOR US, THANKS.” We vote, we now need at least one answer. 4 GOD!

I created my own solution, get free to use. It’s still in Beta 😃

https://github.com/jonathanborges/picles-docker

+1

I agree that option 2 is probably the least invasive. The only feature I see that option 4 might provide is an ability to run commands on existing containers (using docker exec mechanism). For example, if all I want is to run a database migration or unit tests, there might be no reason to create a whole separate container for it if I already have a container running my app. That said, containers are lightweight and I would be perfectly happy with option 2.

@ryneeverett The semantics are one part of it. It’s a problem of self-documenting and findability. Currently, we tell developers docker-compose run --rm foo bar. We invite them to create a shell function or alias, but we don’t maintain a standard alias/rcfile for projects. (We don’t want to standardize things outside of containers; we want to use Docker to standardize.) Adding a new file for some commands creates a hierarchy of importance: The docker-compose.yml file becomes the “default” file for important things, and the “other” file becomes less important.

The other thing is just maintaining relationships between services becomes more onerous. Just because we want something not to run by default doesn’t mean it doesn’t use a link or volume from a service that is long-running. extends doesn’t actually provide all the facilities we would need to link services to “commands” (one-run services). Even if it did, if we have to use multiple yaml files, we will be forced to use extends where we otherwise wouldn’t need to.

@tberne I agree with that. docker-compose is simple and simple is a good thing. we should vote to keep it

This whole discussion, as I understand it, is basically a large number of people providing a large number of very good reasons to implement Option 2, and the developers refusing because somebody hypothetically could potentially misinterpret the existence of this feature and request some other unrelated features.

I find this to be - maybe a bit harsh in tone, but valid in essence - a very good summary by @rysiekpl. So, @shin-, did we come any closer to a decision regarding this issue? Whether to just close it or to implement/accept a PR for option 2?

I totally see and understand your concerns, but also see the advantages outweigh the risk of someone potentially abusing this, since it is a rather small change but offers a lot of people much more possibilities for much more convenient workflows.

@shin- I see the point. What I am pursuing is “optional services” in your footnotes and the use cases for that, not the idea of tasks, can we somehow agree on how we can progress that point as that was the part of the thread I am trying to champion and my tickets clearly on that path where closed in favor of this thread. If we can separate that from the points you have clearly defined as ‘tasks’ we can make some headway. When people are doing a +1 for option 2 in my read it is to the optional service. Happy to take this offline to discuss and we can then come back to the forum, I am at jim@stratdevops.com

+1 for option 2. Would it respect explicit dependencies though… links etc ?

If I am specifying which service to bring up with docker-compose up [service] it is rare that I want to start any services other than the service in question and it’s dependencies.

Is is this issue going to be implemented at some point or is it just something that compose devs does not want to do ? It would be nice to know so the community can start looking for another solution.

By the way I vote for option 2.

Thank you

@acran in your example the only command that has different behaviour from the current Compose is the first one, and that behaviour can be accomplished using up baz. You can also accomplish the same thing by using multiple Compose files.

@dnephin yes, exactly, I know, and in this example it would be the easiest way. But imagine this with 10 services and only one one-off service (like “init” or “backup”/“dump”): in that case you’d have to list all the services on the command line except that one single one-off service like docker-compose up service1 service2 ... service10 (see below). Especially you’d have to remember all the services and manually keep track of any changes in the docker-compose.yml.

I also know about the possibility of using multiple YAML files, see my first comment above, and this as well as explicitly giving all services on the command line does indeed cover most use cases stated here; but it feels more like a workaround to use.

Here another example docker-compose.yml to visualize the problem:

version: "2"
services:
  init:
    image: busybox
    auto_up: false
    command: echo "I do one-time initializing"
  service1:
    image: busybox
    command: echo "I'm service #1"
  service2:
    image: busybox
    command: echo "I'm service #2"
  service3:
    image: busybox
    command: echo "I'm service #3"
  service4:
    image: busybox
    command: echo "I'm service #4"
  service5:
    image: busybox
    command: echo "I'm service #5"
  service6:
    image: busybox
    command: echo "I'm service #6"
  service7:
    image: busybox
    command: echo "I'm service #7"
  service8:
    image: busybox
    command: echo "I'm service #8"
  service9:
    image: busybox
    command: echo "I'm service #9"
  service10:
    image: busybox
    command: echo "I'm service #10"

A simple docker-compose up with the auto_up attribute starts here all services except service init. To achieve the same without it would require typing a much longer command with 10 services or splitting the YAML file and having to specify multiple files.

So the feature requested here is more about convenience and typing less on the cli and not about a completely new feature.

Trying to get more understanding here: Is option 2) just a specialized case of the more general “we want a initial_scale value in docker-compose.yml”? (see #1661, #2496 et. al.)

Except that it would be nice to be able to configure all of the options a short-term container would need (ie the volumes/mount points, networks, links, name, etc) in the docker-compose file so that you don’t have to keep adding them every time you need to run that container…

On September 13, 2016 3:58:41 AM CDT, Ben Firshman notifications@github.com wrote:

Another idea: now we have top-level networks and volumes, we could add an images option too. This would be a hint to Compose that these images have to be pulled or built for the application to work, but they wouldn’t be run when you do docker-compose up.

This would work for two use-cases, off the top of my head:

1. The common problem here, where you want some kind of management command to run. docker-compose run could fall back to running images if the service doesn’t exist.
2. Apps like this which run Docker images but they aren’t running all the time.

/cc @dnephin @aanand @shin-

You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/docker/compose/issues/1896#issuecomment-246618909

Sent from my Android device with K-9 Mail. Please excuse my brevity.

@acran I would like to see you submit your pull request to see if we can get some traction moving this forward, maybe if you rebase to eliminate the conflict on #3047 it will get picked up, not sure how to get attention to this.

I would like to mention this is important for my use case. I use docker-compose to create a testing environment and run a test suite. The problem with starting all containers at once is that I cannot easily allow the service containers (like database or celery workers) to initialize before running the test suite.

It would be preferable to be able to accomplish something like:

$ docker-compose --file="docker-compose-testing.yml" up -d --exclude=tests
$ sleep 5
$ docker-compose --file="docker-compose-testing.yml" run --rm tests

You can imagine something more elaborate than sleep 5 could be written to verify initialization has occurred but it isn’t necessary for the example snippet.

I did not read an option about a cli flag. So I would propose an --exclude flag is added to the list of options. The --exclude flag would tell the up command not to start the specified container. The --exclude flag would be parsed such that multiple containers can be excluded from start up as necessary.

+1 for option 2 Seems like a simple change…

I came upon this issue just a couple days ago because I was looking for a feature like this.

Since it’s unfortunately not implemented, and from what I could gather, the officially supported alternative is to use multiple yaml files.

Well, I find the idea of using multiple yaml files for services not run by default a poor alternative in comparison to using one yaml file for all services containing one additional setting for each service not run by default.

So, here’s my +1.

Just FYI, “the code” is already there: #3047

@rubycut Talk is cheap, show me the code. If you can,send your PR, if you can’t, just report it and wait .

I solved this in my project by providing a simple bash script that calls the appropriate docker-compose command based on provided arguments.

Now if I do ./docker-compose.sh -e testing up, the result is the same as if I did docker-compose -f docker-compose.yml -f docker-compose.testing.yml up. If I just do ./docker-compose.sh up, I get, as you’d expect, regular docker-compose up.

Therefore, my -e ${environment} option (a short for --env) is basically an alias to -f docker-compose.yml -f docker.compose.${environment}.yml. This results in docker-compose.yml being the base/default compose file and files like docker-compose.testing.yml being its extensions.

This somewhat solves @acran’s problem of the user not knowing which files to include (and in which order) to the docker-compose command by enforcing sane defaults. It is not a very robust solution as some may use more complex compose file combinations, but I think docker-compose should be able to do something similar. Most docker-compose files are likely to start with docker-compose. and end with .yml anyway, so why do I always have to type out such a long command (docker-compose -f docker-compose.yml -f docker-compose.testing.yml up) if I just need a single additional service for my testing environment?

+1 on option 2! And I think the drop of docker compose in favor to swarm stack will be a big drawback to the community. Docker compose is so simple and easy to use!!

To me, it looks like stacks on docker-swarm will eventually be a superset of the features of docker-compose. I think the descriptor files are already very similar. Hopefully, this transition will not be too difficult. The biggest annoyance that I can see is managing the swarm separately from the stack(s). More overhead work for me since I have no interest in running composed applications across more than one node.

Or maybe I’m misunderstanding everything…?

So, if docker-compose is going to be deprecated in the long run, what exactly is the problem with merging the pull request that is already out there? I mean, if docker-compose is going the way of the dodo, how is “somebody will request weird features” a real problem?

OT:

But if compose someday gets the ability to interpolate variables into the config, this feature automatically becomes exponentially more powerful.

Hasn’t this feature been available for a long time? See https://docs.docker.com/compose/compose-file/#variable-substitution. I’m using it to allow customization of our Compose dev environment by using the .env file - e.g. source code location, port mapping etc. Since compose file version 2.1 even the shell-style variable default (${VARIABLE:-default} / ${VARIABLE-default) is supported.

I really like the idea of defining multiple files. However including all files with multi -fincluding the original seems to be a total overkill.

There is actually a more practical way, that works for me (see PS below): put all the service you don’t want to run on up in docker-compose.override.yml . And use docker compose -f docker-compose.yml up for normal startup (note that this a one-time command). On all consecutive calls the override will be sourced…

Overall this is all very unintuitive and a bit unsatisfying. Adding an additional:

• an option for disabling services makes sense especially for more complex overrides

• The -foption is minimal but not very intuitive, another -F <name> option that just adds additional overrides would be very welcoming (i.e. docker-compose.<name>.yml. An alternative would be using <name> as a directory in which all .yml files are sourced: together with symlinks this would be similarly powerfull as /etc/apache/sites.available -> /etc/apache/sites.enabled

On the other hand: anyone could provide a simpe wrapper script (function) for their shell to emulate this behaviour…

PS: my use case is that I define a debugger as a service. Obviously this service is not needed all the time but needs the links (also for security reasons) to work correctly.

Here to +1 for option 2, came across this issue hoping this functionality already existed.

Another vote for 2 and 4.

Just to point out a use-case, some services have circular dependencies, the need for generating config files, running test suites among many other possibilities. We can decouple it from Docker or use other tools but it adds unnecessary complexity to setting up the project.

Having a simple command for setup and/or for tests would make things a lot cleaner.

@msabramo compose is basically hamstrung by what is decided at the docker engine level.

The request is tracked over at the docker engine repo https://github.com/docker/docker/issues/23880 That’s where the decision/work will come from.

I suggest that this issue be closed.

I want to build a base image, “extend” this in the same docker-compose.yml and start only the second image, that is not doable in one step at the moment. With option 2 it will be, but maybe the better and cleaner way is to have a “build stage” and a “run stage”, where creation of base images is done. At the moment I have to run 2 commands and not just one.

@vipseixas it’s hard to gauge the sanity of this solution from a file listing – although yes, I do agree having the ability to not auto-start a service is OK. I’m only suggesting that you use a default entrypoint/command that does nothing for those services you prefer not to autostart. E.g. the container simply exits by default… and to execute those services with an alternative command via subsequent docker-compose run.

In most cases you’ll want to ensure health-checks pass on dependent services/containers before executing tests – for instance jenkins is UP and running (accepting connections on :8080 or whatever) before these tests execute. In the past I used dockerize for this.

If I were to propose a silver bullet, it would be to allow setting a compose profile via environmental variables (e.g. DOCKER_COMPOSE_PROFILE=“tests” ), and to allow services to be dependent on a profile – where all services default to the “default” profile or whatever. I would also ensure services that are dependent on another service are also dependent on that services’ HEALTHCHECK. So something like

app:
  image: myapp:v1

tests:
  image: myapp:tests
  dependencies:
    - profile: "tests"
    - service: "app"
      healthcheck: true

# run the things
docker-compose up -d

# test the things
DOCKER_COMPOSE_PROFILE="tests" docker-compose up -d
DOCKER_COMPOSE_PROFILE="tests" docker-compose logs -f

Lots of great options here. The need for the feature is real.

+1, that would be a very useful feature to have.

And another +1 for option 2, with something like auto-start: false or enabled: false.

My use-case is extending services. It is not possible to extend services that have depends_on set, so what I would like to do is to define a service template, and then extend several services from it. Obviously, the template should be built, but not run autmoagically.

Yes I run but that can be achived and controled inside of containers, in my opinion. But I do not want to detract your view or demands, let’s not make this discussion flame. Our requirements on docker compose are not necessarily in conflict.

@bfirsh @ncoghlan I fully agree with @dnephin:

I don’t think it actually fixes the problems identified in this issue. People want containers, not just the image

What I - and I think most of the people here - need for my desired workflow are simple services; they’re defined like any normal service, they’re started/stoped like normal services etc. The only difference is that they are not automatically started by a simple docker-compose up by default but only when specified explicitly (or pulled in as a dependency).

Therefore I don’t see any necessity for a new top-level section or a completely new concept for these one-off containers but only an optional per service config parameter. I implemented this in my pull request #3047 as a proof of concept. It’s a quite small change but would fully satisfy my use-cases. So if there is anything more I can do, to get this merged, please let me know (=

For anyone who wants to test this, here the commands to build and run docker-compose in a container:

# download and build
git clone git@github.com:acran/compose.git -b auto_up
cd compose
docker build -t docker-compose .

# create an example docker-compose.yml
cat > docker-compose.yml <<EOF
version: "2"
services:
  foo:
    image: busybox
    auto_up: false
    command: echo foo
  bar:
    image: busybox
    auto_up: false
    command: echo bar
    depends_on:
      - foo
  baz:
    image: busybox
    command: echo baz
EOF

# start all default services only, i.e. baz
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -ti -v $(pwd):$(pwd) -w $(pwd) docker-compose up

# start service foo only
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -ti -v $(pwd):$(pwd) -w $(pwd) docker-compose up foo

# start service bar, foo is pulled in as a dependeny
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -ti -v $(pwd):$(pwd) -w $(pwd) docker-compose up bar

# clean up all services, i.e. foo, bar and baz
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -ti -v $(pwd):$(pwd) -w $(pwd) docker-compose down

Another idea: now we have top-level networks and volumes, we could add an images option too. This would be a hint to Compose that these images have to be pulled or built for the application to work, but they wouldn’t be run when you do docker-compose up.

This would work for two use-cases, off the top of my head:

1. The common problem here, where you want some kind of management command to run. docker-compose run could fall back to running images if the service doesn’t exist.
2. Apps like this which run Docker images but they aren’t running all the time.

/cc @dnephin @aanand @shin-

Weighing in with another use case where option 2 would be a suitable solution: iterating on a new subservice within an existing application. Most developers on the project aren’t going to need the new service yet, and you definitely don’t want problems with it hindering their work, but you may also want to avoid maintaining an out-of-tree branch for an extended period of time.

A separate “docker-compose-experimental.yml” file could work, but it means all the documentation for working on that component needs to be written one way while the component is still experimental, and then revised once it becomes part of the standard set of services.

I also initially thought option 2 would be best but now I’m starting to think it would be too restrictive and soon enough after adding it, a need for run-once tasks would pop up. In fact I’m working on a project now where I have a need for the following use cases:

• within the stack I want to be able to define containers that run once on initial deployment and any future runs are only triggered externally (e.g. schedule/event, via API call, etc.) so that they can initialize/update state of main service container within the same stack (e.g. DB data/schema, populating data volumes, etc.)
• within the stack I want to be able to define containers that are only triggered externally so that they can be used to encapsulate utility/admin tasks (e.g. state/data inspection, etc.) and processes that are triggered on schedule (e.g. backup) or by external events (e.g. trigger a process on changes to an S3 bucket)

In my case the distinction between state: created (ran 0 times) and state: ran (ran >= 1 times) is important as some of the utility/admin tasks may be destructive and are only to be used in certain circumstances like migrating services.

Given that I’m now leaning more towards state: running | ran | created as in rocker-compose or option 4 with top level task or job object + ability to express a dependency so that a service can trigger a task to run before/after itself.

+1 option 2,

in the use case of multiple containers sharing a common set of envs and configs it makes sense to build a base container to extend from but not to actually ever start the container

auto-start:true|false

+1 for option 4. some configuration should not be allowed for scripts (eg: restart: always)

with option 2, it could have this weird case:

service:
  run_tests:
    build: ./tests/
    restart: always
    auto-start: "false"

what would that mean ?

+1 for opt 4, If I must pick a second favourite its opt 2.

Today I was looking for exactly (2) but ended up with my least favorite solution, a second YML file. My use case:

I have couple of container and all of them link to the same mongo container. I’d like to offer myself and the team the ability to load fixtures into the mongo database and I figured the easiest way to do that is to load the mongo container under a different name fixtures that itself link to mongo and then run mongorestore --host mongo --port 27017 --drop /dump. Since we don’t want to load the fixtures at all time, it felt natural to have it with a start: false but ended up having a separate YML file for both containers fixtures and mongo.

Works well but start: false is much cleaner IMO. If I would have 10 or more permutations of this so-called fixtures container then yes start: false would be a bad idea and I’d go with (1).

I think our proposed solution for https://github.com/docker/compose/issues/1987#issuecomment-139632238 will handle this. “admin” services can be defined into a separate config, and added with -f when an admin command needs to be run against a composition.