3 Docker Compose features for improving team development workflow

A developer today is bombarded with a plethora of tools that cover every possible problem you might have—but, selecting which tools to use is The New Problem. Even in container-land, we’re swimming in an ocean of tool choices, most of which didn’t exist a few years ago.

I’m here to help. I make a living out of helping companies adopt a faster and more efficient workflow for developing, testing, packaging, and shipping code to servers. Today that means containers, but it’s often not just the tool that’s important; it’s the way you use it and the way you scale it in a team.

For now, let’s focus on Docker Compose. It has become the de facto standard for managing container-based developer environments across any major OS. For years, I’ve consistently heard about teams tossing out a list of tools and scripts this single tool replaces. That’s the reason people adopt Compose. It works everywhere, saves time, and is easy to understand.

But getting it to work across dev, test, and prod for a team can be tricky. Here are three main areas to focus on to ensure your Compose workflow works for everyone.

Environment variables

Eventually, you’ll need a compose file to be flexible and you’ll learn that you can use environment variables inside the Compose file. Note, this is not related to the YAML object “environment,” which you want to send to the container on startup. With the notation of ${VARNAME}, you can have Compose resolve these values dynamically during the processing of that YAML file. The most common examples of when to use this are for setting the container image tag or published port. As an example, if your docker-compose.yml file looks like this:

version: '2'
services:
  ghost:
    image: ghost:${GHOST_VERSION}

…then you can control the image version used from the CLI like so:

GHOST_VERSION=2 docker-compose up

You can also set those variables in other ways: by storing them in a .env file, by setting them at the CLI with export, or even setting a default in the YAML itself with ${GHOST_VERSION:-2}. You can read more about variable substitution and various ways to set them in the Docker docs.

Templating

A relatively new and lesser-known feature is Extension Fields, which lets you define a block of text in Compose files that is reused throughout the file itself. This is mostly used when you need to set the same environment objects for a bunch of microservices, and you want to keep the file DRY (Don’t Repeat Yourself). I recently used it to set all the same logging options for each service in a Compose file like so:

version: '3.4'

x-logging:
  &my-logging
  options:
    max-size: '1m'
    max-file: '5'

services:
  ghost:
    image: ghost
    logging: *my-logging
  nginx:
    image: nginx
    logging: *my-logging

You’ll notice a new section starting with an x-, which is the template, that you can then name with the & and call from anywhere in your Compose file with * and the name. Once you start to use microservices and have hundreds or more lines in your Compose file, this will likely save you considerable time and ensure consistency of options throughout. See more details in the Docker docs.

Control your Compose Command Scope

The docker-compose CLI controls one or more containers, volumes, networks, etc., within its scope. It uses two things to create that scope: the Compose YAML config file (it defaults to docker-compose.yml) and the project name (it defaults to the directory name holding the YAML config file). Normally you would start a project with a single docker-compose.yml file and execute commands like docker-compose up in the directory with that file, but there’s a lot of flexibility here as complexity grows.

As things get more complex, you may have multiple YAML config files for different setups and want to control which one the CLI uses, like docker-compose -f custom-compose.yml up. This command ignores the default YAML file and only uses the one you specify with the -f option.

You can combine many Compose files in a layered override approach. Each one listed in the CLI will override the settings of the previous (processed left to right)—e.g., docker-compose -f docker-compose.yml -f docker-override.yml.

If you manually change the project name, you can use the same Compose file in multiple scopes so they don’t “clash.” Clashing happens when Compose tries to control a container that already has another one running with the same name. You likely have noticed that containers, networks, and other objects that Compose creates have a naming standard. The standard comprises three parts: projectname_servicename_index. We can change the projectname, which again, defaults to the directory name with a -p at the command line. So if we had a docker-compose.yml file like this:

version: '2'

services:
  ghost:
    image: ghost:${GHOST_VERSION}
    ports:
      - ${GHOST_PORT}:2368

Then we had it in a directory named “app1” and we started the ghost app with inline environment variables like this:

app1> GHOST_VERSION=2 GHOST_PORT=8080 docker-compose up

We’d see a container running named this:

app1_ghost_1

Now, if we want to run an older version of ghost side-by-side at the same time, we could do that with this same Compose file, as long as we change two things. First, we need to change the project name to ensure the container name will be different and not conflict with our first one. Second, we need to change the published port so they don’t clash with any other running containers.

app1> GHOST_VERSION=1 GHOST_PORT=9090 docker-compose -p app2 up

If I check running containers with a docker container ls, I see:

app1_ghost_1 running ghost:2 on port 8080
app2_ghost_1 running ghost:1 on port 9090

Now you could pull up two browser windows and browse both 8080 and 9090 with two separate ghost versions (and databases) running side by side.

Most of what I’ve learned on advanced Compose workflows has come from trying things I’ve learned in the Docker docs, as well as the teams I work with to make development, testing, and deployments easier. I share these learnings everywhere I can, and I encourage you to do the same. What other features or team standards have you found useful with Docker Compose? Please share with me and the community on Twitter @BretFisher.