Self-hosted Azure DevOps agents running in Docker
Those who are passionate about Azure DevOps and Linux, like me, should find this topic exciting and fun. This post can be interesting for those who are considering running their own agents for CI and CD pipelines, for workloads running on Ubuntu, or want to know something more about Azure Pipelines agents.
It covers the following topics:
- Quick introduction on the subject
- Challenges of installing tools and my opinion, as a user
- Some information on how to install tools on these machines
- Presenting images I created: including one to use Docker inside Docker containers, for build pipelines that need to create and publish docker images
Following along requires intermediate knowledge about:
- Azure DevOps and pipelines
- Docker and Linux
If you look for information about running private agents for Azure DevOps, you should end up in the MSDN documentation at this page:
The documentation clearly describes what are agents and agent pools, what are the options offered by Azure DevOps out of the box, and some reasons why you might want to use your own machines to run your pipelines.
If you are into Docker, your eyes should rapidly catch the link called “Running in Docker”.
Running private agents in Docker has key advantages:
- possibility to destroy and recreate agents from scratch rapidly
- possibility to create new images without making a whole host dirty, while experimenting and installing dependencies
- possibility to share images with others, and start new containers rapidly
Starting an agent this way is extremely easy, and requires only seven steps:
- create a folder for the image
- copy-paste the Dockerfile provided in the documentation
- copy-paste a start script (
start.shfor Linux agents)
build the docker image
docker build -t dockeragent:latest .
- create an access token
- prepare a run command like in the box below
run the command
docker run -e AZP_URL=<Azure DevOps instance> \ -e AZP_TOKEN=<PAT token> \ -e AZP_AGENT_NAME=mydockeragent \ dockeragent:latest
When the machine starts, it will automatically connect to the given Azure DevOps organization, appearing inside the
Default agent pool.
If you want to use a different pool, you can do it by using an additional variable:
docker run -e AZP_URL=<Azure DevOps instance> \ -e AZP_POOL='Self-hosted Ubuntu 16.04' \ -e AZP_TOKEN=<PAT token> \ -e AZP_AGENT_NAME=mydockeragent \ dockeragent:latest
At this point, our agent running inside a Docker container can start accepting jobs:
Love at first sight
My first impression is love at first sight: I truly feel enthusiastic about this feature. Later in this post I will criticize some decisions, so I want to make clear that I am very happy of my experience, and amazed by the amount of good stuff Microsoft is producing since it embraced open source.
So far I only described what you can find in MSDN, adding a few screenshots; in the next paragraphs I will describe something more, including my critique (in the sense of review) of the
start.sh script offered by Microsoft documentation.
The first challenge to face, is: How to install tools? Since I recalled tasks called “Use Node.js” and “Use Python”, I decided to give them a try; expecting they would fail because I didn’t have these tools on my agent.
To my surprise, the task for Node.js succeeded, while the task for Python failed, with error message:
##[error]Version spec 3.x for architecture x64 did not match any version in Agent.ToolsDirectory. Versions in /azp/agent/_work/_tool:
Taking a look inside the running container…
docker exec -it container_name /bin/bash
Reveals that Node.js was installed inside
/azp/agent/_work/_tool. Googling for information, I found the problem described by this good article by Colin Domoney Local Build Agents for Azure DevOps. To resolve the case with Python, it’s necessary to create an exact folder structure, like the following:
/azp/agent/_work/_tool/Python/3.7.3/ x64/bin/python3 x64.complete # <- empty file
Side note: I wrote a bash script that installs Python 3.7.3 to the right location, including symlinks, you can find it here.
This is just an example, but it shows how moving to private agents most likely requires some effort to achieve desired results.
The second question I asked myself was: How do restarts and caching look like? I therefore stopped and restarted my container.
docker ps # to find the name of the running container docker stop container_name docker start -i container_name
To my surprise, as the agent restarted, everything was gone: Python 3.7.3 I installed to the tools folder, and all files required to running the agent were downloaded again.
I don’t like two things in the start scripts that Microsoft offers to run private agents in Docker, both PowerShell and Bash.
- Every single time the agent starts, it wipes out the agent files, and downloads again a ~88MB package of Agent Pipelines files - even though the package it downloads is the same that was just deleted!
- The folder where it installs work tools (such as Node.js and Python), is by default a child of the agent folder: the one that gets deleted at each restart. Therefore, every time you restart your agent, you also need to download again all the tools when a pipeline runs
Imagine a poor guy like me, who wants to enjoy faster builds and releases, by spinning up a Docker container inside his work laptop, not being able to benefit from
caching, only because he doesn’t have a dedicated, always-on machine for this.
Caching tools and build dependencies is one of the main reasons for using private agents in the first place, isn’t it? As it happens, I don`t always have access to optical fiber.
What’s the point of downloading again a package that is no different than the one already on file system?
I therefore improved, I dare to say, the bash script to do the following:
- By default, it downloads the Azure Pipelines agent files only if not already present on file system
- It offers the option to wipe out the files and download the latest version of the agent, if the user desires so
- By default, it uses a different folder to store work tools, so they are not wiped out, even if the agents file are updated; it’s Docker, so to have a clean machine, it’s sufficient to start a new container
- To support restarts, using a method already included in the
start.sh, to reconfigure the agent across restarts
My script can be found in this repository:
AzureDevOps-agents. To force the update of the agent files, use an env variable
Bonus content: Docker images I prepared
I often feel grateful that Microsoft embraced open source, I feel my job is way more fun now. You can find the scripts to build images for the official Azure DevOps Hosted agents. Studying and adopting these scripts, I prepared three Docker images that can be used as agents, with some pre-installed tools and with my
start.sh file. By the way, I am using
Ubuntu 16.04 instead of
18.04, because it’s easier to adopt existing scripts for this OS version.
Docker images for CI and CD pipelines are not like most Docker images: it is necessary to find a compromise between size and having a pool of tools that can be used to cover many scenarios. In fact, the official hosted images look like “factotum” (note: I refused to write English plural form factotums for a latin word), having a ton of tools.
My images can be used as reference, to create agents for ASP.NET Core, Go, Rust, Ruby, etc. They are here: https://github.com/RobertoPrevato/AzureDevOps-agents.
1. Base image
A base image with my
start.sh script and some tools, including Azure CLI, Azure DevOps extension for the CLI, azcopy, curl, wget, ca-certificates, tools to compile
C libraries, etc. Maybe I am wrong, but I think libraries for C compilation are generally important on build agents - please leave a comment if you think I am wrong.
2. Docker inside Docker
A base image with Docker, for build pipelines that need to create and publish Docker images.
To run it on a Linux host having Docker, use these command:
For interactive run:
# interactively: docker run -it -v /var/run/docker.sock:/var/run/docker.sock devopsubuntu16.04-docker:latest /bin/bash # run Azure DevOps agent: AZP_URL=<YOUR-ORGANIZATION-URL> \ AZP_TOKEN=<YOUR-TOKEN> \ AZP_AGENT_NAME='Ubuntu 16.04 Docker' ./start.sh
Straight to the point:
docker run -v /var/run/docker.sock:/var/run/docker.sock \ -e AZP_URL=<YOUR-ORGANIZATION-URL> \ -e AZP_TOKEN=<YOUR-TOKEN> \ -e AZP_AGENT_NAME='Ubuntu 16.04 Docker' devopsubuntu16.04-docker:latest
3. Python.. of course
Of course, an image for Python 3.7.3, PyPy 3.5, Python 3.5 and Python 2.7 - because it’s me. 🐍
Hope you enjoyed this post
I am so excited about these things, I slept little lately and I wrote most of this post at night (to 3 AM); so I hope my work can help others. Cheers!