In this post I share my experience with self-hosted agents for Azure DevOps running in Docker, with Ubuntu; and information about images I prepared and published in GitHub and Docker Hub.

Those who are passionate about Azure DevOps and Linux, like me, should find this topic exciting and fun. This post can be interesting for those who are considering running their own agents for CI and CD pipelines, for workloads running on Ubuntu, or want to know something more about Azure Pipelines agents.

It covers the following topics:

1. Quick introduction on the subject
2. Challenges of installing tools and my opinion, as a user
3. Some information on how to install tools on these machines
4. Presenting images I created: including one to use Docker inside Docker containers, for build pipelines that need to create and publish docker images

Following along requires intermediate knowledge about:

• Azure DevOps and pipelines
• Docker and Linux

Introduction

If you look for information about running private agents for Azure DevOps, you should end up in the MSDN documentation at this page:

• Azure Pipelines agents

The documentation clearly describes what are agents and agent pools, what are the options offered by Azure DevOps out of the box, and some reasons why you might want to use your own machines to run your pipelines.

If you are into Docker, your eyes should rapidly catch the link called “Running in Docker”.

Running private agents in Docker has key advantages:

• possibility to destroy and recreate agents from scratch rapidly
• possibility to create new images without making a whole host dirty, while experimenting and installing dependencies
• possibility to share images with others, and start new containers rapidly

Starting an agent this way is extremely easy, and requires only seven steps:

1. create a folder for the image
2. copy-paste the Dockerfile provided in the documentation
3. copy-paste a start script (start.sh for Linux agents)

4. build the docker image

    docker build -t dockeragent:latest .
   

5. create an access token
6. prepare a run command like in the box below

7. run the command

    docker run -e AZP_URL=<Azure DevOps instance> \
      -e AZP_TOKEN=<PAT token> \
      -e AZP_AGENT_NAME=mydockeragent \
      dockeragent:latest
   

When the machine starts, it will automatically connect to the given Azure DevOps organization, appearing inside the Default agent pool.

If you want to use a different pool, you can do it by using an additional variable:

docker run -e AZP_URL=<Azure DevOps instance> \
  -e AZP_POOL='Self-hosted Ubuntu 16.04' \
  -e AZP_TOKEN=<PAT token> \
  -e AZP_AGENT_NAME=mydockeragent \
  dockeragent:latest

At this point, our agent running inside a Docker container can start accepting jobs:

Love at first sight

My first impression is love at first sight: I truly feel enthusiastic about this feature. Later in this post I will criticize some decisions, so I want to make clear that I am very happy of my experience, and amazed by the amount of good stuff Microsoft is producing since it embraced open source.

So far I only described what you can find in MSDN, adding a few screenshots; in the next paragraphs I will describe something more, including my critique (in the sense of review) of the start.sh script offered by Microsoft documentation.

The challenges

The first challenge to face, is: How to install tools? Since I recalled tasks called “Use Node.js” and “Use Python”, I decided to give them a try; expecting they would fail because I didn’t have these tools on my agent.

To my surprise, the task for Node.js succeeded, while the task for Python failed, with error message:

##[error]Version spec 3.x for architecture x64 did not match any version in Agent.ToolsDirectory.
Versions in /azp/agent/_work/_tool:

Taking a look inside the running container…

docker exec -it container_name /bin/bash

Reveals that Node.js was installed inside /azp/agent/_work/_tool. Googling for information, I found the problem described by this good article by Colin Domoney Local Build Agents for Azure DevOps. To resolve the case with Python, it’s necessary to create an exact folder structure, like the following:

/azp/agent/_work/_tool/Python/3.7.3/
                                    x64/bin/python3
                                    x64.complete     # <- empty file

Side note: I wrote a bash script that installs Python 3.7.3 to the right location, including symlinks, you can find it here.

This is just an example, but it shows how moving to private agents most likely requires some effort to achieve desired results.

Restarts

The second question I asked myself was: How do restarts and caching look like? I therefore stopped and restarted my container.

docker ps  # to find the name of the running container

docker stop container_name

docker start -i container_name

To my surprise, as the agent restarted, everything was gone: Python 3.7.3 I installed to the tools folder, and all files required to running the agent were downloaded again.

I don’t like two things in the start scripts that Microsoft offers to run private agents in Docker, both PowerShell and Bash.

1. Every single time the agent starts, it wipes out the agent files, and downloads again a ~88MB package of Agent Pipelines files - even though the package it downloads is the same that was just deleted!
2. The folder where it installs work tools (such as Node.js and Python), is by default a child of the agent folder: the one that gets deleted at each restart. Therefore, every time you restart your agent, you also need to download again all the tools when a pipeline runs

Imagine a poor guy like me, who wants to enjoy faster builds and releases, by spinning up a Docker container inside his work laptop, not being able to benefit from caching, only because he doesn’t have a dedicated, always-on machine for this. Caching tools and build dependencies is one of the main reasons for using private agents in the first place, isn’t it? As it happens, I don`t always have access to optical fiber.

What’s the point of downloading again a package that is no different than the one already on file system?

I therefore improved, I dare to say, the bash script to do the following:

1. By default, it downloads the Azure Pipelines agent files only if not already present on file system
2. It offers the option to wipe out the files and download the latest version of the agent, if the user desires so
3. By default, it uses a different folder to store work tools, so they are not wiped out, even if the agents file are updated; it’s Docker, so to have a clean machine, it’s sufficient to start a new container
4. To support restarts, using a method already included in the start.sh, to reconfigure the agent across restarts

My script can be found in this repository: AzureDevOps-agents. To force the update of the agent files, use an env variable AZP_UPDATE=1.

Bonus content: Docker images I prepared

I often feel grateful that Microsoft embraced open source, I feel my job is way more fun now. You can find the scripts to build images for the official Azure DevOps Hosted agents. Studying and adopting these scripts, I prepared three Docker images that can be used as agents, with some pre-installed tools and with my start.sh file. By the way, I am using Ubuntu 16.04 instead of 18.04, because it’s easier to adopt existing scripts for this OS version.

Docker images for CI and CD pipelines are not like most Docker images: it is necessary to find a compromise between size and having a pool of tools that can be used to cover many scenarios. In fact, the official hosted images look like “factotum” (note: I refused to write English plural form factotums for a latin word), having a ton of tools.

My images can be used as reference, to create agents for ASP.NET Core, Go, Rust, Ruby, etc. They are here: https://github.com/RobertoPrevato/AzureDevOps-agents.

1. Base image

A base image with my start.sh script and some tools, including Azure CLI, Azure DevOps extension for the CLI, azcopy, curl, wget, ca-certificates, tools to compile C libraries, etc. Maybe I am wrong, but I think libraries for C compilation are generally important on build agents - please leave a comment if you think I am wrong.

2. Docker inside Docker

A base image with Docker, for build pipelines that need to create and publish Docker images.

To run it on a Linux host having Docker, use these command:

For interactive run:

# interactively:
docker run -it -v /var/run/docker.sock:/var/run/docker.sock devopsubuntu16.04-docker:latest /bin/bash

# run Azure DevOps agent:
AZP_URL=<YOUR-ORGANIZATION-URL> \
AZP_TOKEN=<YOUR-TOKEN> \
AZP_AGENT_NAME='Ubuntu 16.04 Docker' ./start.sh

Straight to the point:

docker run -v /var/run/docker.sock:/var/run/docker.sock \
-e AZP_URL=<YOUR-ORGANIZATION-URL> \
-e AZP_TOKEN=<YOUR-TOKEN> \
-e AZP_AGENT_NAME='Ubuntu 16.04 Docker' devopsubuntu16.04-docker:latest

3. Python.. of course

Of course, an image for Python 3.7.3, PyPy 3.5, Python 3.5 and Python 2.7 - because it’s me. 🐍

Hope you enjoyed this post

I am so excited about these things, I slept little lately and I wrote most of this post at night (to 3 AM); so I hope my work can help others. Cheers!