I’ll share my experience with an attempt to create a self-hosted Azure DevOps agent that would be identical to agents running on the Microsoft-hosted agent pool.
At first, I expected that I can just google and find a VHD or VHDX files for DevOps agents. But I couldn’t find any.
Why we wanted to create VM identical to the ones in DevOps
Why go through the hassle of setting up a custom, self-hosted agent that behaves exactly like the one in public pool? There is a difference in the maximum timeouts for DevOps pipeline jobs, which are:
- 6 hours on Microsoft-hosted agents
- Forever (no limit) on self-hosted agents
We found ourselves in a position where we created a tool to migrate data that was running as a DevOps pipeline and the code was tightly coupled with it. Too late we realized that it will take few days to complete, effectively ruling out the use of the convenient Microsoft-hosted agent pool.
Ubuntu used to execute your pipelines is a special Ubuntu
Agents used by Azure DevOps are not default Linux distro installations. They come with a lots of additional packages installed. As an example, see the software installed on the ubuntu-22.04 image. If you just set up a new VM and install Ubuntu on it, it will lack many of those tools. And scripts in pipelines might assume those tools are installed.
Attempt to run exactly the same OS as the Microsoft-hosted agents
Images of systems running on Azure DevOps pools are open source. Or, precisely speaking, only scripts that create those images are open source. Images themselves are not published due to some legal issues.
It means that we would need to build images ourselves. I attempted that, but the scripts died after more than hour with an unclear error (as in the screenshot).
After spending a day trying to make this work, I gave up. It was not worth the invested time and we decided to install the required software on a clean Ubuntu VM. It’s just not easy to do. Far from straightforward. Avoid if you can.
One side comment is that if you still want to invest time in it, there is a tool named azuredevops-buildagentsthat promises to automate building VM image in a VHD form in a pipeline. I didn’t try it, as I only needed a single image build, not a pipeline.
Lessons learned
For me, the lessons learned here are that:
- There is no easy way to run the same VM image on a self-hosted VM as Microsoft, because the image is not published for legal reasons. There seems to be no clean way to set up an agent VM and guarantee 100% compatibility with MS-hosted agent pool.
- Perhaps the best one can do is to just create a custom VM and try to install all the dependencies that we really use in our pipeline. This, unfortunately, requires testing pipelines on the new agents and fixing the found issues.
- You can try building VM images from source scripts. But it takes time (hours). And on my machine it just fails with errors that seem beyond my control to fix.
- Do not design pipelines that need to run more than 6 hours! This is not the right executable environment for such things. Even when we migrated to self-hosted agents which didn’t have the timeout limit, some of the jobs lost connection with DevOps.
If you read to this point, congratulations. And feel free to briefly share in the comments what’s your use case for re-using Microsoft’s agent images. I’m curious what could have brought you here 🙂