Rollback strategy for Azure YAML release pipelines
Introduction
Azure Pipelines supports continuous integration (CI) and continuous delivery (CD) to continuously test, build, and deploy your code. You accomplish this by defining a pipeline.
Continuous delivery automatically deploys and tests code in multiple stages to help drive quality. Continuous integration systems produce deployable artifacts, which include infrastructure and apps. Automated release pipelines consume these artifacts to release new versions and fixes to the target of your choice.
Rolling back deployment is an important consideration of your deployment strategy. The rollback may be required for various reasons — application functionality not working as expected, performance issues, bug found only in production env etc. If you have built your Release pipeline as classic Release pipeline, you can simply choose an earlier release and do a fresh deployment which resembles to rollback to previous version.
The latest way to build release pipeline is using YAML, which Microsoft also recommends. The YAML pipeline although have advantages over classic pipeline, does not support rollback out of the box. In this article, we will look at some of the strategies to rollback when using YAML pipeline
#1 Backup and Restore
The simplest strategy is to backup your deployed application artifacts like folder, configurations etc as part of the pipeline before actual deployment steps. You include a verification step followed by rollback step in your pipeline if verification fails. The verification could be build verification tests or sanity tests post deployment.
The rollback step then would be simply reverting back to the backed up version of your application. Note that, this is mostly applicable to deployments to virtual machines or physical servers. Containers/Kubernetes deployments would need a different strategy.
# Rollback using tags
Backup and restore strategy may work for simpler deployments. A more sophisticated strategy is to rollback using tags.
In this strategy, as part of your release process, you create a release tag on your master or main branch. Lets consider you create tag v1.0.0 and deploy the application to production. The application works as expected and everyone is happy.
Next release, lets say the new version is v1.1.0 and you create another release tag v1.1.0. This is how it looks in Azure ADO
If you do not wish to create tags as part of your release process, you can rely on and use the tags that are created by Azure DevOps every time pipeline is run. I however find it simpler to refer to user friendly tag names, instead of auto generated tags. The tag name is made up of <branch name> + timestamp + sequence number. A sample screenshot of auto generated tags is shown below.
When you are ready to deploy to production, you deploy the latest build that corresponds to v1.1.0. Post deployment you notice that the application does not work as expected and you need to roll back.
Using the below steps, you can roll back to version v1.0.0
Create build artifacts for earlier version v1.0.0
1/ Go to Azure DevOps -> Pipelines and select your build pipeline.
2/ Click on the pipeline which will open run history.
3/ Select Run Pipeline. In the pop-up, select tag v1.0.0 and click on Run. This run will checkout code corresponding to tag v1.0.0, create build artifacts and publish to Azure Artifacts.
Deploy v1.0.0
The deployment pipeline is configured to use Latest version of build artifacts from Azure Artifacts. Perform the below steps to deploy v1.0.0
1/ Go to Azure DevOps -> Pipelines and select your deployment pipeline.
2/ Click on the pipeline which will open runs history
3/ Select Run Pipeline. In the pop-up, select tag v1.0.0 and click on Run.
The pipeline will download the latest build artifacts (which is v1.0.0) and deploy to the target servers. Although the steps are manual, this resembles a rollback to the previous version.
Please ensure that, if you have database schema to revert, it should be already planned as part of your rollback steps to avoid any last minute surprises.
# Rollback by reverting PR
Rollback using PR revert is the third strategy. This includes multiple steps and is time consuming. I would not recommend this unless you are absolutely sure about PR revert and understand the time requirements to complete these steps.
1/ Go to Azure ADO -> Repos -> Pull Requests. Ensure you have selected correct repo. You will see a list of PRs.
2/ Select the last one to revert. It is always recommended to revert in reverse chronological order which otherwise may result in conflicts.
3/ Provide the name of the topic branch where PR will be reverted or accept the name auto populated by Azure ADO. Click on Revert
4/ A new PR create screen will be shown. Click on Create to create new PR.
5/ Review the changes being reverted and complete the PR.
6/ Once the PR completed, your CI pipeline should trigger, build code and publish new build artifacts to Azure Artifacts.
7/ Verify that the build artifacts are published and manually run release pipeline. Select main/master branch as the source. The pipeline execution will use the latest build artifacts and deploy to target servers.
Please comment and share your feedback to improve this article. If you have implemented any other rollback strategy, I would be happy to know more
