For several years now, this idea has occupied my thoughts, and at last, a few months ago, I finally got the opportunity to pen it down and understand its potential value in the domain of CI/CD.
Look, I understand that you might be thinking, 'This isn't anything new,' and I urge you to have a little patience as we navigate through this article. There is something to learn for everyone. Trust me!
Here is the carefully crafted diagram. It is a bit technical but worth understanding.
Background
From the early days of my professional career, despite having used SVN, CVS, and Mercurial; Git was my favorite. The reason was simple: 'It's from Linus (Torvalds). It has to be great.' Laughably, I had no other reasons. Git has come a long way, and there is hardly anything left to write about, given its de facto position in the realm of Version Control Systems.
One of the great characteristics of Git is its branches, and as they say, 'Branches are cheap.' When I started using Git at different levels, I quickly realized the challenges associated with the branching model. Like many of you, I initially used Git-Flow, which served its purpose at the time. However, I soon encountered difficulties with merge conflicts, managing multiple long-lived branches, handling hot-fixes correctly, and versioning. Furthermore, with the rise of mini-services, micro-services, and nano-services, the problem is compounded. We can argue that such issues are due to inefficiency in the system, processes, people skills, and other factors. Nonetheless, the fact remains that every team encounters these issues.
Before we begin, I want to thank the articles below for helping me through this thought process:
Concept
We will discuss about three core concept that this article is build upon:
- Branching Strategy
- Monorepo
- Versioning
Let's understand each while I highlight the key feature supporting the overall strategy.
Branching Strategy
Throughout my tenure in IT, I have utilized various branching strategies, such as Git flow, GitHub flow, and GitLab flow. For a quick summary of their differences, please refer to the articles listed below:
It became apparent to me that in order to reduce complexity and promote certain development practices, adopting Trunk Based Development (TBD) was necessary. The key features that drew me towards TBD were:
- Single long-lived branch (main) => This reduces branching complexity
- Use of Feature Flags => Feature Flags allow control over capabilities and improve Mean Time To Recover (MTTR).
- Continuous Integration => Short-lived (feature) branches integrate into the Trunk (main branch).
Furthermore, I was also seeking a built-in rollback capability within the CI/CD process, rather than an afterthought. To my surprise, both TBD and VSTS articles discussed the use of a separate 'Release Branch.' For periodic releases, having a separate Release Branch makes it convenient to automate code rollbacks."
Overall, based on my experience thus far, Trunk Based Development appears to be a favorable option.
Repository Structure
In our setup, we have numerous micro-services and nano-services, making it cumbersome to maintain multiple repositories. This not only increases the maintenance burden but also adds complexity to automated dependency testing and the release process. Moreover, the rollback functionality is limited to individual repositories, requiring an overarching wrapper automation for a comprehensive release deployment or rollback across all services.
Considering these factors, it seems like a good idea to adopt a Monorepo approach. There are numerous articles available on how Google utilizes Monorepo company-wide, but ultimately, it's up to you to determine where to draw the line. If you have concerns about Monorepo, starting at the application level with multiple services can be a good initial step. It's important to feel comfortable about monorepo before pushing the boundaries.
Further, Monorepo enforces code sharing among teams, improving testing capabilities and cross-dependencies between applications. Additionally, individual teams can still retain control over managing and writing their CI/CD processes, which are invoked during the build process.
For a deeper understanding of Monorepo, please refer to the TBD article.
Versioning
There are several versioning schemas available, and for this purpose, I will be selecting semver (Semantic Versioning). The semver schema follows the format MAJOR.MINOR.PATCH. To keep it concise, I recommend prefixing branch names with the corresponding Semver component:
- MAJOR => major branch name = major/*
- MINOR => feature branch name = minor/*
- PATCH => hotfix branch name = patch/*
In other words, it is possible to create an automation that automatically increments the version based on the branch name. Additionally, there is a Python library called 'semver' that can be integrated into the CI/CD automation. By combining the Git tag query and the semver Python library, the version can be incremented without any manual intervention.
Furthermore, considering the monorepo structure, the branch name can be further modified as follows:
- MAJOR => major branch name = major/<svc_name>/*
- MINOR => feature branch name = minor/<svc_name>/*
- PATCH => hotfix branch name = patch/<svc_name>/*
When this branches are merged back to the main trunk, it is easy to tag based on the branch name giving below tagging format (assuming the below order of merging starting from beginning):
- MAJOR merging to Trunk => Tag = <svc_name>/v1.0.0
- MINOR merging to Trunk => Tag = <svc_name>/v1.1.0
- PATCH merging to Trunk => Tag = <svc_name>/v1.1.1
Tags on the trunk now makes it easy to identify the overall release version. For new commits that make it to the release branch (requiring comparison with the previous release version):
- If any service increments a major tag, the entire release version will increment the MAJOR component, while the MINOR and PATCH components will reset to 0.
- If any service increments a minor tag, the entire release version will increment the MINOR component, while the MAJOR component remains the same and the PATCH component resets to 0.
- If any service increments a patch tag, the entire release version will increment the PATCH component, while the MAJOR and MINOR components remain the same.
The Flow
Let's explore how these concepts work in a real scenario. Imagine two services, service X and service Y, within a Monorepo (single repository). Changes are made in short-lived branches and then merged back into the main trunk.
Feature Branch
Let's take an example of feature X1. A feature branch named 'minor/X/1' is created for this specific feature. Once the development of feature X1 is completed, it is merged into the Trunk, and a tag 'X/v1.0.0' is applied to mark the version. Similarly, for the next feature, X2, once it is merged into the Trunk, it will be tagged with 'X/v1.1.0'.
Hotfix Branch
Let's consider a hotfix scenario. In this case, the branch name is 'patch/X/1'. Once the hotfix is completed and merged into the Trunk, a tag 'X/v1.1.1' is applied.
Major Branch
Let's consider a major or platform related change, the branch name follows the pattern 'major/X/M'. And once the major changes are completed and merged into the Trunk, it is tagged with 'X/v2.0.0' to signify a significant version update.
Release Branch
When it comes to the release branch, you have flexibility in choosing the branch name format. For simplicity, I propose using the format 'release/2023S1', where S1 is Sprint-1. The crucial aspect here is that once the release branch is created, it will be tagged based on the rules outlined in the Versioning section.
In the given example, the release branch is created from the Trunk, which includes changes X1 and Y1. The hotfix is then cherry-picked into the release branch, and the tag is incremented to the next patch version.
After the release, the release branch can remain in place if further fixes are required, serving as an archive for a code snapshot. Alternatively, it can be deleted after a reasonable time-frame, depending on your specific requirements.
Deployment
For our example, we will focus on three environments: QA, UAT, and Production. When it comes to deployment, it's crucial to optimize the process by building and deploying only the services that have undergone changes, rather than deploying all services within the monorepo. While this approach may require additional automation and effort, the benefits it brings make it well worth it.
QA Environment
QA environment is paired with the main trunk. Whenever a new commit is merged into main trunk, pipeline will automatically deploy changes to QA environment. This ensures that the latest changes are swiftly available for testing and evaluation.
Additionally, the team has the freedom to schedule specific times for code review and kick off the merge train. This allows for a thorough examination and collaboration before moving forward. Once the code successfully passes all the tests in the QA environment, it's considered ready for the next phase.
UAT Environment
Now let's explore the UAT environment, which is closely aligned with the release branch. Whenever a release branch is created or updated, the code automatically deploys to the UAT environment without the need for manual intervention.
This streamlined process ensures that the latest code changes in the release branch are readily available for thorough testing in the UAT environment. By automating the deployment, we eliminate potential delays and human errors, allowing the team to focus on comprehensive testing and validation.
PROD Environment
When it comes to deploying to the production environment, it's important to exercise caution and implement sufficient measures to safeguard the live system. In our setup, even though we have automated processes, we incorporate manual intervention before promoting the code from the release branch to the production environment.
This approach provides the flexibility needed to ensure a secure and controlled release. The deployment strategy is ultimately determined by the team's preferences and organizational requirements, allowing for customization to meet specific needs and considerations.
Summary
I would like to summarize the overall capabilities that this particular approach provides. Although not all the features of this pipeline are explained here, it is still easy to incorporate the remaining feature set.
- Trunk Based Development
- Monorepo
- Feature Flags
- GitOps
- Versioning for services and releases
- Tagging for services and releases
- Auto-generated release notes
- Continuous Integration, Delivery and Deployment
- Service-level rollback
- Release-level rollback
- Security Compliance
- Automated Testing
- Production Deploy Approval
Furthermore, depending on the capabilities of the DevSecOps platform, there are many other features that can be leveraged, such as Queue Merge Request, Directed Acyclic Graph, Deployments, Deploy Freeze, Release Archives, Protected Tags, Branches, and Environments. These additional features enhance the overall CI/CD workflow and provide more options for customization and control.
Acknowledgements
I would like to acknowledge the valuable contribution of Brian Aabel in assisting with the implementation and testing of the concepts discussed in this blog. His expertise and support were instrumental in bringing this CI/CD workflow to life. Thank you, Brian Aabel, for your valuable collaboration and assistance throughout this process.
Comments
Post a Comment