#17 - The hateful GitOps
Hi there! My name is Diego Parrilla. I’m a developer that became an entrepreneur and my latest company is Threatjammer. Subscribe now to my weekly digest about tech, threat Intel, privacy, and security!
GitOps is an operational framework that applies DevOps best practices to infrastructure automation: version control, collaboration, compliance, and CI/CD tooling. The main idea of GitOps is to have a Git repository that contains declarative descriptions of the infrastructure of the production environment and an automated process to make the production environment match the described state in the repository.
And I love it. I think it’s the best thing since sliced bread. Applying automatic changes to the infrastructure by creating a Pull Request -a process of change of the (Infrastructure as) code in the repository that developers are very familiar with- without having access permissions to the systems in production is appealing.
Well… it seems not all people are as in love with GitOps as me. It’s one of the new technology trends that is creating more friction to implement, and especially some SysAdmins -in their own words- are not "drinking the kool-aid." I can understand that a new way of doing things can create a significant amount of friction, but when I remind the first steps of the DevOps culture, I can’t remember such a drag.
What is going on here?
GitOps got significant traction in the Kubernetes community because it was created by Weaveworks to perform Kubernetes configuration in Git and a cloud-native stack (initially AWS). They took from the original ‘this is what works for us’ to ‘this should work for everybody.’ And they created products around the concept and, of course, a lot of marketing. The fact is that automated Kubernetes provisioning does not require ‘Git,’ so linking the ‘GitOps’ to it can be misleading. Some engineers say it’s only marketing, and they don’t buy it.
Still, I think the GitOps first definition has been surpassed, and now GitOps is often defined as a way of implementing Continuous Deployment of system architectures for cloud-native applications. With this broader approach, some of the complaints from the practitioners are:
The programmatic configuration of CI/CD tools can be challenging: The process must perform changes in several repositories (Ex: testing, staging, production) to transition the state of the architecture and its components. These tools (Jenkins, Github Actions, etc.) sometimes lack the programmatic features needed to perform some actions. As a result, some simple actions like bumping a new version can take a lot of development time.
Visibility: Depending on the architectural complexity, it can be challenging to have an accurate view of what is going on. Cloud providers can help to alleviate this pain with their administration APIs and consoles, but I don't think this is a problem for GitOps. We can find the same problems with a traditional Infrastructure as Code deployment.
Secrets Management: This is one of the main concerns in any GitOps project, and the existing tooling (Github Secrets, Hashicorp Vault, etc.) comes to the rescue. Again, an ad-hoc deployment process can have the same challenges. It is not a specific GitOps issue but one that has to do with the maturity of the tooling.
Auditing: Complaining about the poor auditing of the Git repository. Git cannot answer questions about the architecture described in the code stored. It cannot tell us how many times was modified a Load Balancer; it can only tell us that the file that describes the Load Balancer was modified. It’s not the same.
I think all these complaints have to do with the lack of enough maturity in the framework that it’s still young. Better tooling adapted to the needs of the nature of the GitOps projects, different from the needs of Development projects.
After talking with several System Administrators, I feel the problem has nothing to do with the maturity of GitOps. It has to do with a radical change in how they do their job. A Sysadmin working under the umbrella of GitOps needs to juggle the arduous task of keeping the lights on and dealing with a pseudo-programming language to describe the architecture, a Git repository, a CD/CI tool, and a complex (and automated!) change approval system. Oh! And probably the direct access to the servers, storage, and network devices is severely restricted thanks to the GitOps process. No fun anymore!
The shocking truth about the DevOps culture is it enthrones the Developer as the King of the IT department and leaves no room for the SysAdmins to shine. We want them to embrace a process that mimics parts of a DevOps culture that developers probably love, but it says very few to Sysadmins.
So will GitOps succeed? It’s a framework that takes the automation of the infrastructure to the next level. There are tangible cost savings, faster deployment cycles, and better control of the errors. So any organization will embrace it once they can measure these benefits and compare.
And the Sysadmins? Several years ago, they were in charge of deploying the new version of the software in different environments. Nowadays, that’s performed automatically, even several times a day. You can’t fight against the automation of the processes.
Make an educated guess.
The music snippet
“Don’t drink the Kool-Aid, no matter what the preacher says.”