If you want to implement a DevOps principle that takes your deployments from weeks to days, then you will need to walk the path, build trust, and fix your own house before you can ever automate.
Unfortunately, when your team is attempting to modernize your delivery chain using DevOps principles, trust is tough at first.
Nobody trusts anybody.
Developers don’t trust Engineers.
Engineers don’t trust business leads.
Business leads don’t trust Testers.
And Testers don’t trust anybody!
With all of this lack of trust of just people, how in the world will your team trust an automated process? A sad fact is that your team will probably trust a computer program more than a human. Possibly due to the fact that a computer program doesn’t have an agenda. But when you’re trying to modernize your delivery chain, trust is essential.
Without trust, DevOps principles do not work.
Trust Takes Time
So how do you design an appropriate level of trust in your team to start using automated tools?
First, you need to understand that automation is not DevOps. Automation is a way to achieve a goal, not a replacement for thinking and decision making. Any automated technology is only as good as the data it is allowed to select, transform, or use to control a process. Automation is a happy dog — it does what you tell it to do, not what you want it to do. When it works, it can extend the performance of developers, testers, and engineers by removing routine tasks from their daily work. It can remove the guesswork associated with testing scripts, configuration, and deployments — all perfect candidates for automation.
However, the key phrase is “when it works”. In an increasingly complex infrastructure, automated tools need to be given the correct data and configurations to be effective. Anything else is a disaster waiting to happen. Just like how a mountain climber has to know — without a doubt — that their climbing gear will save their life, your team needs to know that the automated tool will produce the correct result. The first time it produces an incorrect result will be painful.
A failure when you’re at the beginning of an automation project will cause your team to loses faith in that tool. However, using the DevOps principle of “If it hurts, do more of it,” that failure will make your team double-check and verify everything and that extra level of scrutiny will be even more painful. Which is why you need to do more of it. The goal is to get that trust back, and the difficult of once bitten, twice shy is not an easy hurdle to overcome.
What is the alternative? If you don’t trust that your automated tools are producing correct information, you will have to continuously slam on the brakes before anyone will let you move forward. By having to go through the rigger of double-checking every output, your team learns two things:
#1 — The whole team gets intimate with what a good result should look like.
#2 — The whole team starts to spot the parts that are going unchecked.
Do you know what is worse than a bad test that registers as a good test? It’s the test that you are not running that will break your software further downstream. This is the benefit of having no trust at the beginning. Your team’s skepticism will identify (hopefully) all the tests you should be running, but never have.
Prioritize Your Efforts
Fortunately, not everything needs to be automated at first. In fact, fast delivery can exist without considerable automation, especially when teams are just starting to think differently about their delivery chain. You should always introduce automation slowly.
Sure, it will be something you can use to speed up routine tasks, and it will become essential in changing your delivery cycles from weeks to days. But you can not automate a single process within your delivery chain until you know EVERYTHING about it. You need a very clear understanding of the inputs and outputs of each process, who provide those inputs and outputs, and what is the quality of that data.
Do you know those things about all of your processes? If the answer is yes, you are way ahead of the average. Or you’re full of it!
Chances are you know some of it, and maybe even don’t know what parts of your process are either a black box or out of your control. It’s ok, most teams do not know everything. Silos and shadow IT groups have historically obscured may parts of each process as a way to protect themselves. These black-box processes will always prevent your team from understanding each input and output and in many organizations, they exist all along the delivery chain.
If this is you, then you have to some work to do.
Walk the Path
Before automation can even become a possibility, you need to walk that path. This is an old strategy that I learned from a US Army General. Once upon a time, it took 3-weeks to get a regular G.I. Recruit to go from arriving at the airport in Frankfurt, Germany, to finally being placed in their unit. Having this process take 3-weeks was causing all sorts of problems with other soldiers rolling back to the United States, or with the effectiveness of US Army units waiting for replacements to arrive. The General ordered a few of his officers to find out why it took so long, but never received a satisfactory reason.
So what did the General do? He drove to Frankfurt, found himself a newly arrived G.I. recruit, and followed him through the check-in process. Not wanting to bias the process by having his rank prejudice the results, he remained incognito as he traveled around the US Army base collecting all the signatures a new arrival needed before they could be placed with their unit. What the General found was shocking.
He discovered bizarre requirements and nonsensical administration throughout the process. One example was the requirement for single soldiers to go check in with family housing, while inversely having married soldiers — traveling with their families — being required to go check in with the bachelor barracks. To have a complete picture, the General also followed another soldier through the check-out process. This produce equally frustrating requirements that included having the soldier go check out with the base library. Only that soldier had never been to the base library in Frankfurt.
Furthermore, with the exception of arriving and leaving Germany, that soldier had not been within 50-miles of Frankfurt during the entire 3-years of being stationed in Germany. The experience gave the General a perspective that had to be earned by observing the actual process as it happened. He learned firsthand all the strange and undocumented parts of what seemed like a straightforward process. He also uncovered what the US Army personnel traveling in and out of Germany had become used to. They had always thought that this was “just how it was done” without questioning why it was being done this way. Some of these problems where cultural acceptance, others where a lack of empowerment to do anything about it. However, the General took what he had learned and transformed a 3-week process into a 3-day process.
How did he do this? By having essential knowledge of a process gained by walking the path.
Have you walked all of your paths within all your processes? Until you do — or your team does — you will keep checking out of a library you’ve never been to.
Trust Versus Control
Walking the path of your delivery chain from conception to deployment will require you to do some documenting along the way. This might seem like an obvious approach, but watch out! You’re going to hit black-box processes or even entire groups that do not know all the details of what they do. This seems crazy to think that there are teams that place results into one end of a black box, get an answer out the other end and just accept it as the way you do it. However, to automate your process, you cannot keep any black boxes locked along your path. This means you’re going to have to challenge the status quo.
Unfortunately, when you challenge the status quo, you will inevitably create some roadblocks. Fear is the number one reason for these roadblocks to appear. Fear that if you crack open their black box you’ll discover that it’s actually empty. There has never been a secret in their black box, only the illusion of one. Meanwhile, other groups may not trust your motives in trying to understand what they provide to the delivery chain. These may think that automation will replace them and try to stop your efforts cold. Fear can be a powerful motivator, especially if they think their jobs are on the line.
When your team hits these obstacles, you may need to go around them if you are to see your path walking efforts through to the end.
Automation Takes Time
Fortunately, you don’t need to change your entire organization overnight. The good news is once you’ve documented all the parts of your delivery chain you control, it will give you incredible insight into the parts you do not control. It could lead you to new innovations, remove the need to smash through your blockers, or even teach you what can and cannot be automated.
Ultimately, the trust in your automation journey comes from knowing what is going on, and what is coming out of your given processes. Owning the data gives you the power to build trust that your automated process is working correctly. It is also the only way to know — for a fact — that your automation results are the verifiable part of any bigger process. This means that you should prioritize your automation efforts on only the parts of your process that you do have input control over. It will take time before your team begins to trust your own automation. For other teams, it may take even longer.
This trust, in turn, will start to give your team momentum that hopefully spreads to others. Give yourself domain knowledge by walking the painful path of your process yourself. Become the example for other teams to follow by doing more of what hurts within your own team first, then teaching others to do the same.Tags: automation, devops, process improvement
Last modified: July 1, 2021