Introducing change: Risk based regression
When I joined a company I observed a regression cycle where the team would run through every single test case that they had. Most of the test scripts had to be run by team members as they had not been automated yet. It took the majority of a week to complete the regression cycle, which created problems with new sprints starting and the testers were not fully present for them.
As a leader, my mind started rushing around on how we fix this problem. Initially, I shared my observations that we appear to be running through everything even when there are little changes. I didn’t dive into making changes. On my leadership journey so far, I’ve learnt it’s important not to dive straight into making changes and instead ‘plant a seed’ in people’s minds. I use this concept a lot rather than telling people what to do — share a thought and see if anything blossoms from the idea. Sometimes it works, other times the seed needs more nurturing by the leader, which was needed in this case.
So what I decided to do is run a workshop on our regression strategy and create a vision statement of where we wanted to be. There was a fair majority that favoured a risk based regression approach, so we wrote that statement down. This is where things started to get tricky. There were a few of the team that felt strongly about not introducing this strategy because we may miss things. We perhaps didn’t understand the interconnections well enough and there were often cases where something was changed and an unknown area of the product was broken. This voice started to get traction and the risk based regression started to lose it’s traction.
It was very frustrating. Changing our approach seemed to be taking a long time, but I was determined that this was the right thing to do. We needed to reduce our regression run time significantly (but in a safe manner). There was too much legacy to automate, so I went back to basics. A manager pointed me in the right direction and said go back to education. I read absolutely every article about risk based regression and shared this with the team. Even with this, we didn’t appear to make any more progress, which again felt frustrating.
I continued my research and came across Jenna Carlton’s Math of Risk approach that involved creating an overall risk score. I decided this was going to be my last shot. I prepared as much as I could and started to draw the stakeholders in. I’ve got to be honest, I was nervous about introducing the Math of Risk. What if I got the same reaction as before? It was mind blowing how the concept appeared to click with the stakeholders. Numbers make sense. Alongside the concept of Math of Risk I discussed the concept of risk taking and trying something new. This was important to get the stakeholders buy in as if it went completely wrong, then we would not start a blame game, instead a learning game. They were completely onboard with it and accepted this risk.
We let the teams take their time with it. It felt slow, however, it was the right thing to do. It took a couple of regression cycles for the team to become comfortable with the concept. One team took the approach of not running the low score areas and eventually they became more confident in not running the medium score areas. Other teams didn’t make any changes at all to start and said they didn’t feel confident about not running anything. However, months down the line this has changed completely.
We did have opinions that we were introducing more bugs after the risk based regression strategy, so we got to work and created a dashboard of production bug root causes. What was interesting was that there wasn’t a single bug related to missing something in regression.
What have I learnt from the process
- Be patient. Change takes time. When you look back you’re going to be amazed in how much you achieved!
- Keep going and don’t give up. Speak to your peers/manager to let off some steam.
- Go back to basics and revisit what you want to change and why.
- Educate your team. Share articles. Podcasts. Talks.
- Get buy-in from stakeholders early.
- Everyone on your team is going to have an opinion. Respect that, however, use facts/numbers wherever you can. It’s hard to argue with these.
