How to do Scrum in a DevOps team?
I'm leading a DevOps team and I've been applying the Scrum methodology to do so and in many ways it works really well. There are a few difficulties that I'd like to hear your thoughts about, however, which I've outlined below. Before I get started, I'd like to note that the company I work at focuses on in-house development (a big in-house project with tons of devs focusing on different aspects of it), it's not a development agency with many small teams focusing on different small projects.
1. Many if not most projects are fairly experimental and hard to estimate. Some examples:
- "Set up automated, cyclical E2E test running against the production environment"
- "Set up automated, cyclical E2E test running between our mobile app and API (versions/branches of both codebases configurable)"
- "Set up kubernetes staging environments which allow spawning new environments directly from GitLab's Merge Request UI with proper building, branch switching, change updating etc."
What's in common in these projects is that they require the engineers to do some work without a real precedent (something we've not done before) in a complex environment (complex build processes, interaction of various tools & services in an infrastructure). These projects are not even close to the standard "Build new UI/section/feature in Website X" kind of tasks tasks common in coding teams which are fairly easy for devs to estimate once you've been working with such tasks for a while.
Sure, even those standard development tasks have uncertainty in them, but I'd argue that there's a lot more uncertainty here, because in code development you're at least for the most part working in the same codebases/environments and you're used to all of the rules and restrictions in them, while in the projects I outlined above you're almost always working in a new area.
The "kubernetes staging environments" project is a real project that in the end took months to finish, while originally it seemed like maybe it would only take a few weeks. I tried planning timeboxed R&D first, but even with the overall abstract plan being clear after that R&D, there was still a lot of uncertainty moving forward and the engineer had to kind of learn as he went on. In this case I will note that the engineer didn't have a lot of experience, but I'd argue that this situation is pretty common in this sphere.
2. I can't figure out how it would be possible to have a single sprint goal and engineers all working on the same tickets in this situation. So one of the "rules" of Scrum is that tickets shouldn't be assigned to anyone in particular and the entire team should collaborate on the same set of tickets to achieve the sprint goal. I can't really figure out how this would be possible in this team.
Generally the workflow looks like this: DevOps engineers write IaC (Infrastructure-as-code), this IaC is then code reviewed, tested in a staging environment (e.g. Google Cloud account for staging usage only) and then executed in the production cloud management account to set up all of the resources. There are also other times when we can't use IaC and there are things that need to be done manually.
It seems to me that if the engineers would be working on the same project (e.g. setting up and configuring stagings with kubernetes) they'd be constantly stepping on each others toes, possibly even going in contradictory directions.
Does anyone have success having a single sprint goal in a DevOps team?
3. Besides our own fairly experimental backlog we also need to support other teams with ad-hoc support requests like "fix the broken CI/CD pipeline for project X" or "SSL certificates are broken on stagings", which I had to split out as a Kanban board outside of the sprint. Since these kind of requests couldn't be predicted and had to be dealt with quickly, they couldn't really be planned in a Scrum format.
What I ended up doing was rotating 1 DevOps engineer each week to not have a weekly sprint, but to be in charge of this support board. Does this make sense to you or do you think it could be done better? It's not ideal, of course, because this complicates Scrum planning.
You could say that this is because the DevOps team is a component team supporting the rest of the IT department, but at least currently it's not really possible to make all development teams be full feature teams handling their own features from development to infrastructure, especially because we're all working on the same IT systems for the most part.
Thanks in advance for all of the commentary and suggestions!
It sounds as though you're making a lot of decisions on behalf of the people doing the work. Perhaps Scrum isn't a methodology at all, but a framework with which team members collaboratively find the most effective way to self-organize. My advice is to encourage them to do so, and to think of each Sprint as being a real project.
I recall being in similar situation and the best opportunity here for you is to deal with the uncertainty. Definitely, you would need to leverage the expertise of DevOps team to deal with the unplanned activities. But few suggestions below to try for.
- Try sizing the story rather than estimating. This will eventually help the team learn from experience and become adaptive in their approach
- I am not sure about the capacity utilization in your team but I am sure there is always a share of the capacity planned for support, fixing bugs etc. kind of activities. Try negotiating on the same with the PO if do not suffice
- I did not understand the point about having a single goal. A sprint can have multiple goals! And in case of scaled agile too there are multiple sprint goals.
I agree with @Ian Mitchell. You say multiple times that you decided and not once do you mention discussing it with the team of people doing the work. In my opinion, it is apparent that you do not understand the Scrum framework at all. If you did you would realize that there is no single person that makes decisions for the Scrum Team and instead the team will self-organize to solve issues/impediments that they face.
If you really do want to advocate the use of the Scrum framework, you should take time to actually study the Scrum Guide to fully understand it. To me it seems that you know a few terms but what you are doing is not using the Scrum framework. Not once did you mention any of the Scrum Events. No mention of a Product Owner, Scrum Master, Product or Sprint Backlog. I am actually struggling to find anything even Scrum-like in your statements.
Everyone that contributes answers in these forums know that it is very difficult to use the Scrum framework in its purest state. And we will all admit that sometimes organizations do not fully support the use of the framework. Also, most of us have worked in places where some parts of the framework were used but not all of it. In order for the Scrum framework to be successful, there has to be organizational changes to support it.
Your questions are all about procedural issues. The Scrum framework does not say anything about process. That is left to the individual teams and organizations to determine. Scrum is not a methodology as you state. It is a framework within which many processes may fit. Perhaps that should be your first area to research for better understanding. In fact your engineers could explain how frameworks are used in software development. It is very much the same ways that the Scrum framework is intended to be used.
I know I've done nothing to answer your questions and there is a simple reason why. I am not involved in the work, have no knowledge of the technology stack, problems faced, or product goals. This is why both @Ian Mitchell (if I may put words into his mouth) and I have mentioned that you should be discussing these things with the team doing the work and let them solve the issues if there are any. They are the most knowledgeable of the situations and thus would know better than anyone else on how to address them.
Agree with Ian, let the team decide and experiment.
I also lead devops teams and successfully manage to use Scrum. Few sugestions from my side.
1. Product thinking.
You deliver infrastructure, monitoring etc. So you need to understand and identify who is your customer. Is it developer? Is it engineering manager? Is it support engineer? This will help you to organize effective sprint reviews and focus on most important aspects. What is actually your product? You deliver full product for your customer, what is it? components? infrastructure? or maybe platform. This will help you to shape goals and to identify stakeholders.
2. Strong requirements engineering
I applied FURPS framework to manage requirements (FURPS - Wikipedia). I do not use stories in devops teams. We use requirements of the RPS types mostly (sometimes F). When you define for each devops activity requirement using FURPS, it will help you to manage your PBIs, build transparency, organize PBIs into epics (or something similar) and to shape and communicate your sprint goals. You may be focused on completely different activities from technical (or even functional) perspective, however still these activities support one goal like "improve security for part of product" "improve monitoring for module X" etc. Based on FURPS sprint goals are outcome oriented, widely understood and also could help team to identify missing FURPS requirements. These goals could be also achieved by team in alternative ways. Thinking from the perspective of non functional requirements (using FURPS) help Scrum Teams to shape one sprint goal.
3. Manage escalations smartly
Measure and try to better understand how much time you spend on escalations each sprint. Is there any predictability / regularity you can identify? Is understanding infrastructure bottlenecks and future potential problems (like: we approach scalling limits in two months) helpful to build predictability about 'how much time goes next sprint for ops escalations'?This should reduce team's commitments during each sprint planning. Probably you will not have stable velocity over few sprints, but still you can use both velocity+time expected for escalations to better plan your sprint.
Thanks to everyone, except Daniel, for your answers, I've found your notes to be useful and they've given me something to think about! I definitely involve my entire team in the decisions of how to improve our workflows and processes (e.g. through retrospectives), but the company I work at doesn't really have any pure implementation of Scrum anywhere, especially not for DevOps, so there aren't many examples to go by. The Scrum guide and books on Scrum are helpful as far as they go - with explaining the theory, but sometimes it's tough finding the correct implementation in circumstances that can be very different from the standard.
Daniel, your reply is a "lose-lose" comment - I would've been better off not reading it, and you would've been better off not writing it. Unnecessarily aggressive and full of baseless, unfounded assumptions about me and my team. Not just that, but there's 0 value otherwise in it as well. Please don't participate in my thread anymore, thanks.