Scrum for R&D Software Development

Do you mean Research & Design or Research & Development?

I mean Research and Development for a software product we are developing. I guess there is a bit of design involved as well.

I mean Research and Development for a software product we are developing. I guess there is a bit of design involved as well.

There's no reason to assume that "Scrum does not work" in this context. Scrum may be seen as an empirical approach in which validated learning is important

When you were told that Scrum does not work for R&D, can you explain the reasoning behind this assertion, or why you suspect "modification" might be necessary?

Should work very well if not ask team why they are not satisfied, what do they expect ? Try KANBAN maybe.

I would think that Scrum works really well for Research and Development. 

• You state a problem to research. 
• You refine the story into workable experiments. 
• You start the experiments. 
• Evaluate the results as you go and at each sprint boundary.
• Determine the next steps based on what you just learned.

Isn't that pretty much the definition of Research and Development?

Research and Development is harder with scrum because the product backlog is harder to categorize and prioritize when the result of a given research project may be unclear or take unbounded amounts of time to complete. In my experience most engineering teams are pretty used to a sequential approach along the lines of: design, implement, test, measure. Each step typically has a pretty predictable amount of time to commit. Whereas research has a success vs time tradeoff; often times success is possible by committing more time to research. But this isn't very agile. Furthermore if product doesn't have a deep understanding of the research subject, it can make it hard to communicate what's possible and how to prioritize without selling exaggerations. So how do you make a product backlog when the downstream usability is unclear? Furthermore how do you make a product backlog when success is not guaranteed or the time to success is unknown? And if your product manager is not a researcher in your field, how do you decouple the researched technology from an actual product increment?

As an initial process, I've broken my R&D projects up into several steps, each one of which can result in a "don't pursue" decision:

1. Hypothesis testing: gather the initial data to suggest that your hypothesis might even be possible to validate
2. proof of concept: often times by providing an initial mock-up 
3. Scientific process: generate a hypothesis, ask a research question (probably involving company KPIs), propose an experiment, 

Step 3 is essentially a normal scrum/product process. Asking a research question can be worded in terms of a user story. This is the point where we converge from a typical research process to a typical engineering process. Once we have a proof of concept we have a better idea of how the technology works and can do our best to categorize it's utility and potential failings so that product can better understand how to 

The hardest part of this whole thing is resisting the sunk cost fallacy. Once your developers have spent so many human-hours performing steps 1 or 2 it may seem desirable to spend time on step 3 for a lack-luster product output. Don't do it. More than the typical cost-benefit analysis of hours spent for value generated, there's additional costs of technical debt of your product, and user cognitive load to having extra features that don't deliver too much value. Hopefully your engineering leads and design leads will support these last statements! This is why it's so important to make the emotionally difficult decision to not continue a research project early on if it doesn't look promising immediately. (And let's avoid that "don't be emotional" narrative too: acknowledge the emotion upfront and move on when it's prudent)

Great reply by @Stefan Sullivan,

I just signed up to thank you.

Implementing Scrum for R&D can indeed be challenging, but it is definitely possible with some adjustments. I faced a similar situation while working on fitness app development projects with my teams. Initially, it was difficult to apply the Scrum framework to R&D work due to the uncertainty and exploratory nature of our tasks.

One approach that worked for us was to focus more on the iterative nature of Scrum rather than strictly adhering to every element of the framework. We adapted by setting shorter sprints and emphasizing frequent reviews and adjustments. This allowed us to stay flexible and respond quickly to new findings or changes in direction. Additionally, we used the Scrum ceremonies (like daily stand-ups, sprint reviews, and retrospectives) to maintain communication and ensure alignment across the team.

For anyone interested in developing wellness or fitness applications, I highly recommend checking out this insightful article here. It provides valuable guidance and best practices that can be particularly useful for navigating the unique challenges of R&D in this field.

Some people tell me Scrum does not work for R&D.

I think you need to probe further and ask what exactly is the problem they face when using Scrum. I dont think it is not possible to use Scrum for R & D.

From my experience some of the issues that comes up when you have work which is exploratory in nature is the time boxing ,delivery value (by value I mean for the end user) each sprint, vertical splitting of stories is not possible, estimating tasks is also challenging. 

Based on the response from the team you can decide how best to help them. Remember the end goal is to deliver value and try to take advantage of the iterative way of working in Scrum.

Being a Scrum Master of 3 teams, one of which is dedicated to R&D, you may come across difficulties based on different aspects of Scrum implementations. Some critics have said that Scrum is not ideal for research and development, mainly because of its frequently repeating and stepwise structure. Still, the ones that argue in favor of Scrum say that this methodology is versatile enough to be manipulated for a good fit. Focusing on flexibility and creativity as primary tenets of the Scrum framework will be the means of moving forward with R&D Scrum. Start by defining the primary objectives and goals, although they might not correspond to a certain end product at first. R&D experiments should not be exclusively aimed at finding solutions, thus, this method would still help in this case. Make it a point to conduct regular reviews and retrospectives, so the team will have the ability to change their tactics as needed, depending on what they find out and get from the savvy.

Moreover, if needed, let’s extend the sprints longer to be able to adapt to the often unpredictable nature of the research tasks. Remember, you must place emphasis on the importance of teamwork and knowledge sharing among all the members of the project so as to stimulate the innovative ideas. Finally, convince them that the stakeholders are also part of the exploratory and uncertain process of research and development which may require them to be more flexible with the deadlines and deliverables. Employing a modified Scrum methodology R&D team for you can not only be a great addition to the benefits of Scrum but also can benefit from flexibility which is critical for research and development.