Thoughts on Agile BI

In your view, does BI/analytics represent non-complex work, for which empirical process control would offer no advantage?

BI/ Analytics does represent complex work. However,Most of the Use cases with Scrum is with software development for a Product. However with machine learning, data science and analytics, it needs Data Architecture, Data analysis, Data Analysis, Business SMEs across product lines and you can’t really spin up anything in Sprint 1.

Apparently 70% of the BI projects using Scrum fail. Most of the BI projects use some variation of Scrum. Extreme Scoping is supposed to be widely used in BI.

just wanted to know from folks who have first hand experience in this space.

Thanks !

However with machine learning, data science and analytics, it needs Data Architecture, Data analysis, Data Analysis, Business SMEs across product lines and you can’t really spin up anything in Sprint 1.

It sounds as though complex work would be undertaken from the beginning, and perhaps any investment made ought to be validated. In my experience Scrum finds no application only when there is no hypothesis to test and nothing for a team to learn.

I see 2 scenarios where Scrum in a BI environment is not optimal.

Scenario 1: when data analysts mostly work on relatively small reports and dashboards or they support development and business teams with their data knowledge. In this case, they are not really working as a team on their tasks, and their tasks are pretty much irrelevant to one another. Thus, it would not make sense to bundle their work for the sake of a Sprint. Kanban or something similar would better suit their environment.

Scenario 2: when the data engineering team is cross-functional in programming, but dependent on the work of others (who are not in any Sprint) while building a product. As they heavily depend on others, they can't efficiently plan a Sprint and can't do much to complete it when they lack the necessary input (data, content, solution components, etc.) from others.

In short, when we do not have the cross-functional team which is the 'essence' of Scrum.

On the other hand, the problems you mention, Subramaniam, are more rooted in semantic debates. Some agile thumb rules are often oversimplified and overpromoted:

• Many people perceive agile as a jump into the dark where we have no upfront plans AT ALL. Infact, we should have multiple planning horizons, and the ideal level of detail of each depends on our product. The Scrum Guide mentions at least 4 of these (plan the day at Daily Scrum, Sprint Backlog, Product Backlog, product vision) and does not exclude having more.
• Many people translate the 'Sprint 0 is a myth' to 'Start your Sprint 1 with no preparation but finish with working functionality'. In fact, greenfield work definitely needs preparation - though, perhaps not planned in a Sprint. In practice, many Scrum teams still find their Scrum bubble too comfortable not to have the preparation work as Sprint 0. When we say emergent architecture is a good agile practice, we should also consider that the 'last responsible moment' is sometimes before Sprint 1.
• When we say the development team is trusted with architectural decisions, that does not mean we gave them unlimited time, budget and wisdom. They may or may not need external specialists, they still have to follow organisational directives, they have to cooperate with other teams, etc. There is a concept in law philosophy and politics called subsidiarity which describes the same problem.

To be honest, I never used agile BI. On the other hand, considering it has the same iterative, incremental approach, coupled with similar principles, I do not see how would agile BI address the above problems better than Scrum. I guess agile BI's comparative advantage is in addressing data issues. I suppose these practices can be incorporated into a Scrum implementation, too, since from this point of view Scrum is just the framework.

So one of the issues I find that is that Scrum requires you to have a single Product Backlog and a single Product Owner. However, Analytics solutions often are at an Enterprise level and I am finding that are multiple Product Owners involved.

So it would seem that either something needs to be changed and a scaling framework like SaFE or LeSS might need to be incorporated ?

In my experience, this is not a peculiarity of analytics solutions. When your product is a part of a large ecosystem of enterprise applications and its services are impacting multiple departments, you will have multiple strong - as we call them in Scrum - stakeholders. Regardless of whether we have "multiple product owners" or "multiple stakeholders but no product owner", there is an elephant in the room: who is managing the product backlog? How do the development team know what they should work on? How do they resolve conflicting interests? How can they figure out what the solution should do at all? It is not Scrum's interest to have 1 product owner. Any other setup is simply suboptimal, even if viable. This is a question we cannot get rid of by applying SAFe or LeSS, though these frameworks may support the work of the product owner.

an implementation of Scrum is to create 1 Product, that's why you have 1 Product Manager and 1 Product Backlog.

The definition of the Product is the key. 

Is there several different Product ? Yes ? then what is the level of interdependency between them ? must each team coordinate often with each other ? Are the Products really different or are they in reality part of the same Product ?

After theses questions are answered, ask yourself what you want to achieve as a result ?

Then maybe look for Scaling technique IF NEEDED. Scaling should not be the priority, it should be last resort when seeing that the need can't be overcome in any other way. After you can choose what is best for you: Nexus, LeSS, SAFE, Spotify or any mix that suits your need/the maturity of agility of the different Teams, the business searched.