Effort for Increment got out of hand: how to get a grip on it as a team?

From a Scrum Values perspective, you could call this a problem with focus (on the agreed-upon scope and goals), openness (with the work being done), and commitment (to the Sprint Goal).

I don't think it's a matter of the developer sharing their vision, especially once the work started. I would want to understand what was happening during refinement when the problem and work were being discussed and decomposed, during Sprint Planning when the Sprint Goal and the initial plan were created, and during every Daily Scrum when the expanded scope became apparent (or didn't become apparent) to the team. There may be contextual information as to why the developer(s) doing the work weren't open or why the team didn't see what was going on at the Daily Scrum.

One of our developers conceived of an idea that this change would benefit other entities

Had the idea been empirically validated with even one entity at this point?

This is something that I have experienced almost to the exact details. Here is how the team I was working with handled it. 

At the end of the first Sprint, it was brought up in the Retrospective.  Discussions focused not on why did we "fail" but on how did this occur.  It turned out that the developer that expanded the scope did not realize that his expansion actually had an effect on the work that others were doing.  In essence, the team discussed the work, had a plan that everyone understood but some of them understood it differently than the others.  There were not enough questions asked and no one took the time to say "here is how I understand our plan, is that correct?".  That question became a standard from that point on.  

We explained this to the stakeholders and they actually laughed about it because it had happened to them on some work that they were doing.  We all learned from the experience. 

The team then revisited the work, determined a path that would allow delivery in two sprints but it did not allow all of the "expanded" functionality.  The work to add the "expanded" functionality was added to the Product Backlog and was worked on during future Sprints.

Remember that you should expect that no one will intentionally do anything that caused harm.  Help everyone realize and understand that his isn't a failure, it is an opportunity to learn and help them to learn.  Help them understand what happened, how it occurred and then encourage them to find ways to account for the situation in the future. 

Thank you, everyone! That is quite a bit of food for thought. I appreciate it. Time to reflect, but also to follow up.

Starting with the question from Ian. I will also follow up with Thomas and Daniel.

Had the idea been empirically validated with even one entity at this point?

Good question, had to give this some thought. The answer is "no". The team did not have the opportunity to roll out the change and see its effects. Nor functionality-wise, nor from an architectural/technical point of view. Thanks, this is a good argument for working incrementally, and using the intervals that are sprints to build up the architecture for the new plan.

At times, our team is struggling with architectural complexity,:"might as well build this now, for the future". However that I have opted emergent architecture before, there were a lot of counter arguments against it. Counter arguments like: "we'd have to revise the structure all over again every time we make changes to the architecture" and "it would cost us more effort to update existing architecture over and over again, instead of doing it well all at once". 

But how sure can you be of the validity of those counter arguments? And did the developers actually experience the increase in effort? What's more, did the developers experience negative side effects of the increase in effort?

Validating ideas (business value, architectural impact, technical added value) is something that the team needs to take to heart, to a greater understanding than it does now, I assume.

Following up on Thomas:

I would want to understand what was happening during refinement when the problem and work were being discussed and decomposed, during Sprint Planning when the Sprint Goal and the initial plan were created, and during every Daily Scrum when the expanded scope became apparent (or didn't become apparent) to the team.

First up: our dailies. 

In hindsight, I don't think the team members were sharing the right insight into what their plan for reaching the goal entials. We've been too focused on the three questions, even though we do not answer them explicitly and compulsory anymore. But I struggled with giving the team guidance on what actually is meaningful to share at the daily. Would the following be correct?

• Share which share you have in achieving the sprint goal, not what you did yesterday.
• Share your plan, the bigger picture, you want to achieve, and the next puzzle pieces you will realize today.
• Share the difficulties you face while trying to achieve this.

Next: backlog refinement. 

We did miss a degree of understanding the technical scope of the backlog items. As in: grossly underestimating items time and time again. Hence, we allowed technical discussions on backlog items, to ensure the developers could give a proper estimation. So when we discussed the backlog items in question, we felt more confident on our understanding of the matter. Do we not touch the scope of the stories, unless we are refining the backlog for future sprints? I guess under the flag of "working pragmatically", the team tends to be rather flexible with the scope, relying on judgement of individuals rather than the team's. So I think we should use the backlog refinement more for this purpose. To attempt avoiding to cause harm to the scope of the sprint and its goal.

Regarding sprint planning.

I had the feeling that, when we started the sprint, there was consensus about the goal. But I am not sure of the shared understanding, or the sense of the goal's intention  (scoping just to what is asked, rather than potentially building some of us think that we need in the future). I think following up with the team multiple times throughout the session might be a good idea: "do you understand XYZ?" Where "XYZ" refers to:

• The scope
• The goal
• The technical impact
• Whatever there might not be consensus, shared understanding or shared sense of purpose for yet

I think there are some points of improvement, based on my reflection. Would this be on the right track? 

To address Daniel: very relatable, indeed.

Our retrospectives focus on our "failures", and don't seem to delve deep enough into the "why". I think I failed to embody the desire to delve deeper into finding the root cause.

Discussions focused not on why did we "fail" but on how did this occur.

Time to revisit "5 why's", and lead by example.

Here is how I understand our plan, is that correct?

To ask this question at key moments (daily, planning, refinement) will involve more openness about what is in everyone's mind. I will try to evoke this line of thought: "what is your understanding of our plan?", following up with "is this correct?". The team was able to pick up "new habits" in the past, so I think that this evocative approach might work for us.

Remember that you should expect that no one will intentionally do anything that caused harm.

I never doubted this with the team. Their intentions are pure, however that sometimes they are headstrong on the matter of technicalities and architectures.

Help everyone realize and understand that his isn't a failure, it is an opportunity to learn and help them to learn.

Over time, the willingness to embrace a situation to learn from it eroded slightly. It is good to remind everyone about this once more.

Help them understand what happened, how it occurred and then encourage them to find ways to account for the situation in the future.

Based on all your replies, I am able to reflect personally as a scrum master. However, my reflections are based on assumptions. My guess would be to discuss my assumptions and ideas about going forward during the upcoming retrospectives, and at the times that my ideas would apply.