Team is not finishing committed sprint stories (goal)

Why are they committing to stories? That seems like a rather strange goal to achieve when work is complex. Is it another decision which management have imposed?

I see a few problems:

• Did your Sprint Planning session result in a Sprint Goal that represents the desired valuable step to be able to deliver by the Sprint Review? If not, you should work toward crafting good goals and making the work to achieve that goal more flexible.
• When you estimate in hours, are you estimating in clock hours (or wall time) or ideal hours? In my experience, people can get reasonably good at estimating the ideal time to do something, given that they have enough experience in doing the work and the context that they are working in. However, the problem comes when people treat ideal hours and clock hours as the same - there's almost always interruptions that reduce the number of actual, productive, hands-on working hours that a person has.
• Are you leaving a buffer? Similar to interruptions, there are other things that happen. There are often tasks or activities outside the Scrum Team's work (such as corporate mandated events). There is work that emerges by doing the work. You need to make sure that your goal can be achieved even as you deal with overhead or emergent work.

@Ian Mitchell By committing to stories I mean, the stories in the sprint backlog that have been refined and prioritized by the Product owner also factoring the capacity (in hours). We commit to this plan with the expectation to deliver an increment at the end of the sprint.

@Thomas Owens: I am just learning there's a difference between Clock hours & Ideal hours, you wanna shed more light on the difference please... Is the sprint goal crafted based on the stories in the sprint backlog or the sprint goal is first crafted and stories are then picked  that align with that goal. This could be something we can look into.

Yes we have buffers, i think its more about stories not been estimated accurately E.g: A story is estimated for 20hr but it takes 30hours to complete it, if the Sprint capacity for the team is 200hours (excl. Sprint events & a focus factor of 70%= 140hrs), total estimate of stories planned are about 140hours but because time spent to deliver some stories go beyond estimations, other stories ranked lower in the sprint backlog don't even get touched before the sprint ends....

There are three concepts to be aware of - clock hours (or wall time or clock time or a few other names), touch time, and ideal hours.

Let's say it's 8am on Monday morning and you start the work. You finish the work at 2pm that afternoon. The elapsed clock hours is 6 hours. However, you probably weren't working on that one piece of work for 6 hours straight. Maybe you got up to get a drink, go to the bathroom, chat with a coworker, respond to an email. Your touch time - the time you were actively working on the thing - is far less than 6. Maybe it's something like 3 or 4 hours. However, because you were interrupted at different points, not all of your time was high quality. It could take about 20 minutes or more to return to focus.

In my experience, when most people give an estimate in hours, they are thinking about the ideal time - the time it would take if they sat down to do the work, had everything they needed, and wouldn't be interrupted. In this example, that piece of work may have had an ideal estimate of as low as 1 or 2 hours. However, the interruptions may have almost doubled the time it took to complete it. And, on top of that, you're now 6 hours into your week.

So although estimating in hours can be helpful, if you have a pile of work that you've estimated to take 40 hours, you won't be getting that work done in a standard 40-hour work week. If a 2 hour task takes 6 hours to complete, you'll only get through 5 hours of that pile of work.

Although your buffer is a good start, it's probably not accounting for overhead unplanned work, interruptions, context switching (and recovery). It seems like you're already setting aside 30% of your time for planned work. Make your Sprint Goal 70-80% of that. So if you have a standard capacity of 200 hours, pull in 140 hours of Product Backlog Items, but then make your Sprint Goal something that is achievable in ~100 hours.

As far as crafting the Sprint Goal, in my experience, it's an iterative process. Typically, I see the Product Owner come in with an idea of a Sprint Goal. However, it's often lofty. When the team starts picking the Product Backlog Items that would need to be complete to meet that goal, it exceeds what they believe they can do. So there are discussions between the Developers and the Product Owner to find that balance between a valuable Sprint Goal, but also making sure that the Sprint Goal is achievable. Depending on your organization's (or stakeholders' or team's) tolerance for risk, you'll be able to adjust how aggressive your Sprint Goal is, or how much of your capacity is required to likely achieve the Sprint Goal.

In the short run, you may want to have a closer look at Sprint Planning and the use of the Daily Scrum.

- Does Sprint Planning produce the plan with enough granularity, which provides enough confidence in the Sprint Goal achievement? Sometimes the developers need to step down to a Day-by-Day plan.

- Do the developers inspect and adapt towards achieving the Sprint Goal on the Daily Scrum? The purpose of this meeting is to look at the plan and replan if things do not go how they should.

In the longer run, I suggest brainstorming with the Product Owner about how you can switch to Sprint Goals, which are delivering measurable Value but are flexible in realization. 

For example, the team is developing the game. The Sprint Gaol could be "Increase the User Satisfaction Score by enhancing the collaboration between the players"

Now, there might be many PBIs in the Product Backlog, which are about collaboration between the players. Such as - 'basic chat messaging', 'one-to-one Voice communication', 'Group chat', 'voice recognition' etc... 

This is just a simple example but of course, other products are not so straightforward.

It is not easy, it might require getting a new perspective on the product altogether.

You may want to read some articles about Product Goals, Sprint Goals, and Evidence-Based Management to get a solid grasp of Value and Goals in Scrum

Switching to Goals/Value-driven development might also need broader discussion with other stakeholders and the Scrum Team.

You may want to call for a bigger audience meeting and use some facilitation techniques (such as "1-2-4 All" from Liberating Structures) to collect the ideas from the bigger group and whittle down a few best ones. 

My team has not been completing stories that are being committed to in the sprint

We commit to this plan with the expectation to deliver an increment at the end of the sprint.

The commitment coming out of Sprint Planning is to the Sprint Goal, the single objective of the Sprint. Not the plan, not a list of stories. Both of these may need to change when building adaptive solutions to complex problems. The Daily Scrum is an Event purpose built for adapting the plan and the Sprint Backlog (if needed) as the team progresses and learns throughout the Sprint.

Any backlog items (stories) pulled in at Sprint Planning are a forecast of what the team believes is needed to achieve the Sprint Goal. Forecast, not commitment.

Is the problem that the team is unable to achieve their Sprint Goals? Are those goals focused and coherent? Perhaps look at how Sprint Goals are being drafted and leveraged (particularly if they are lists today). I have found success in leveraging this template from Steve Trapps as it focuses on outcome, impact and confirmation. Thomas offers some great advice above for buffers or leaving room for handling change and adaptations...

due to complexities

I agree with everything that everyone has already said.  I will also admit that changing the perspective from hours worked to a Sprint Goal is not easy.  It takes some experimentation and practice to get it right.  However, that is exactly what empiricism is based upon.  So it should be familiar to everyone that is using the Scrum framework.

While your organization does not use Story Points, I'm going to point you to this article anyway.  It is written by Ron Jeffries who is said to have created them.  In the article he explains your problem.  I am not providing this to advocate that you use Story Point estimation. I am sharing it because it shows that this problem has existed for a long time and there still isn't a single "best way" to handle it. 

Very helpful tips and perspectives, I will digest them and experiment as it fits. Thanks!

The Scrum is a risk management tool if you look at it from purely business perspective. It highlights problems early, and this is what he did this time.

So what is the problem? The whole approach. The Scrum team does not produce completion of story points or other forecast metrics. Its not the purpose.

It produces increment of value(or fails which is a feedback loop for adjustment)

What is a "value"? Ask your PO. In most cases its revenue, if you have non profit then establish it based on what your goals are.

Likely unrealistic goal is agreed upon every time. 

If product owner is not strict on goals but developers agree with the goal and still estimate it so they can never reach the goal, commit to it wrong then its on scrum master to coach them on how to estimate it better.

What you want to avoid is pushing someone to overcommit as it is easier to move item back to backlog and complete something slower then find a new dev once he burns out. 

Perhaps you could try coaching them all to start with a goal that team can make, make a very small goal that everyone is 100% sure they will make it.

Do not think scrum intention is to make everyone pushed to the limit in the name of being productive, you will just incurre hidden loss from losing devs to other companies.

If company does not agree with this assessment I am not sure that scrum has anything to do with it, it is just one agile framework not a magic formula for every organizational problem.

To add some thoughts, if you are looking for the reason why team is not able to convert the PBIs they choose into increment of value(I understand that this is the situation? Or please correct me if I am wrong),then the  problem might be deeper then techniques which Devs choose for estimating velocity.

Using Time-based estimate is not a bed technique by the way(like estimating WIP time for each item, or user story if you call it that, or use cycle time). And Scrum team should be able to produce increment even if they use time-for-work to estimate the work.

Also if you and developers team feel that you want to use story points for estimation, what is an objection?

I understand that organization wants to the to use time, which is already an indicator that there are deep problems in the relationship between Scrum team and organization. It means team is not self managing under this circumstances...

 But here is a question.  What is an objection to estimate both in hours and story points? After all the estimation is internal job by developers for developers? Or the bosses not only ordering your team to have estimation in time, but also prohibiting Scrum team from having their own internal techniques for using velocity?

To make long story short if the firm is going so deep into micromanagement of the Scrum team, the project which is run there isn't Scrum yet. It is called Scrum on paper only.

If firm WANTS to have Scrum, but does not know how, then Scrum master can help them understand the framework better.

If they don't, and if they call the project Scrum for some other reason(why?) then just accept that in practice Scrum master is simple team manager in waterfall project there, and act as such, or find other place to work if you want to work in Agile environment