what happens when you don't finish a story in a sprint? fractional story points?

Look at the other work remaining in the Product Backlog. You'll likely see 1, 2, 3, 5, 8, and 13 point stories.

Is the difference between an estimate of 0 or 1 for this unfinished story significant...and if so, why?

If the Product Backlog Item (in this case, the story) isn't done, it's not integrated with the rest of the Product for a potential release. I would suggest that there is no "partial credit". The team does not get to count this story as completed, whether they count done stories or velocity or something else.

Something else to keep in mind is that the purpose of estimation and sizing is to help the team assess how much work is appropriate to bring into a Sprint. It's independent of the "done"ness of work, which is a binary yes or no.

My recommendation to the team is that they would not count this story as done and it would go back into the Product Backlog. The Product Owner can reorder the Product Backlog. One of the considerations would be the value of doing this work in the next Sprint versus the partially done work. It may or may not be appropriate for selection in the next Sprint.

Maybe you finish 80% of it.

I find it hard to tell how many percent you have done. You can say you worked a certain amount of hours or so. That's empiricism, but you can't tell how long you will really need for the rest. So is it 80% or 70% 

If my team has a sprint backlog item that is not done, it is not done and the points will not be counted for that sprint. PO will then have to decide if this will be finished in next sprint or put back to PBL.

If it is taken over to the next sprint and will be continued, we will re-estimate. For example we will say that implementation is done, but documentation and review still needs to be done, by that it is no longer a 5 to go, but a 1. We do that, as I don't want to spoil the velocity by keeping this a 5 point, which were not delivered in that sprint.

There is a short sentence in the cancelling a sprint paragraph: "All incomplete Product Backlog Items are re-estimated and put back on the Product Backlog." I guess the Scrum Guide otherwise assume it never happens that a PBI is partially implemented at the end of the Sprint...

What you do is look at the unfinished Product Backlog Item (PBI) together with the PO and, based on value, decide if you need to finish it and put it back on the product backlog or if it has become obsolete.

If the PBI has lost its value, every extra minute spent on it is wasted. Delete the PBI from the product backlog and delete all the work you have done for it so you don't leave a mess. 

If it still has value, I usually suggest doing it first thing during the next sprint to not lose the knowledge your team now has of the PBI. A small story that is 80% completed usually will not disrupt the new sprint goal.

If you want to finish it in the future but right now it is really low priority, then you are probably best off deleting the 80% of the work you have done so far. It is inventory (waste) that will not evolve with other parts of the product. The current knowledge of the team will diminish, making it hard and time consuming to pick it up again when it is given priority again.

As for story points, who cares?

One story point is trivial and won't distort any velocity trends, if you feel the need to have these. If you really can't resist the urge to award every single story point so it reflects in the average velocity, then I would count the full amount of it in the sprint where it is completed. Also, I would spend some time during a retrospective reflecting why story points are so important to you. There probably is some kind of dysfunction somewhere in your organisation or in your team.

Predicting a dysfunction if Story Points are used and reflected is a bit thick.

Why are Story Points used? To estimate the Backlog Items with less waste than an old school estimate and to be able to say what can probably be done within a sprint. In a maturing team the velocity may go up, but if it goes up, as you have taken over a number of points that were actually delivered the sprint before, the velocity will no longer support you.

Thanks for calling me thick, that really helps me grow. What I thought I wrote is that using velocity as an absolute measure of work you did in the past is a sign of a potential dysfunction. It is a sign that the team or the larger organisation is focused on output rather than on outcome.

In the past I have heard remarks like "But I have worked on this for 9 hours, I should be rewarded for that work with some story points in the velocity". In all those cases that I experienced, the customer judged the teams based on velocity. Deliver less points this sprint than last sprint and report to management.

I have even seen contracts where a minimum amount of story points was enforced per team. If the team did not deliver X story points in a sprint, the customer would hold back a percentage of the amount paid per sprint. Long and unpleasant arguments occurred where the customer would argue that a certain story was 3 story points and not 5 as the team had estimated. It was an utter waste of time and energy, all of which could not be spent on actually delivering working software. It turned into contract negotiation, rather than customer collaboration.

What matters in Scrum is delivering value to the stakeholders. Having worked on something for 9 hours without finishing it does not deliver value to the stakeholders. Why would you need to be rewarded points on effort without outcome? In the next sprint, you will probably finish it and the point will be part of that velocity. It will weigh the same as 0.8 plus 0.2 in averages and trends.

Yes, you can use story points to estimate future work. And yes, you can (and should) use average numbers from the past to have some kind of idea how much you might be able to do in the future. But you have to realise that all estimates are inprecise and therefore wrong. We are not psychics and cannot precisely predict the future. Past experience and trends may allow you to guess better and be less wrong.

Is there a dysfunction per definition when a discussion arises whether or not 0.8 story points should be counted for unfinished work? No. But I would still investigate whether there is one. Which is what I thought I suggested in my previous response.

I meant not you in person, only judging the usage of points and velocity as a dysfunction.

Using points and velocity is not neccesarily caring about output over outcome. You can still have a good Product Backlog priorization based on value to optimize outcome and use points and velocity from the past (empiricism) to predict what the team is capable of delivering.

If you grant points for undone work in the old sprint, just to have a velocity that should tell how much you have done, that is questionable, yes. In my case it was just for the new sprint, not to have full points.

Contracts is a different topic and yes, there you should not work with points