estimation by non cross-functional teams

So first off, Cross-Functional does NOT mean Full-Stack Developers. Full-Stack Devs are folks that successfully write code, test the code, on all aspects of the product (front end, back end, etc etc). Cross-Functional teams that contain all the skills necessary to build the product. 

That said, cross-functional teams discuss the different values given and then come up with an agreed upon point of value. For instance, in estimating a backlog item the following values are given from each component level:

Front end - 5

Back end - 8

QA - 2

UAT - 3

Everything area is all over the board so the team would discuss why it's a larger effort for Back end but relatively easy to test and the team decides on an amount to use for the backlog item. It doesn't matter if they do the average of all of these (going with roughy a 5) or they could go with an 8, or really whatever value the team wants. There is no right or wrong answer for this. Estimating is for the team only, purely for the reason to use this information for planning.

I once developed in a team where we would estimate our perception (in story points) of the collective effort for the entire team to get an item to "Done".

We sometimes had discrepancies, of course, because we didn't fully understand each other's specialist skills, but the act of trying to understand where (for example) our QA would find it difficult to test something gave us greater empathy for each other, and helped us find alternative solutions that worked for the whole team.

And we eventually found we were pretty good at predicting when something would require more backend or frontend work, or would require more thorough testing. We were able to come up with relatively consistent estimates.

From your comments I would say that you have a cross functional team and that the exercise is doing exactly what it should.  Does the team discuss and arrive at an estimate they could all accept?  If so, why do you feel it isn't working?

in poker planning each member of the team chooses a card based on the complexity from his point of view 

How well do they collaborate during the Sprint? How does each team member demonstrate teamwork? Do they ever show an interest in someone else's point of view?

This is my problem. My QA writes thorough test cases that take up a lot of time. So when the team has tried coming up with a collective number, my tester has a number always on the higher side. he never agrees with the developers. he then tells them why he is asking for a story to be a 13 pointer instead of 8 (all developers opinion). and since we are non-cross-functional, I have to go with the QA since he is the one who is wearing the QA hat and responsibilities. 

can you help me with how to coach the team with this mindset? Also, the ratio to the developer and a is 5 to 1, there are still few stories that go non tested in the sprint since my QA cant pick up more. in this case, we split the stories as dev and testing. the reason being, 

example 1 : Story A : effort 13 (collective effort) , since only developer is working and no QA will touch it in this sprint, this entire story gets spilled

example 2 : Story A : effort for dev 8

Story B : QA effort for 5

in example 2, we never take the story B in sprint scope, and only pick up when QA has bandwidth. devs pick story A, complete it , we mark it done.

What is the reason that the QA writes detailed test cases if they are the only individual that will use them?  And are they ever used again?  If not, that is waste (taken from Lean Programming) and needs to be addressed.  I have spent 20 of my 33 years of software development as QA.  One of the first things I learned is that those very detailed test cases I was writing that I used for manual testing were a complete waste of time.  No one but me ever used them and if I had a reason to revisit that part of the product it was because something had changed and my test cases were usually out of date.  I learned that manual exploratory testing was much more effective than detailed scripted testing.  But that is hard for some people to learn/understand and some companies have policies that require the wasted effort of creating very detailed test cases. 

So, if your QA is required or insists on writing wasteful detailed test cases, you have an opportunity to have the developers execute those test cases instead of waiting on the QA to do so. If a story is expected to be done early in the Sprint, the QA should focus on that story first and write the test cases.  Then the developer can run those test cases while the QA focuses on writing other very detailed test cases for another story.  

Another option is for the QA to be included in the refinement activities for the stories and introduce the testing plan at that point.  Then the entire Scrum Team can discuss whether it would be possible to break the stories up smaller or find an alternative for testing, such as developers writing automated unit/integration/system tests to cover a majority of the backend functionality so that the only UI focused testing needed is to check that the UI is behaving properly.  

In the end it is up to the team to decide how to address this.  As a self-managed, self-organized, cross-functional team they should be empowered and willing to discuss this issue and arrive at a solution that they are all comfortable with using.