Newly taken over role of scrum master of critical project

I have taken over role as scrum master of troubled project.

Trouble can be minimized and better controlled once it is recognized that each Sprint is a project.

The team size is 20+, it contains backend, UI, QA team.

That doesn't sound like a team. It sounds more like a structural impediment to teamwork and to the collaborative delivery of finished product increments.

They are following scrum method.

There isn't a Scrum method to follow; what you will find is a framework to implement.

There is huge number of defects open, stories are getting spill over. Customer has strict timelines. they are following 2 weeks of sprint cycle. 

When Sprint timelines are strict work cannot "spill over": any undone work is reviewed and re-estimated on the Product Backlog, such that it may or may not be planned into a future Sprint. The Product Backlog should tell the truth at all times about how much work remains to be Done including any technical debt.

Not sure how to start . they have tried all combinations like splitting the team, dedicating one sprint to fixing all defects but not able to manage. Lot of pending work / technical debt. 

Any suggestions to improve the situation ? 

Scrum begins with Done. That's where to start. Encourage teams to self-organize around a Definition of Done and to pull work so a finished and usable increment -- however small -- is delivered at least once per Sprint. That's how empirical process control will be established and maintained. Use the Scrum Guide as your friend.

Find the reason for pending work/ technical debt. Work with PO to understand the PB, Business expectation, Teams expectation. Reason for turbulance could be 

1. may be team members are very new/changing always (very few stable team members)

2. Team need expert advice on few items

3. Environment hurdles

4. Team health - how team is formed, how team is feeling for the project

5. Team collaborative instinct, communication method/frequency

6. Availability of BA/PO for any doubt clarification

7. Silos inside siloed teams

8. What it means by done to Team, to PO - set the expectation right in terms of acceptance criteria.

9. Knowledge silos rather than team goal/team work - as SM work on forming the team, understand pain of team. No split of work, no dedicated sprint for bug solving will work if team doesn't know why to proceed , where to proceed (direction), how to proceed(expert advice)

Talk to product owner. Understand their pain point. Things can be sorted in due course of time.

Come up with plan , work with Team and PO to get high prioritized items list, work on them, gradually work on delivering the items.

Focus on delivering few prioritized items at first as per acceptance criteria. Things will clear up gradually with Scrum implementation. 

Let's break it down.

First the size of the team doesn't help for sure. You said it was tried before to split it and it didn't work. How was this done? Why didn't work?  

Then you said they are following the Scrum method. There's no such thing and in the long run it would benefit you to study and have a deeper understanding on how to apply Scrum.

Finally the technical debt. That can happen for several reasons like:

• weak definition of done
• definition of done not respected
• lack of inspection
• ...

The list can go on and on. 

The best advise I can give you is to tackle that big problem of yours in smaller pieces. Take a bite into the size for example. After you have that solved, discuss the Definition of Done. And so on.

Good luck on your journey!

Customer has strict timelines

Timeline and backlog both cannot be fixed. I see the reason for issues could be the team want to plan more in their sprints(knowing that it could be more than their capacity) to satisfy may be the management ?. And end up with quality issues and spill over stories with the push to complete whole backlog in the given time.

A product can have multiple Product goals. They need to fulfil one objective before taking the next. Do your product have a Product goal ? That could be an immediate step to suggest PO to come up with one and get it agreed with the customer and other stakeholders. Commit to the Product goal not to the product backlog.