Skip to main content

ETL Development using Scrum

Last post 06:02 pm May 9, 2024 by Balaji Dhamodaran
3 replies
06:57 am May 5, 2024

Hi,

I'm an ETL developer and team leader (also known as data engineers), we are asked to start managing our work using Scrum.

I've tried to understand how can I fit an ETL Development to Scrum, here is my scenario:

  1. we get a requirement for data we need to gather - usually from sources outside our organization
  2. the project starts by describing the required data to our sources (it might take several months until we see sample data, which is usually not what we asked)
  3. than we need to design the QA / DQA for the whole process
  4. than we simulate the required data and start developing the ETL implementing the DQA within the process (might take more than a month it should be synced with QA team)
  5. starting integration with the consumers (usually BI)
  6. starting integration with our sources and QA (usually requires lots of back and forth - might take months until we see response)

I guess my questions are

  1. How can I manage ETL development that might take several months in sprints, it an entire process and can't be broken into sub processes or tested until completion
  2. How would you manage a project when your progress is limited by resources outside of your control, for example we needed to supply data by the end of April, We've had an obligation from our source that the data will be available by the end of march, in the end we've got the data at the end of April and the structure and granularity are wrong, and we need to reDevelop some of the process

Thanks,

Eyal


06:59 pm May 6, 2024

Who is asking the team to use Scrum, why, and what do they aim to achieve?

It's not clear to me if you need or can benefit from Scrum other than someone outside the team asking for it. Trying to shoehorn Scrum (or any framework) into a place where it doesn't belong won't lead to a successful team. If you're not operating in a complex environment and can get key stakeholder involvement to iterate and plan on the next steps, I'm not sure that Scrum would be useful to the team.


07:26 pm May 6, 2024

we are asked to start managing our work using Scrum.

I've tried to understand how can I fit an ETL Development to Scrum

Why? Presumably different outcomes are expected, and a different way-of-working will be needed. So why try and fit the process you have described, which would produce the same old outcomes, into something where it doesn't fit?

Scrum is about innovation, and learning to build the right thing at the right time. The conversation you need to have is not about fitting an old process in, but about the appetite in the organization for deep and pervasive change.


06:02 pm May 9, 2024
  1. How can I manage ETL development that might take several months in sprints, it an entire process and can't be broken into sub processes or tested until completion

With this constraint, You can only count the sprints but nothing will change.  

in the end we've got the data at the end of April and the structure and granularity are wrong, and we need to reDevelop some of the process

if the expectation is to address this challenge through Scrum, it would require stakeholder involvement as well. 


By posting on our forums you are agreeing to our Terms of Use.

Please note that the first and last name from your Scrum.org member profile will be displayed next to any topic or comment you post on the forums. For privacy concerns, we cannot allow you to post email addresses. All user-submitted content on our Forums may be subject to deletion if it is found to be in violation of our Terms of Use. Scrum.org does not endorse user-submitted content or the content of links to any third-party websites.

Terms of Use

Scrum.org may, at its discretion, remove any post that it deems unsuitable for these forums. Unsuitable post content includes, but is not limited to, Scrum.org Professional-level assessment questions and answers, profanity, insults, racism or sexually explicit content. Using our forum as a platform for the marketing and solicitation of products or services is also prohibited. Forum members who post content deemed unsuitable by Scrum.org may have their access revoked at any time, without warning. Scrum.org may, but is not obliged to, monitor submissions.