Data Integration development and Scrum
I work as a BA within a team that predominantly works on Data Integration activities. We have a vendor system, into which we load different types of data, from a variety of sources. We validate the data and create a golden copy. We export data from it, with differing content and formats, it to a variety of target systems.
Traditionally the BA’s have written mapping documents (using Excel) to describe the data import/export requirements. These documents are fairly tightly controlled and aligned to the software release versions.
As a team we have recently moved to a Scrum way of working and User Stories have been developed which include the mapping document. To keep the stories small they are not entirely independent; there is a basic story and then subsequent refinements.
Not a great example but here goes:
Story 1: Send of file of all invoices to system Z.
Story 2: Only send invoices from a subset of suppliers to system Z.
Story 3: Only want to send invoices that haven’t been fully paid to system Z.
Story 4: Only send an invoices over a certain value weekly, all other are sent monthly.
You get the gist. So the mapping document would normally include these additional criteria specified in stories 2,3 and 4 but I do not want to include these in the mapping document for Story 1 but I also do not want to have 4 versions of this mapping document that I need to keep updated (for example an error in version for story 1 would have to be corrected in four versions).
How would others approach this issue ? I’m all for following the “working software over comprehensive documentation” paradigm but our dev and test teams are difficult to budge.
Why do you think any of these constitute user stories? How do they promote discussion, and the emergence of scope? What is the problem with the old way-of-working that you are trying to solve, and why do you think Scrum might help?
To be fair, the decision to use Scrum as a way of working is decision that was imposed on the team and I'm not sure exactly what problems were being targeted. Sadly with all the Agile hype it's seen as a magic bullet and also a means of furthering one's career by having a CV full of Agile projects. Having used Scrum elsewhere at another organisation, I'm not convinced that it is appropriate for this type of work.
However this is what I'm faced with and so I'll press on.
Peter,
One of the popular guidelines to creating PBIs is that each item should be able to stand on its own (independent). Does attaching Mapping Documents, instead of documenting the requirements and acceptance criteria in each item, support the independence of each PBI?
I see nothing wrong with your 4-story example. In my opinion, you seem to be creating unnecessary complexity by attaching common mapping documents (requirements) to multiple PBIs.
Perhaps any proposed Mapping Document change should be managed by a separate PBI that pertains specifically to that change?
So glad to hear about your team's journey towards Scrum for Data Integration. I'm also wondering where I can get the best data integration tools? Tnx!
Hi @Peter Davis,
I don't see any problem with the way the Stories have been created by your team. What your team is actually trying to do by way of this is to build the mapping document 'incrementally' - one brick at a time. There is definitely value in doing this because team would be able to gather feedback fast on completing the first Story (a catch all Story) and then build upon that as part of subsequent Stories.
And in case there is any defect in Story 1 (which I ideally believe should be worked upon before starting work on Story 2), even if the rest of the Stories are completed, change would only need to be done in Story 1 & not the rest of them, as long as the filter criteria for all the rest of the Stories remain valid.
The important thing to consider here is that the purpose of splitting PBIs into smaller units is not only that it should be completed within a Sprint, but also to speed up the feedback cycle on each of them. By splitting the Stories in 4 parts, you're actually speeding up the process of gathering feedback in each of them individually, rather than waiting for the last one to be completed (final brick laid) and then gather feedback on the whole.
Great to see the team is adapting to Scrum for Data Integration development. I get the challenge of incorporating additional requirements from the User Stories into the mapping documents. Have you considered using a data integration tool to streamline this process?