Agile Analytics: Slicing Data Warehousing User Stories for Business Value

Value-Added Agile Strategies

How to Deal With Common Issues Encountered During a Transition to an Agile Approach

I am excited about my latest publishing adventure with The Cutter Consortium. The article is called, “Agile Analytics: Slicing Data Warehousing User Stories for Business Value.”

Before you read it, I first want to tell you about Cutter.

Cutter Consortium is a global information technology research company located in Massachusetts founded by Karen Fine Coburn. Their research focuses on the areas of agile software development, agile project management, business architecture and enterprise architecture, business intelligence, outsourcing, business-technology strategy, risk management and benchmarking. Their services include research, training, and consulting. Cutter is highly regarded and very well known, particularly on the East Coast. The best agile experts work with Cutter, like Tim Lister, Israel Gat and Tom DeMarco (just to name a few!).

Cutter is big on education and working with executives. Every year, Cutter sponsors the Cutter Summit, a three-day executive education program that provides a high ratio of experts to attendees. This year, I will be giving a workshop called Iterative Development Strategies for Business Intelligence Projects. The workshop will include my best work and ideas and I can’t speak highly enough of the Cutter Summit. I think you should sincerely consider attending.

Now, on to my article.  Included in Cutter’s educational services are several publications. One of them is a monthly journal, available for members-only, called The Cutter IT Journal. It is a collection of half a dozen articles on a specific topic selected by a guest editor. I’ve had the honor of serving as guest editor, working closely with the Cutter staff and remarkable agilists putting together the publication, Does Agile Equal Better DW/BI?.

In March, Cutter Senior Consultant Dave Rooney wrote an introduction to the Cutter IT Journal,“Value-Added Agile Strategies” (Vol. 28, No. 3). In his introduction, Dave explains that this issue is a follow-up to the October 2014 Cutter IT Journal issue,“Agile in the Real World.” This March issue focuses on “strategies and techniques that have helped organizations deal with common issues encountered during a transition to an Agile approach.”

I wrote an article for the March edition for those involved in Data Warehouse and Business Intelligence.

Agile Analytics: Slicing Data Warehousing User Stories for Business Value

by Lynn Winterboer

Our second article, by Cutter Senior Consultant Lynn Winterboer, examines one of the less traveled paths of Agile — the implementation of Agile approaches in the DW/BI space. Winterboer addresses the challenges of breaking this type of work down into small slices, when it has traditionally been an “all or nothing” deal. She describes what is perceived as technical inefficiency in the small slice approach and how that is balanced by the business efficiency. Winterboer shows not only the advantages gained by earlier delivery of value, but also the effort required and “instability” incurred during the transition. In the end, both organizations in her case studies found ways to deliver smaller aspects of the total solution, providing more value sooner to their stakeholders.– — Dave Rooney, Senior Consultant, Cutter Consortium

Dave summarized my article very well!

THE ARTICLE:

Teams (and management) are often more confused about “Agile” when they focus on the process (i.e., what the book, the instructor, the article said they need to do) rather than the values and principles that inspired the process to begin with.1 In data warehousing teams, I often see this issue manifest itself in the following key areas:

• Product ownership
• People management
• Program management

This article will begin the discussion on these topics by addressing the challenge of slicing data warehousing and business intelligence (DW/BI) user stories into small, business-valued deliverables to align with the Agile principle of “Deliver[ing] working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”

Breaking technical requirements down into small enough pieces that the team can deliver something of value to the end users every few weeks is a really difficult challenge for most development teams. DW teams in particular regularly get stuck on delivering something “valuable” in a short timeframe. I don’t think we are significantly different from other development teams that want to learn about and start their Agile journeys, but the available articles, books, and training materials are usually not crafted in DW/BI-focused domain languages, and therefore it can be hard for us to make the leap of imagination and understand how to apply these established ideas to our work.

Many DW/BI teams investigate “Agile” enough to decide it’s not right for them, since they can’t imagine how to get something valuable from the source to the DW and finally to the BI layer in such a short amount of time. We are typically used to delivering to our users large, complex solutions where “Value” is perceived to be at the highest level when executives get keen business insights to drive revenue and reduce costs. We, and our users, are not used to thinking of the sort of value with a small “v” — something useful and necessary to get to Value with a capital “V” — that can be demonstrated to our users in a matter of weeks.

Ideally we could slice the end request into such teeny pieces of work that the delivery team could pull data through from the source, all the way through all the DW layers, and present it in the BI front end in just a few weeks. That is the goal. However, it doesn’t mean the team should give up and never start their Agile journey if they can’t figure out how to do that after reading a few books or taking some training. There are several ways a team could start slicing stories into small pieces, including these two:

1. Pull one field at a time through all the layers of the architecture
2. Identify the business value in steps of the DW/BI delivery process

It is possible to pull fields through from source to BI, one at a time. It’s most likely not going to be technically efficient to do this until the DW/BI team has the right enabling infrastructure (configuration management, version control, continuous integration, and test automation2, 3) to safely and quickly deliver “thin steel wires” of value on a regular, frequent cadence. However, the question DW/BI teams and their stakeholders need to ask themselves is whether it’s more important to be technically efficient at the cost of business efficiency, or to be business-efficient at the cost of technical efficiency. This is a HUGE decision! In the following case studies, drawn from my experience, we see how two organizations answered this question in opposite ways.

CASE STUDY 1: PULL ONE FIELD AT A TIME THROUGH ALL THE LAYERS OF THE ARCHITECTURE

This approach can be the most business-efficient, yet it may feel very technically inefficient to DW/BI teams.

SCENARIO

Company A is a 1,000-person Web-based marketing company that had recently acquired a competitor, Company B. Since it’s likely that customers would buy from both Company A and its competitors, the executive team wanted to understand how many new customers had been acquired with this purchase, versus how many were already customers of Company A.

REQUEST

The initial request to the DW/BI team was to provide a report identifying the number and percentage of total customers who were new to Company A so the marketing, sales, billing, and support teams could understand the scale of effort it would take to integrate these new customers into the company.

FEASIBILITY

In terms of DW/BI requirements, this is already a fairly “small” request, yet the team was understandably hesitant to commit to this within the company’s standard two-week Scrum sprint for the following reasons:

• The team was fairly new to working together. It consisted of recently hired contractors brought on board at various times over the prior few months, so team members did not know what they could or could not accomplish together in a few weeks.
• The team did not know anything about the source systems of the newly acquired Company B — not even how many sources there were for customer data, let alone how to connect with them, what the data quality would be like, and which source system SMEs could explain how to find the data they needed to answer this question.
• The DW/BI team suspected that the request would soon lead to many more detailed requests beyond just “What number and percentage of the total Company B customer set is new to Company A?” Consequently, they felt they should really do a full customer data integration in order to quickly answer future unknown needs (a very common perspective among DW/BI teams that have been successful in traditional project management approaches).

CONVERSATION

I encouraged the team and the product owner to embark on an exploratory conversation to figure out if there was anything the team could do in two weeks that would provide value to the organization. After an interesting — and challenging — conversation, they came to agreement on the approach outlined below.

COMMITMENT

The scope of this first story would be limited to:

• A single new source of Company B customer data, identified either by an SME at the newly acquired company or by a quick profile of candidate sources
• A single email field within that source, identified by data profiling as being most populated with customer email addresses (There would be no data quality work to improve the values in the single email field.)
• De-duplicating against Company A’s existing customer database and “Primary_Email” field by comparing this field to the email field pulled from the new source

The demonstration of this first story would show a simple tabular report (see Table 1). The executives agreed this first step would provide value in that it would give them a “directionally correct” idea of how many new customers they had acquired.

SUBSEQUENT STORIES

Subsequent sprints would provide more and more accuracy for this request by:

• Adding a phone number from this same source to the de-duplication rules
• Adding other email and phone number fields from this source
• Adding other sources of Company B customer data, if any

Eventually, the DW/BI team would answer the question of how many new customers had been acquired from Company B by eliminating duplicates identified by a match on an email and a phone number on a customer record.

THE TEAM’S REFLECTION

PROS

This approach worked really well for this new team in terms of enabling them to quickly understand what they could accomplish in a two-week sprint and building a sense of confidence in themselves as a team. They also appreciated the clear understanding of what the organization was going to do with the information and how much was “sufficient” to meet the organization’s needs at this time. Furthermore, no one on the team had deep testing experience, so the very small scope of the effort was a relief as together they designed the tests for each field. They also found they could do rudimentary automation of such simple tests and reuse them for subsequent sprints.

CONS

On further reflection, however, they still felt that it was quite technically inefficient to pull in only one field at a time. If they’d had four to six weeks of focused time, they could have completed a much more elegant and efficient customer data integration that would have included more than just email and phone data.

As data professionals (and as contractors!), it made them uncomfortable to be “moving so slowly” given all of the work the company wanted from them. Although the business was seeing results much more quickly than it would have in a traditional approach, the relative technical inefficiency of this approach made the team feel that it would take much longer to finish the whole project.

THE BUSINESS’S REFLECTION

PROS

The executives started to get a feel for the scale of their customer integration efforts once they understood roughly how many new customers they had acquired. As an Agile company, the business was thrilled to see the DW/BI team making strides toward embracing Scrum and learning to deliver small bits of business value regularly. This team had been the last of all the company’s IT teams to convert from waterfall to Scrum and had experienced a lot of turnover in the process.

CONS

The executives also agreed that there was a lot of work ahead for the DW/BI team due to the disruptions caused by the team’s instability while adopting Agile. Since the enabling infrastructure used by the rest of IT (code management, version control, continuous integration, and test automation) was not yet easily accessible by the DW/BI team, this technically inefficient approach concerned the business as well.

They were also worried about the cost of paying contractors for such detailed, thorough work when there was such a huge backlog of DW/BI needs. They wondered if there was a balance that could allow the team to be more technically/budgetarily efficient, while at the same time delivering business value in small increments. Thus far they are still trying to find that balance.

CASE STUDY 2: IDENTIFY THE BUSINESS VALUE IN STEPS OF THE DW/BI DELIVERY PROCESS

This approach can be more technically efficient for DW/BI teams (and therefore more efficient from a budgetary point of view), yet it requires a more DW/BI-savvy business product owner, and therefore may be perceived as less business-efficient.

SCENARIO

A large financial services company was two years into a fairly successful IT Agile transformation, leveraging mainly the Scrum framework with some teams experimenting with Kanban. However, the many DW teams were still struggling to figure out what “Agile” meant to them, particularly since most of the Agile support in the company was for two- to three-week Scrum iterations. An example of a business request the data teams found hard to commit to and fulfill in one sprint involved working with external data vendors that enhance the company’s data assets.

REQUEST

A recent business request was to support a new cell phone texting marketing campaign that required opt-in data on existing customers, to be provided by a third-party vendor that the company had not worked with in the past. The basic flow of the data included:

• The data team providing a list of candidate customers and cell phone numbers to the third-party data enhancement vendor
• The vendor evaluating each customer cell phone number to determine if the cell phone is eligible for text messaging marketing
• The vendor updating the original file and returning the results to the data team
• The data team integrating the results into the DW
• The data team providing a file to marketing with relevant customer data for those who are eligible for this marketing campaign

FEASIBILITY

On the surface, this looked like a fairly straightforward request. However, the data team was not comfortable committing to delivering all of this functionality in a single three-week sprint. They had estimated two to three sprints to complete the work in order to accommodate the uncertainty entailed in working with this new data vendor. They had originally set the expectation with the business that they would have a “demo-able” product at the end of the third sprint, at the latest. They knew this was a “mini-waterfall” and asked for my help in figuring out how else they could approach this in a more Agile fashion.

CONVERSATION

We discussed whether there was any other way to get something useful to the business for review sooner than nine weeks. We looked for opportunities for smaller analyze-design-build-test-demo cycles that would move us toward the final goal. From marketing’s point of view, the campaign (which would generate Value in terms of leads and revenue) could not start until all the work was complete. However, the marketing product owner explained that a dependency like this with an unknown timeframe increases marketing’s risk of not being able to effectively “pull the trigger” on the campaign at the right time. Knowing early on whether there were going to be delays in getting to the final file had business value, although it was small value.

COMMITMENT

When we broke down the steps and discussed the importance of each one to the quality and timing of the final data set, we were able to identify several checkpoints along the way at which marketing could review and approve valuable, albeit small, results. We all agreed that the starting list of “candidate customers with cell phones” needed to be correct for the whole campaign to be effective. Therefore, the first analyze-design-build-test-demo cycle would require pulling the right set of customers and attributes from the customer subject area in the DW and getting product owner acceptance on that list prior to sending the list to the data enhancement vendor.

Working with the new vendor had involved two key checkpoints, each of which would have its own analyze-design-build-test-demo cycle and would be reviewed and accepted jointly by the product owner, data team lead, and vendor lead. These were:

1. Ensuring that the data file sent by the data team was in the correct format and available for use by the vendor
2. Ensuring that the results appended to the data file and returned from the vendor were accurate, in the right format, and available for use by the data team

The final deliverable — a select customer list with vendor-provided opt-in and additional DW customer attributes — would then be developed (i.e., analyzed-designed-built-tested) for the product owner’s final review and acceptance.

THE TEAM’S REFLECTION

PROS

The team was relieved to find an approach to working in an Agile way that didn’t require nights and weekends or delay business value for several months. They also found value in the small testing phases this approach created. It meant that the next time they had an opportunity to interact with this vendor, they would be able to reuse many of the test scripts to confirm both companies’ abilities to turn this type of work around quickly.

CONS

Since the data team didn’t have any idea of how difficult it would be to work with the new vendor, they still felt uncomfortable committing to the send-enhance-receive process in a single sprint. However, they agreed with the product owner to try this experiment and learn from the experience whether this was feasible in a single sprint or not.

THE BUSINESS’S REFLECTION

PROS

The biggest relief for the business came from having insight into the progress of the request and knowing clearly whether each step in the process was “on schedule.” Of course, marketing really values good data for their campaigns, so they were delighted to be working so closely with the data team to confirm that the right data would be used for the campaign.

CONS

This approach would take more time from the product owner than past requests. She and the team committed to being honest in the sprint retrospectives and finding the right balance of time needed from the product owner.

WHERE THERE’S A WILL, THERE’S A WAY

The Agile principle of delivering working software frequently, in the shortest feasible timeframe, can scare DW/BI teams into avoiding Agile practices if they can’t conceive of how they would deliver working software in a matter of weeks. However, teams that think creatively about how to work toward this principle have found effective approaches. In this article, I have provided case studies that show how two teams approached this challenge: one by pulling one field at a time through all the layers of the architecture, and the other by identifying the business value in the steps of the DW/BI delivery process. Both approaches help DW/BI teams and their product owners identify incremental business value instead of expecting teams to magically deliver full-blown features in a matter of weeks.

ENDNOTES

1“Principles Behind the Agile Manifesto”.

2 Collier, Ken. “A Best Practice in Agile Business Intelligence: Automated Testing Framework.” Cutter Consortium Agile Product & Project Management Advisor, 28 April 2005.

3 Ambler, Scott W. (ed.). “Agile Data Techniques.” Cutter IT Journal, Vol. 18, No. 12, 2005.

I wrote another well received article about building user stories and slicing them for Data Warehousing. Check out, Why Build Architecturally Complete BI User Stories?

---

Questions?

I’m here, ready to answer and help in any way I can. Drop me a line (contact) or add a comment to this post. Good luck on your agile journey!

---