Lynn Winterboer

Always learning; always growing

Kanban for Newly Agile Data Warehouse and Business Intelligence Teams 

by

Data teams often start with Scrum. But, Kanban is a more effective framework.

By Lynn Winterboer, Agile Analytics Educator and Coach

Reprinted with the permission of 1105 Media, Inc.

Data teams brand-new to agile typically start by implementing the most popular agile framework, Scrum, hoping to achieve the success they’ve seen from their software brethren. The data warehouse and business intelligence (DW/BI) teams that are most successful with Scrum already have a robust infrastructure. They also have the technical discipline that enables push-button builds, continuous integration, automated testing, and production-quality testing.

However, I have found that most DW/BI teams do not have this type of infrastructure and technical discipline built out. Teams without these essential practices spend a majority of their Scrum sprint focused on manually managing builds, migrations, and testing, leaving little time for developing new, valuable DW/BI functionality.

I recommend that DW/BI teams new to agile evaluate several agile frameworks prior to choosing one. Certainly Scrum is a good candidate for some. Extreme Programming (XP) is another good candidate. This article describes another option, Kanban, which is very effective for teams disciplined enough to embrace it.

Kanban will help your team measure and incrementally improve your delivery approach. In my experience, the main benefits to a data team of using Kanban are the heavy use of metrics to drive improvement decisions (right up our alley!), its focus on reducing waste, and increased predictability.

Metrics

The foundation of Kanban is visualizing the work as it flows through the steps in the delivery process, measuring the time spent in each step along the way. Kanban leverages lean value-stream-mapping practices to measure the process cycle efficiency (PCE) of the DW/BI team’s delivery cycle. PCE is the time spent on value-added work as a percentage of the total time the request takes to become value for the organization. (See flowchart below.)

Reducing Waste

Measuring the PCE of each iteration of value provides the DW/BI team members with the metrics they need to reduce waste and get value into their stakeholder’s hands sooner. Metrics help the teams understand where their processes have bottlenecks (waiting) and inefficiencies (long duration for shorter amount of actual value creation). The goal is to see the amount of waste get smaller over time, thereby increasing PCE and delivering value faster. This is the kind of thing we want to see our organizations do using the information we provide them. It’s our turn to apply what we do well to our own delivery processes!

Two key ways DW/BI teams can reduce wasted time are by limiting work in process (WIP) and by making process improvements. Limiting WIP significantly reduces waiting time between steps in the process for a single iteration of value. By limiting WIP for a team, the organization makes a business decision to prioritize the delivery of value to the organization—over “100 percent utilization” of team member time. You can see in the flowchart above that this team is clearly not limiting WIP, because there is so much waiting time between steps and the duration of each step is much longer than the value creation time. For example, the reason the request sits in the “Development” step for 15 days, even though it only took 3 days of actual development time, is likely due to this team being spread across too many projects or tasks at one time. For a more detailed description of limiting WIP for DW/BI teams, see my article “Project Management: Do Less by Committing to More.”

In the meantime, there will be days when team members do not have useful project tasks to work on. This is the perfect time for them to make process improvements to their work, so the next time they will create value faster by spending less time waiting, manually handling automatable tasks, and expending unnecessary effort. For example, team members could focus on:

  • Researching and implementing a test automation framework
  • Building automatable regression tests for existing functionality
  • Learning about hyper-modeling approaches that allow for faster and more flexible changes as requirements evolve
  • Streamlining project documentation requirements
  • Automating build and deployment procedures for development, test, and production environments

Increasing Predictability

Measuring the team’s short increments allows the team to understand where waste is occurring and determine how long it takes, on average, to deliver business requests. The team can use these metrics to establish a “reality-based” expectation as to how long requests take from initial request to final production.

For example, a team could plot their complete cycle times (request through delivery of value) for all requests over a period of time on a scatter chart. From this, they might determine that 98 percent of requests are completed in 28 days or less. That metric implies that the organization this team supports can expect their requests to be met within 28 days for 98 percent of their requests.

Summary: The Choice Is Yours

Now that you have a sense for some of the key benefits of using Kanban in your agile approach to DW/BI, your team can decide for itself whether the metrics, improvements, and predictability inherent in Kanban are a good fit for your organization.

Lynn Winterboer teaches and coaches data warehouse and business intelligence (DW/BI) teams to effectively apply agile principles and practices to their work. For 20 years, Lynn has served in numerous roles on analytics and DW/BI teams. She understands the unique set of challenges faced by DW/BI teams who want to benefit from incremental agile project delivery. Clients include Capital One, Wal-Mart, Intel, Bankrate Insurance, CoBank, Sports Authority, Polycom, and McAfee.