Q&A

Using Kanban with Aspects of Scrum for Agile Data Warehousing

May 16, 2017 by Lynn Winterboer

Is Your Data Warehouse New to Agile?

If you’re beginning to use Agile in your Data Warehouse, you might wonder which approach to try: Scrum or Kanban.

This was the case with someone who attended one of my Agile Analytics seminars in Vienna. He sent me an email, asking for some guidance. Here’s his request:

Dear Mrs. Winterboer,

I had the pleasure to attend one of your Agile Data Warehousing trainings in Vienna a few years ago, which I still fondly remember. I thought a lot of you in the last days since we (a team of three) are trying to seed Agile Data Warehousing (and Agile ETL) in our department. We might try a Kanban approach this time and see how it works out. As a first implementation of real time data warehousing is bound to be implemented in the near future, this might be a good opportunity. May I contact you for one question or another if need be? We will probably add some Scrum methodology to our Kanban (like daily standups) to help us with our self-discipline.”

Of course, I let him know I am happy to help from afar, and I sent him my perspective on Scrum and Kanban for a team new to Agile. Now, I know how it is with us coaches. We often reply to questions like this, “Well, it depends.” But, not this time! Here’s how I counseled him:

When it’s up to me to recommend a good approach to start with, I usually suggest a team use Kanban to manage workflow and limit work in process (WIP), and add the roles, cadence and ceremonies of Scrum (but not the time boxes – which are a different way of limiting WIP). By the way, I’ve heard people call this ScrumBan – but I’ve also heard the term “ScrumBan” used to mean something completely different (and harmful!), so I’m careful what I call it. I don’t call this ScrumBan. It’s simply using Kanban with aspects of Scrum.

My good friend, Ken Collier, describes the difference between limiting WIP in Scrum and limiting WIP in Kanban with this nice illustration:

As data professionals, the measuring and metrics inherent in a good Kanban approach should be attractive to us. That’s really where the power of Kanban lies – in measuring how you are doing, using the data to clarify the bottlenecks and issue spots, focusing improvements there, and continuing to get better and better at delivering high-quality, frequent results! A great side benefit of embracing the data side of Kanban is that the team can calculate the average time it takes to complete any request that comes to them – and therefore provide a data-based estimate for planning and do away entirely with “estimating.”

All, of course, honoring and holding the agile values and principles along the way.

Should the BI Scrum Master Report to the Team, or the PMO?

November 19, 2014 by Lynn Winterboer

A BI developer moving into a Scrum Master role wonders where he should report?

Earlier this year, a friend of mine, someone I met years ago at an agile conference, emailed me some questions about being a BI developer moving to a BI Scrum Master role. My friend asked,

In most enterprise organizations where do the Scrum Masters report? Project Management Office (PMO) or within the same management structure as the developers?

He continued to add some context:

As you know I come from the development side of the house and my group is looking at making my position a full time Scrum Master position. They think that I would fit more in the PMO and I am thinking that I should stay in the team. What do you think?

I replied:

If the organization is new to agile and has a PMO, then you might do better working from there. If the organization has been practicing agile for awhile, then follow their protocol. That said, there are pros and cons of each.

Scrum Master (SM) in the Project Management Office (PMO):

– Pro: If the organization is new to agile and you are one of the first SMs in the organization, then you may be able to do a better job at removing impediments organization-wide if you’re in the PMO. (see the “con” for SM in the Team, below).

– Pro: If the organization is NOT new to agile and has an established practice of having the SMs report to the PMO, then it could be good to be a part of that “community of practice” and work with other SMs to remove organizational impediments for all teams.

– Con: If the organization is new to agile and the PMO is not convinced, you might get more wrapped up in convincing them and dealing with the overhead of that organization… meanwhile missing the opportunity to be effective for the team (your number one priority).

– Con: You’ll could find yourself further from the heart of the team and over time could lose touch with what’s really going on with them. Where will you sit – with the team or with the PMO? If you reported to the PMO but sat with the team it might not be so bad…

Scrum Master (SM) in the Team:

– Pro: You’ll continue to be a member of that team and be driven by the same management guidance and metrics as the team, which keeps you aligned well with them.

– Con: If you’re not in the PMO and the PMO itself provides impediments to the team (e.g. requesting Gantt charts and milestone commitments, etc.), then it’s harder to remove them from outside the PMO. Being in the PMO might give you more influence over that organization as a whole.

WHERE DID MY FRIEND END UP?

I recently followed up with him when I didn’t run into him at Agile 2014 in Orlando, and here is his update:

Within the months following our initial conversations several things have changed within my situation.

I have been in a SM role for some time and have been working within my team as such. The PMO has started a great boot camp program and has that moving forward within the company. I advised the boot camp trainers that for this program to be adopted and maintained, they’re going to need coaches to follow up with groups after the initial boot camp. This is a recognized issue but there is some confusion on what a coach is and how that position works. There seems to be a perception within my group that I should be able to coach in my spare time without direction and/or further training. I have inquired about going to Coaching training or the Coaching Retreat for ideas, direction and support from the Agile community and the response is, “Show me what you can do – first.”

I think we are heading the right direction, but it’s going to take some give and take between the development side of the house and the PMO. The PMO and their management structure recognize the need for coaching, but they are putting all of their eggs in one basket with their Boot Camp trainer. He is currently doing the training and coaching and telling them all the while that he is overloaded. I am sure that this is not at all uncommon.

I appreciate that you have gotten back with me again. I always enjoy our conversations. I am still pushing toward my goal of CSC, and with the help of my friends in the Agile community that is going to happen.

Agile Boot Camp Is Not Enough

My friend is right, it is not all that uncommon for an organization to believe that “boot camp” training is sufficient to change long-established mindsets within a few days. Ongoing, consistent support is so important that we actually use the term “Coach.” Imagine the Denver Broncos getting 2 days of training then expecting to play an entire season without further coaching!

Question for Our Agile BI/DW Community:

Who else has faced a similar situation, what did you do about it, and how did that work out?

BI Definition of Ready

September 17, 2014 by Lynn Winterboer

What to include in the BI Definition of Ready

In an online forum, someone recently asked:

What’s in your Definition of Ready?
When does the ‘ready’ criteria apply? — i.e. ready by the end of Sprint Planning (How) or ready for the beginning of Sprint Planning?

Here are the thoughts I shared with that group.

[Read more…]

Acceptance Criteria in Agile User Stories

May 23, 2014 by Lynn Winterboer

How to Create Valuable User Stories

Get Your Testers Involved in Acceptance Criteria

FOUR FREQUENTLY ASKED QUESTIONS

Here’s another popular question and my answer.

Question #3: “What’s an example of good acceptance criteria — outside of that the report should match the mockup/data should be accurate.”

LW: Acceptance criteria is a big topic. A simple example can be seen in figure 2, in the most profitable transactions.

Figure 2 — Watch Out for Epics

We’ve simplified it down to where we need to see profit per customer per transaction, so that we can identify the characteristics of the most profitable transactions. One example is when XZY manufacturing places order number 13579 for 3 widgets for $100 each. This is our acceptance criteria. They’ve placed an order for 3 of our products at $100 each, and the widgets are on sale for 10% off, and the cost of goods sold for each widget is $50, then the profit for this order should be $120.

If you do the math, that works out that $300 less 10% ($30) less $150 = $120. This is what’s called behavior-driven development. This is a format. When something happens, and it has X criteria, and it also has Y criteria, then Z happens. It’s a great method in the data world for the stakeholders and product owner to clarify what they mean and what they’re expecting to see. In this case, you might have several different scenarios — and this could come out of a use case scenario — so you might have a handful of these examples.

And I like it when the person requesting this, the stakeholder, comes to me and says, “Okay, here are five orders that I’ve picked out of our system and they each meet a different criteria. This one has 10% off. This one has $5.00 off each. This one doesn’t have any discounts.”

And you would even want one that says, “If the cost of the product, less the discount, less the cost of goods sold, is negative, we want to know that it was a negative profit.”

So it’s good to come up with a handful of examples that flesh out what they want to see. It gives you test data to unit test on, and you can automate that. So when you have an actual order, where they say, “Run this through and here’s what I expect the profit number to be,” then you can automate that very easily. You can say, “Here’s the input of this order with these elements and here’s the expected profit that’s going to calculate in the end.”

That’s a beautiful way to go.

However, it is very time consuming. And it can feel completely overwhelming to some teams who aren’t set up to do that. It’s very useful to have the testers involved, or someone with testing skills involved because he or she can say, “I see your story, here. Let’s pull a few orders out that meet these scenarios.” The testers can get going on writing the tests and getting the test data set up and automating those tests right away before the developer even does any development.

That’s a great way to do it.

COMING UP ON LYNN’S BLOG:

LYNN ANSWERS THIS POPULAR QUESTION:

Q4: What is your recommended sprint length for a data warehouse team given the fact that it takes longer to deliver a Data warehouse solution?

Share in the Comments:

Do you have a tester you can include on your team from the beginning? How are your stories? Do you wonder if you have an epic? Post your story here and let’s discuss as a community.

Sprint Lengths

May 19, 2014 by Lynn Winterboer

How Long a Sprint Should Be

Data Warehouse and Business Intelligence Teams Benefit from Shorter Sprints

FOUR FREQUENTLY ASKED QUESTIONS

I recently had the honor of presenting a series of webinars on “Writing User Stories and Slicing Epics for DW/BI Teams” for Cutter Consortium, where I am a senior consultant. After each webinar, I addressed questions from the audience. I’ve been using my blog to report the questions and the answers. I answered the first question, QUESTION #1: HOW CAN A USER STORY SPAN ALL LAYERS — IS THAT NOT AN EPIC? Now, here’s another frequently asked question and my answer.

Q4: “What is your recommended sprint length for a data warehouse team given the fact that it takes longer to deliver a data warehouse solution?

LW: I personally think that teams should start out with shorter sprints. If you’re doing an agile transformation and you’re moving towards an agile approach, if you start out with a one-week iteration with really thin-sliced stories, you’ll learn the agile process and you’ll learn what your team can and cannot handle much faster.
A key aspect of an agile approach are the learning cycles. In a traditional data warehouse, you might have two releases a year, and you don’t show anything to the customer until they are in UAT (user acceptance testing). That’s two learning cycles a year. We do these things called postmortems to reflect back on the project and for us to get feedback. But two learning cycles is not that many in a year.

So if you’re a new team starting out with agile, and you go to one-week iterations, you have a learning cycle at the end of every iteration, at the end of every week. It’s going to slow you down, but you’re learning faster. And then you can expand to two- or three-week iterations. The typical sprint length, or iteration length, is two weeks. I have seen DW teams who try to start off with nine-week iterations, but that is not enough learning cycles and you always bite off more than you can chew in your first few iterations. We’re used to doing really big work, not little teeny pieces of work. If you start out with a nine-week iteration, and you work as fast and hard as you can for nine weeks, and you’re doing it in this new mindset with these new approaches, and you’ve got all this overhead of learning with all this process change, and you get to the end of the nine weeks and you don’t hit your goal, it’s really disheartening. And you can lose your business’ attention. You can lose their support.

I suggest when teams start off, they go with shorter iterations, and once they feel they’ve got the agile process down, and they’re working together well, and they’ve got a good mindset shift, then they can expand to two- or three-weeks. Anything bigger than that and you’re shortchanging yourself on your learning cycles.

Agile Documentation

April 23, 2014 by Lynn Winterboer

How To Manage Regulated Documentation in Agile Projects

FOUR FREQUENTLY ASKED QUESTIONS

I recently had the honor of presenting a series of webinars on “Writing User Stories and Slicing Epics for DW/BI Teams” for Cutter Consortium, where I am a senior consultant. After each webinar, I addressed questions from the audience. Last week, I answered the first question, QUESTION #1: HOW CAN A USER STORY SPAN ALL LAYERS — IS THAT NOT AN EPIC?

Here’s another popular question and my answer.

QUESTION #2: In regulated industries, such as medical devices, how are the regulatory requirements taken into account? Releases are difficult to coordinate when there has to be a definitive implementation to draw a line in the sand from a documentation standpoint.

LW: That’s a really good question. And a common one. In regulated industries, such as medical healthcare and financial industries, there are a lot of questions that come up. I believe you really need to try and include that information in your definition of done. If you can fill out your documentation a little bit at a time, it makes it not such a big task to fill it out in the end. Including your regulatory requirements in your definition of done is one way to manage it.

Another way that I’ve seen teams do it is to have what they call a hardening sprint, which is useful for Data Warehousing teams because DW teams might do pre-production testing, or production-level testing, or they might do load and stress testing during that time — things that may be difficult to do every couple of weeks given your data volumes and maybe your technical infrastructure can’t support testing on huge data volumes every couple of weeks, but you do have a test environment where you can load lots and lots of data and really pound it. A hardening sprint would be right before your release and during that hardening sprint is when I have seen financial institutions, in particular, take that time to really meet all of their regulatory requirements.

You have to step back and think creatively and look at values and principles. You have to say that your goal is to deliver value to the business and that value is defined by the business, not by the IT team, and you want to deliver it frequently and consistently. So if you do a release every 3 months, and you need to take a hardening sprint to work on meeting regulatory requirements, typically your business doesn’t mind because they understand. They’re part of that business and they respect that the regulatory requirements are there for a reason and you can’t get out of them. So you have to do them.

COMING UP ON LYNN’S BLOG:

LYNN ANSWERS THESE POPULAR QUESTIONS:

Q3: What’s an example of good acceptance criteria — outside of that the report should match the mockup/data should be accurate?
Q4: What is your recommended sprint length for a data warehouse team given the fact that it takes longer to deliver a Data warehouse solution?

Questions or comments about Done? About managing releases in agile? Chime in!