Doing sprint planning well

In my last post I stated that the goal of Agile sprint planning is for the development team to know what pieces of working functionality – prioritized by the business – they can commit to delivering with relative certainty at the end of the two-week sprint.  I also wrote that the keys to planning are priority and effort, allowing the team to pull as many of the most important items as possible into a given sprint and giving flexibility if they find capacity changes.  Here I want to share some thoughts on the best ways to do sprint planning.  As always my ideas are informed both by the reading and thinking I do on this topic and by things that I have seen work well in my current context; and of course all ideas are ultimately my own so direct your critiques my way here or on Twitter @asbiv.

We are spending a lot of time and energy thinking about the best ways to plan in my current company, so many of the topics I’ve been writing about are issues we have been wrestling with in the past few months.  As regular readers of this blog know, I work closely with two project teams and have been doing so for the past year or so.  One of the teams is smaller and focused on adding specific features to a developing product; the other team is larger and working on multiple facets of a product with more users in the marketplace.  Both teams work in two-week sprints and take time for planning both short-term and longer-term items to be addressed.  As these two teams (and the other development teams across our company) have learned about and practiced sprint planning, three essential components have emerged that make for effective sprint planning:

  1. A prioritized backlog free from clutter
  2. Appropriate estimates of near-term items (2-4 sprints)
  3. Balancing the need for refactoring with the desire for new functionality

In our experience, doing sprint planning well on a consistent basis requires these three key elements.

For some product teams, the backlog becomes a veritable dumping ground for any idea, tweak, enhancement, feature or product change that has ever come up.  Unfortunately this makes the backlog almost useless for planning purposes, since it is cluttered with ideas of questionable merit many of which may never get worked on.  I’ve written before (for instance here and here) about the importance of grooming the backlog to make sure it captures the items that the team actually plans to work on and indicates some level of priority for the items.  The other element of a strong and useful backlog is that items are clustered around elements of usable functionality.  The teams with which I work think of these as epics and user stories which provide adequate detail on the important aspects of functionality to be developed and prioritized (you can find hundreds of pages on the web about these concepts, but why not start here if you want some basic information).

In addition to a prioritized backlog free from clutter, the second key requirement for effective sprint planning is having appropriate estimates for near-term items.  If as I stated above the keys to good sprint planning involve knowing priority and effort, then this second component provides insight into the effort anticipated in developing the next pieces of important functionality.  A motivated development team can pull what they have capacity to work on from the prioritized backlog in an informed manner when they know the effort involved in completing a given task.  Working with the product manager and product owner, the development team can plan how much to commit to for a given sprint by reviewing the estimates of the highest priority items and deciding together what clustered sets of stories to work on.  Assessing the effort involved can also feed helpfully into the prioritization process itself; a new feature might seem more important to build if the effort required is relatively small but less crucial if it would require a larger investment to develop.

While there are many different ways to create estimates for items on the backlog, planning poker is one that the teams I work with have found helpful.  This simple process (which you can read more about here if you’re interested) allows the team to collaboratively decide the relative effort of a given task while also ensuring that estimated items have appropriate levels of detail before the team commits to delivering the requested functionality.  The same end result can come from collaboration between the product owner and development lead determining the approximate effort (as in small, medium, or large) and the development team further estimating more detailed tasks to gain greater granularity and precision on the estimates.

While understanding priority and effort are fundamental to engaging in effective sprint planning, the third element I mentioned above – balancing the need for refactoring with the desire for new functionality – adds a vital ingredient especially for on-going development teams.  One of the central principles for Agile software development is that continuous attention to technical excellence and good design enhances agility, and this means that the team must give regular attention to cleaning up older code so that the product functions as effectively as possible.  Addressing this need shouldn’t be left to chance or squeezed in if there is time; instead the teams I work with make it a priority to spend some time each sprint refactoring existing code even as we also build out new features.  Of course, this refactoring should have understandable business benefit for the product manager in terms of enhancing scalability, speed, or reliability for clients or reducing the support burden for the team.  This keeps ongoing investment in the product codebase as part of the overall set of priorities for the product.

As I mentioned in my last post, there is a ton of material available on the web and elsewhere about sprint planning and estimation.  Here’s a link to a set of articles exploring these topics if you’re interested in learning more: http://bit.ly/1nyHBn4.  Planning well is crucial to ensure that the team stays efficient and focused on moving the product forward.  Too much advanced planning wastes time because priorities change, new things are learned, and unexpected issues arise; on the other hand, too little planning means the team doesn’t know what to work on or doesn’t have enough information to start working on prioritized items.  A prioritized backlog with meaningful estimates on near-term items provides valuable insight as the team does its planning, while balancing continuous improvement on existing code with development of new features allows the team to plan progress in both key areas.  When the team knows the priority of issues in the backlog and the effort required to address the near-term items, they can make plans about what pieces of working functionality they can commit to delivering with relative certainty for a given sprint.  Planning this way doesn’t solve every issue that might arise for a team of course because in truth it’s not that simple.

The goal of sprint planning

Early on in this set of blog posts about Agile planning I talked about the common image of the planning onion, in which there are different layers of planning that start with the widest strategic view and progress all the way to daily planning.  So far I have written about strategy and portfolio planning in the earliest blogs in this series, then progressed to talk about product and release planning when sharing thoughts on product roadmaps.  Now I’d like to focus still further in and discuss planning for a specific sprint or iteration.  In my current context we work with two-week sprints and so my thoughts relate to team planning for a given two-week period of time.

This is really a huge topic; I could create several posts about it and people have written whole books on sprint planning.  Maybe I will write more extensively about this topic in the future, but for now I want to share thoughts on two big areas (broken into two posts):

  1. What is the goal of sprint planning?
  2. What is a good way to do this planning?

Weaving through both posts will be the idea that the key to effective planning is understanding priority and effort – relatively how important and how difficult are the various tasks.  This allows the development team to pull as many of the most important items as possible into a given sprint while also giving them flexibility if they find capacity changes.  With all of this in mind, let’s begin by talking about the goal of sprint planning.

To understand the goal of sprint planning, it is useful first to think about the reasons for developing software using an Agile methodology in the first place.  In this context several of the principles behind the Agile manifesto standout to me, including these:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale

Working software is the primary measure of progress (for a complete list of the Agile principles see this link: http://agilemanifesto.org/principles.html)

The goal of Agile development with sprints is regularly delivering the most important increments of working software to the customer.

With this in mind, we can talk about sprint planning as having twin goals, one from the development team’s perspective and one from the viewpoint of the business.  For the coders, testers, designers, and architects working on developing the product, the goal of sprint planning is to make clear what they should work on next and roughly how much effort it will require.  For the business, the goal is to know what increment of working software they can expect at the end of the two-week sprint.  Pursuing these twin goals lets the development team know what to work on for the sprint and the business know what to anticipate from the sprint.

Taken together the goal of Agile sprint planning is for the development team to know what pieces of working functionality – prioritized by the business – they can commit to delivering with relative certainty at the end of the two-week sprint.  Regularly making and fulfilling sprint goals deepens the trust between the business and development teams as both come to understand the capacity or velocity of the team to produce on a consistent basis.  Certainly there are times when unanticipated problems arise that complicate development and compromise the team’s ability to deliver on the sprint goals – just as there are times when obstacles prove easier to overcome than expected and the team has added capacity at the end of the sprint.  However, making clear and realistic commitments through the regular process of sprint planning strengthens the relationships among the development and business team members as they together identify the most important things to focus on each sprint.  Disciplined planning promotes this focus as all the relevant stakeholders know what to expect each sprint.

In my current context I have seen greater benefits flow from sprint planning as we have invested more time in doing it well.  I’ll talk in my next post more about what ‘doing it well’ looks like for us, but for now I can confidently state that having the team take time to think through and plan its commitments for a given sprint makes the entire development process more satisfying both for the development team and for the business.  It doesn’t mean we never have problems meeting sprint goals or that we never have to abandon those goals mid-sprint when something unexpected occurs (you know that if you’ve been reading this blog for any length of time); but planning what we want to accomplish in a given two-week period and then tracking our progress against those goals energizes our team and deepens the trust that the business and sales folks have in our work.  Getting to this point hasn’t been easy of course because in truth it’s not that simple.

2-part sprint planning blogs coming soon

I’ve been on vacation recently and there have been things going on at work that have kept me very busy.  But I am nearly finished writing a two-part blog post on sprint planning that I want to get out soon.  Look for part one in a day or two once it is polished and then part two likely next week.  Thanks for your patience loyal readers.

So who decides when someone has to?

In my previous (long) post I talked about navigating potential conflicts that can emerge in Agile planning when a company has an inter-related set of products.  I suggested that product owners have a valuable role to play in navigating these collisions by facilitating communication, articulating trade-offs, recognizing leverage opportunities, and offering a wider perspective.  I also made clear that responsibility for each of these four areas must be embraced by all members of the development teams including designers, architects, coders, testers, and product managers.  I further made the (bold? naive?) statement that in an Agile environment that values open communication and collaboration across self-organizing and empowered teams everyone involved strives to navigate the inevitable collisions that occur across team plans without appealing to position, power, or expertise as trump cards to force compliance.  This all sounds nice in theory, but how do such ideals stack up against the real world of product development?

Here I must admit that my answer to this is partly shaped by my good fortune in being part of an organization that values striving for ideals.  Founded on principles and consciously pursuing multiple bottom lines, the company in which I currently serve as a product owner challenges every employee to be the best versions of ourselves and to seek the best ways to face and resolve conflicts within and across teams.  In a company where we seek to recruit, hire, and retain people based on character and values as well as skill sets we can both expect a higher level of adherence to the ideals I upheld in my last post and legitimately hold people accountable for doing so.  We certainly aren’t perfect, but we may have a higher than average willingness to seek ideal solutions in our honest pursuit of our goal to make a difference and not just a profit.

Of course, in truth it’s not simple.  We don’t always act or think in accordance with our stated ideals, and even when we do we still have different perspectives and sometimes conflicting agendas.  As each product team also has to be conscious of its own bottom line and not simply what might seem best for the company as a whole, there can be an understandable element of game theory that emerges around product decisions.  If one team builds a new feature first then that team will likely bear the higher cost (both financially and in terms of development time) and so it might be ‘cheaper’ to leverage someone else’s development than to collaborate and share the costs together.  If cross-team projects can be broken into their own separate development ‘threads’ with separate resources then the normal product teams can get more done by keeping ‘their’ developers working on ‘their’ priorities while a separate team of developers can be assigned to some cross-team projects.  And since every team worries about controlling its expenses all teams can fall into arguments about the proper ways to allocate the costs associated with projects that benefit multiple teams or the company as a whole based on how much they ‘wanted’ these projects or when they would have prioritized them on their own.  In spite of our shared base of common ideals we have yet to find an easy way to sort through these complexities and continue to search for the best approach to resolving the conflicts that arise from them.

While it doesn’t solve all of these difficulties, perhaps a metaphor might shed some light on one good way to frame the issues.  As usual, I will probably stretch this metaphor close to its breaking point so please bear with me.

In an orchestra each member and each section seeks to do its best both to play its own parts and to harmonize with the whole.  In an orchestra each person seeks to be the best she can be in full collaboration with other members and in overall pursuit of the music itself.  Each recognizes the valuable contributions that others can make and knows that the most beautiful music comes not from dominating others but from playing well together.  While at some times and in some pieces certain parts of the orchestra might gain more prominence, all together know that their unifying goal is to thrill the overall client base and keep them coming back (and paying) for more.

Agile principles and corporate strategy set the music we strive toward as we look to build the best products we can in the best ways to fulfill our purposes and achieve our multiple bottom lines.  Product owners, product managers, coders, testers, designers, architects, DBAs – all strive both to be the best they can at their roles and to collaborate effectively.  Working in concert with each other and our principles puts our potential conflicts into a broader context and motivates us collectively to pursue the best experience for our clients consistent with serving them sustainably and profitably.  Thinking along these lines certainly won’t eliminate conflicts, but it creates the right mindset for navigating them when they arise.

The context for working things out when well-meaning people pursuing the same overarching goals have different perspectives on the best way forward involves some forum for mutual conversation and decision making (think of this forum as the conductor of the orchestra if that doesn’t stretch the metaphor too far).  Some team of people meeting regularly to assess overall product strategy and help assess if each team is making wise choices can keep the entire product organization abreast of best practices.  A cohort of product managers and product owners must surely be a key part of that decision making team when clarity about the best path forward does not emerge organically from the teams themselves.  However such a group would likely also benefit from the valuable contributions of design, development, and architecture leads as well whose technical knowledge and development wisdom could offer helpful perspective in crafting product portfolio priorities.  Ideally – and in my experience this ideal is realized more often than you might expect – decisions about product direction rarely need the intervention of such a group; instead their regular discussions serve to foster a broadly collaborative environment in which solutions to potential conflicts emerge from those most directly involved and impacted.  But knowing that such a team can be appealed to when necessary provides helpful boundaries for resolving conflicts because in truth it’s not that simple.

When worlds collide

Groucho Marx once famously observed that outside of a dog a book is a man’s best friend while inside of a dog it’s too dark to read.  We could use a similar analogy to talk about Agile planning and say that good planning does not happen in a vacuum because either it’s too dark, cramped, and dusty to plan in there if you’re thinking of this vacuum http://bit.ly/1mNOBPK or else it’s incredibly hard to breathe if you’re thinking of this vacuum http://bit.ly/1jqJBRS.  And neither is a good place for planning.

Effective and realistic Agile planning does not happen in a vacuum isolated from any other influences beyond the development team itself.  This is true not just externally – where good planning leans heavily on market intelligence and strategic insight about product direction – but also internally.  Plans for the development of one product may intersect with plans for another product built on the same underlying architecture.  Plans in the mind of the product manager may conflict with ideas that the design team envisions for the coming months, and these plans may in turn conflict with areas that the development team feels need attention.  As I continue writing about Agile planning, it is important to recognize the context in which this planning occurs.  Several of the past few posts have focused on the external context as I’ve discussed the need for solid market intelligence and holistic strategic goals.  Here I want to share some thoughts about the internal context for Agile planning, and to talk specifically about what happens when different plans collide.

First let me share briefly about the organizational context in which I serve as a product owner.  We are developing several different products targeted at specific market segments with overlapping but slightly different needs.  We also have a large group of internal users of our products who rely on them to empower the advisory and consulting work we offer.  These web-based products rest upon the same database and architecture and are increasingly accessed via a web face that is coalescing around common design themes.  Some of our clients use only one of our products while others integrate several in their workflow, making a unified user experience across these products important.  And all of this software development has accelerated in the past couple of years as we aggressively expand into new market opportunities even as we collectively figure out the best ways to apply Agile methodologies to our development efforts.

While in general we work well together across our different development and consulting teams, there are definitely times when our plans collide rather than coincide.  Two teams may have different perspectives on the best way to redesign parts of our website (while the designers themselves may have yet a third view in mind).  Two product teams may have conflicting views on when certain functionality should be prioritized for development or on the shape of new features that will be used across our client base even if they are first developed by a single team.  Product managers may want to enhance existing features or add new ones while development leads and testers may stress the importance of refactoring existing code to eliminate costly technical debt to make automated test coverage more comprehensive.

In theory these conflicts could be resolved by appeals to power or expertise: the ‘most senior’ person could insist on getting her way or the ‘expert’ in a certain area could mandate that everyone follow her opinions.  But in an Agile environment that values open communication and collaboration across self-organizing and empowered teams we strive to find a better way to navigate the inevitable collisions that occur across our team plans (more on this in my next blog post).  And in this navigation product owners can play a vital role.

I have written before about the image of a product owner as the bridge connecting different people involved in the development process (you can check out some of my earliest thoughts on this here if you’d like: http://wp.me/p2BePD-r).  Hopefully without egregiously overtaxing this metaphor, the product owner serves as a multi-pronged bridge spanning the gaps potentially separating coders, testers, designers, product managers, and even distinct product teams so that all of the relevant stakeholders can engage in the set of conversations required to avoid or work through these collisions.  (And if the idea of a multi-pronged bridge goes too far, maybe view this as a web of interconnections with the product owner as the web-slinger – and who wouldn’t want to do that: http://bit.ly/1krM0Hm).  In a context with several inter-related products the team of product owners collectively plays a strategically important role ensuring that the relevant factors are given appropriate weighting in deciding how to navigate potential conflicts both within and across teams.  Strong product owners do this in at least four key ways which I will comment on briefly below recognizing that each one could be a separate blog entry on its own.

  1. Facilitating communication. A focus on communication is of course crucial in the light documentation environment of Agile development, but it becomes especially crucial when potential collisions arise.  Regular formal and informal settings for communication among the product owner team members allow them to spot potential conflicts as they emerge and sometimes avoid them through fostering the right discussions ahead of time.  Once actual conflicts within or across teams surface regarding priorities or development direction product owners can facilitate conversations involving the relevant people who can work out a solution soon after the issue becomes apparent.
  2. Articulating trade-offs. In their roles working with the team to move product development forward, product owners are often in a position both to recognize and to explain the potential trade-offs involved in pursuing one path over another.  The ability to help all parties involved in potential conflicts to grasp these trade-offs can allow the team to decide more objectively how to proceed when two ideas or teams are headed on a collision course.
  3. Recognizing opportunities for leverage or cross-product benefits. Because they see across products in the portfolio and because they have perspective on the product roadmap I talked about in my last blog post, product owners can also be uniquely well positioned to recognize opportunities for one team to leverage the efforts of another to enhance both product lines.  Teams might rearrange items on their backlog to tackle larger problems together or determine that both teams need to redesign one aspect of the product workflow to benefit both teams in their ultimate goals.  Regular conversations across the product owner team help to bring such opportunities to light, allowing the development teams to capitalize on opportunities to leverage both new development and refactoring efforts for greater mutual benefit.  This type of collective ‘backlog mining’ can of course benefit from the insights of folks beyond the product owner team as well, as the wider set of people focused on product development corporately find the best path forward for the entire product portfolio.
  4. Taking a dispassionate perspective that helps the entire team weigh the relative merits of competing plans. In addition to facilitating communication, articulating trade-offs, and recognizing leverage opportunities, strong product owners can also help the teams navigate potential collisions by offering a wider perspective.  Product managers focus on understanding and solving the persistent market problems of their target market, while developers and designers concentrate on the specific strengths of their respective disciplines.  Good product owners, with their emphasis on building the best products in the best ways, are able to bring a larger focus on the process itself.  The best Agile product owners are passionate about the way the team collaborates to develop the best product or set of products, allowing product owners to help the set of stakeholders weigh the relative merits of conflicting plans and determine the optimal path forward.

Strong product owners can offer a unique viewpoint when worlds collide.  Having (hopefully) strong relationships with one another and across the product management, architecture, design, and development teams, good product owners can help the organization navigate these potential collisions and reach strategically beneficial conclusions that strengthen the company’s entire product portfolio.  Of course, product owners are by no means the only members of a strong Agile team who can help with this navigation (that would be too simple).  In the interdependent web of architects, coders, testers, designers, product managers, and product owners, all members need to prioritize communication and a willingness to make trade-offs that benefit the entire company.  Even as product owners look across projects to recognize opportunities for leverage they must also expect to learn about these opportunities from other team members who can see potentials for conflict or convergence related to design, database, architecture, and codebase areas that product owners might not be aware of (which is a great reason to encourage regular team meetings of designers, architects, testers and others just as there ought to be regular team meetings of product owners).  While there are particular ways that product owners can facilitate this cross-team collaboration, they are by no means the only ones seeking to successfully navigate through potential conflicts.

Finding or making a way through possible collisions is a vital role in Agile product planning for growing organizations with an inter-related set of products.  The ability to serve as a bridge or facilitate a web among the many relevant stakeholders can make a crucial difference in the successful evolution of a product and the growth of the entire team involved in this development.  Fostering the kind of communication and collaboration necessary for the various self-organizing development teams to all move the product portfolio forward coherently is a challenge for product owners, but success in this endeavour is vital for both good Agile planning and strong product development.  Nurturing this strong web of relationships of course won’t allow the team to avoid conflicts altogether because in truth it’s not that simple.

Summary on my next two posts

It has been a little while since I’ve been in a good rhythm with my blogging but I am trying to get to one now.  I want to work through the remaining posts in the current arc about Agile planning.  Believe it or not I work hard to keep my blog posts concise.  I think a lot about product development and have many ideas I am eager to share, but I try to keep each individual post relatively short and focused.

My next post has grown beyond that.  As I tried to summarize my thoughts on how to handle some of the conflicts that can emerge around Agile planning I found I had too much to say to write a brief post.  I looked for ways to break it into smaller chunks but could not come up with a coherent way to separate the ideas I wanted to express.  So unfortunately my next entry is going to be a bit long, and there are a couple of lingering issues that I will put into a separate post beyond that one.

For those tempted to dismiss the next entry with a TL; DR (that is, too long; didn’t read) I give you the highlights here.  Hopefully this whets your appetite for digesting the full post later.  Thanks.

When conflicts emerge in the planning for the development of inter-related products there are four keys to navigating these conflicts successfully:

  1. Facilitating communication
  2. Articulating trade-offs
  3. Recognizing opportunities for leverage or cross-product benefits
  4. Taking a dispassionate perspective that helps the entire team weigh the relative merits of competing plans

Good product owners can help with these, but of course, in truth it’s not that simple.

Meditations

The second book I want to share brief thoughts on here (the one I referenced in my earlier post on Wolfe’s A Man in Full) is a modern translation of The Emperor’s Handbook by Marcus Aurelius.  Translated by David V. Hicks and his brother C. Scot Hicks, this modernized version of the meditations of a Roman emperor and general is something I’ve been tweeting about for months as I slowly read through it.  If you follow me on Twitter (@asbiv) or if you read my blog post about this book back in January (http://wp.me/p2BePD-2X) you know that I have found Marcus Aurelius’ thoughts on life, suffering, character, and purpose interesting even when I disagreed with him.  I have also really enjoyed the pithy and modernized language that the Hicks brothers use in their translation; it makes this book of Stoic philosophy easily approachable for contemporary readers.  And as I mentioned in the other book review I recently wrote, I was delighted to see how intertwined some of the themes in The Emperor’s Handbook were with those of Thomas Wolfe’s novel A Man in Full – though perhaps I would not have been surprised if I had remembered that both books were on the same list of recommended books found here (http://blogs.hbr.org/2012/10/11-books-every-young-leader-mu/) and that I blogged about here (http://wp.me/p2BePD-W).

There are of course some keys areas on which I differ from Aurelius in looking at life.  I don’t believe in Zeus or the other gods he references (although the fact that he uses Zeus, God, the gods, and Providence almost interchangeably may suggest that Aurelius also didn’t take these deities too literally).  I think pain, disappointment, heart-ache, and suffering are real and not simply a matter of an inaccurate perspective on the world and our experience of it that we can change at will.  Beyond this I fundamentally disagree with the inherent dualism in Marcus Aurelius’ worldview – the notion that the mind is true and eternal while the flesh and all physical existence is fleeting, unsubstantial, unimportant, and ultimately something to be denied and risen above.  My own view is that life must be understood as it is experienced: as holistic and corporeal beings whose bodies are just as much a part of us as our hearts and minds.  We cannot ‘rise above’ our bodies and the physical aspects of life because we are our bodies and we live as physical and spiritual persons in a world with real substance.

In spite of this significant philosophical disagreement with Aurelius and the wider Stoic perspective, I still found this book encouraging, interesting, and thought-provoking.  As I read this book I regularly tweeted quotes from it, mostly about the simple joys of being alive, our ability to choose virtue and wisdom even in the face of hardship, and the importance of keeping our own brief lives and pains in wider perspective.  I like the idea that no matter what we go through in life we have a fundamental ability to choose how we respond and that with those choices we can display our true character even when confronted by oppression or suffering.  I appreciated Aurelius’ repeated declarations that life is short and we are far less important than we tend to think of ourselves; this attitude reminds me to be humble and to take myself and my concerns less seriously.  And I resonated with his assertion that we should never allow other people and our desire to please them to control our lives or diminish our goals.  Plus I found many of his ancient observations both relevant and humorous in our post-modern context.

So while I cannot embrace everything from The Emperor’s Handbook I did truly enjoy the meditations on life, leadership, and character that Marcus Aurelius offered in his writing.  He didn’t make me a Stoic, but he did remind me to take myself and my problems less seriously, to remain true to my character in the face of both suffering and flattery, and to strive for a virtuous life no matter what obstacles I encounter.  Like most books I read (or movies I watch) I found elements I liked as well as those I disagreed with, but that should come as no surprise because in truth it’s not that simple.