Listening to the market – outside in

So a couple of weeks ago I started a ten-post blog series on Agile planning with the following rough outline:

  1. What’s so hard about Agile planning?
  2. The planning onion (or is it a flame?)
  3. Listening to the market – outside in
  4. Product roadmaps
  5. When worlds collide: functionality that impacts across products
  6. Sprint planning: how much is too much (or too little)?
  7. Incorporating new feedback between and within sprints
  8. Planning for today – would Kanban help?
  9. Context is crucial but planned work should be in small chunks
  10. Use the retrospective to improve future planning

So far I have written about the challenges of Agile planning (thanks to those of you who favorited my tweets about this topic or who commented on the blog itself) and about the planning onion that describes a model for planning within Agile.  Now I want to talk more about the outer layers of Agile planning.

The outer layers of the planning onion – corporate strategy and product portfolio planning – focus on the big picture of where a company wants to go and what vision it has for the way its products and services will be deployed.  In thinking about these planning layers from an Agile perspective, one of the most crucial areas of focus is on gathering solid market intelligence.  The folks at Pragmatic Marketing tout the vital importance of identifying and solving market problems that are pervasive, painful, and which clients will pay for someone to solve as central to successful corporate strategy.  This is why they call their framework for product development a “market-driven model for managing and marketing technology products” (check out their framework here: http://bit.ly/1mE6Gw9).

Pragmatic Marketing is of course not the only firm championing the importance of listening to the market when creating strategy and defining a portfolio of product and service offerings.  I came across a blog post by Hutch Carpenter that stressed the need to listen to clients and prospects when developing products rather than simply building something that the company thinks the market will buy.  Entitled The Folly of Inside-Out Product Thinking, this post decried the tendency of some companies to assume they already know what prospects want or what will sell, leading them to ignore the vital role of customer validation both before and during development.  Here’s a key quote from this piece that captures the Agile priority of gaining true market intelligence: Customer-driven development “involves the customer at two key decision points in the product development process so we are focused on building what matters to them. First, we need to really understand what they need and why it matters. This requires that we are the customer too or we are immersed in their motivations and frustrations. And second, when we think we truly know what they need, we need to share what we plan to build with them for feedback and enhancements. We can do that through discussions, sharing mockups, or building product and iterating on it based on what they say and how they act.”  And you can find the whole thing here: http://bit.ly/1dt7PBo.

The central significance of listening to the market when developing products infuses the outer layers of the planning onion that I wrote about in my previous post.  Good Agile product planning begins by listening to the market and gaining actionable intelligence about the kinds of problems faced by the market sector you want to serve, the alternative solutions available, and the kinds of products and services that will thrill customers.  This broad window into the market and the corporate strategy that flows from it keep the inner layers of Agile planning from being simply reactive to the most recent signs of customer interest.  The firm where I work has been looking for increased opportunities to interact with customers and prospects, gathering and synthesizing the market intelligence we gain into a sharper corporate strategy and a clearer vision of our product and service portfolios.  We are learning exciting things that draw us closer to our target market and that will help us continue to evolve our product development organization.

At the same time, because of our history as a consulting firm sought out for our industry expertise, my company wrestles with some aspects of this outside-in approach.  Experts in our firm (both business and technology experts) have incredibly deep and broad knowledge about our market, our current and potential customers, and the wider trends impacting the companies we serve.  Even as we seek ways to listen to our clients about what they want and need, many of these same clients pay us to educate them on what they need to know about where our industry is headed.  In fact, part of our key competitive advantage as a firm comes from our unique combination of deep market knowledge and incredible technological prowess.  There are times when we really do know better than our clients about what they are going to need – although of course we have to be careful that our insight in some areas doesn’t fool us into thinking we don’t need to listen and learn in other areas.  Balancing our twin identities as both an insightful consulting company and an emerging technology company makes wrestling with the importance of an outside-in approach crucial.  Hopefully we keep these two poles in tension as we shape the outer two layers of our overall planning.

In any case, having a well-developed market perspective roots all Agile product planning in a context of overall vision for how the company wants to serve its customers.  This outside-in approach ensures that the more tactical elements of Agile planning are informed by valuable strategic intelligence from the people who will ultimately buy and use the products being built.  Such a context doesn’t eliminate the risks inherent in software development; nor does it solve all planning problems for the Agile development team.  Because of course in truth it’s not that simple.

The Agile planning onion

I recently heard someone at my firm lodge the following complaint about Agile: ‘The trouble with developing software with Agile is that it tends to be very reactive as we build to satisfy the most recent requests; instead we need a more proactive and long-term approach to thinking about our software development priorities.’  To offer some context for this comment, this person has a much deeper familiarity with our company culture (which has often been reactive) than with either software development in general or the Agile approach in particular.  But even people who know more about creating software often have the misconception that Agile is anti-planning.  While the Agile Manifesto values responsiveness to new information more highly than holding fast to a predetermined plan, from my own experience Agile works best in the context of well-articulated plans that look at both short-term and long-term priorities.

As I plunge into this set of blog posts on Agile planning, it’s important to start with the picture that many Agile developers use to describe the different levels of planning employed in this approach.  You can find many discussions of this on the web, where it is alternately referred to as the planning onion or the planning flame.  You can check out one discussion here (http://bit.ly/1dUxGTo) and if you want a longer discussion and some interesting background music you can check out this presentation here (http://bit.ly/1m9gaiR).  The picture below comes from the following link that provides only a basic overview of each stage (http://bit.ly/1j9vm2q).

The idea of the Agile planning onion (or planning flame) is that planning actually involves multiple inter-related levels that go from the widest perspective to the most focused.  Corporate-level strategy informs the entire context for software development and provides a 3-5 year time horizon on the overall direction a company intends to follow.  Beneath that broader corporate strategy is a plan for a more targeted portfolio of products and services designed to reach a particular market or market segment; planning at this level extends 2-4 years as a company determines how it intends to reach and serve a set of customers.  These outer two layers of the planning onion set the general themes and broader context for the more detailed planning that comes below.

Within that broader vision there is a 12-24 month product plan that generally encompasses a product roadmap of key feature areas to be built or enhanced; this level of planning begins to become more specific and helps to inform plans for the product team to recruit and train the right set of people to design, develop, and deliver the product.  Product planning is also a key area of focus for the Product Manager who collaborates with other product managers in setting portfolio plans.  The Product Owner and the development team more broadly get more deeply involved at the level of release planning, where along with the Product Manager decisions are made about what features or feature sets will constitute a given release; this level of planning often encompasses one to two quarters.  At these middle two layers of the planning onion the broader corporate strategy takes more definite shape in reference to specific products and product features.

More granular planning happens at the iteration or the individual sprint level, where the work to be accomplished in a given 1-4 week time period is prioritized, estimated, and planned.  Finally, the deepest level of Agile planning focuses on a single day, where the team meets for a daily scrum (or comparable brief touch point) to reflect on the previous day’s work, the plans for the given day, and any obstacles the team faces in achieving its daily goals.  These inner-most layers of planning are where the strengths of the Agile approach most fully come to bear, as the team responds dynamically to both the direction from the outer layers and the complexities encountered in their development efforts. (Can’t resist putting in this bit about onions and layers: http://www.youtube.com/watch?v=v6CaGr1zgH8.)

Agile planning allows for flexibility in response to new information, but the amount and type of flexibility is different at each of these various planning levels.  If a company regularly pivots its 3-5 strategy then the entire firm is likely to have trouble succeeding.  The strategy level cannot be set in stone, but the fundamental direction of the company should not shift rapidly or regularly or else product development will lack a crucial foundation in corporate vision.  At the product and portfolio level, more flexibility will be visible; at this level there is greater responsiveness to market trends for both launching and retiring products.  Release planning walks a more delicate balance as certain features or feature sets may be promised for an upcoming release of the product; the need to make trade-offs on hitting deadlines versus adding functionality means release planning exhibits a greater degree of flexibility in Agile development than it often does in more traditional approaches.

The greatest levels of flexibility come in the iteration and daily planning that are the hallmark of effective Agile development; here the teams estimate and prioritize upcoming work and plans shift as new information emerges in the midst of the development process.  In this way the Agile approach to both planning and development maps well to Lean development (which is a topic I should probably write more about sometime soon).  One quote from a simple Wikipedia discussion of Lean helps to make this point: “As software development is always associated with some uncertainty, better results should be achieved with an options-based approach, delaying decisions as much as possible until they can be made based on facts and not on uncertain assumptions and predictions. The more complex a system is, the more capacity for change should be built into it, thus enabling the delay of important and crucial commitments. The iterative approach promotes this principle – the ability to adapt to changes and correct mistakes, which might be very costly if discovered after the release of the system” (see this article for more on Lean: http://bit.ly/1fWmbli).

Agile planning is certainly designed to be flexible and responsive, but in the context of the planning onion it ought also to be proactive rather than reactive.  This flexible and responsive planning can be anxiety-producing for those who want defined product development timelines and Gantt charts; but in fact the multi-layered and iterative approach is actually more successful for developing complex software since so much learning will take place in the midst of the development process.  If the only type of planning that happens for an Agile development team is the daily- and sprint-level planning, my colleague’s misguided assessment of Agile can be unfortunately accurate.  But this is not how Agile is supposed to work; I will have lots more to say about effective Agile planning in the upcoming posts in this series, but hopefully it is clear already that strong Agile teams engage in a regular discipline of planning at multiple levels.  This is how it is supposed to work anyway; of course, in truth, it’s not that simple.

What’s so hard about Agile planning?

If you follow me on Twitter (@asbiv) or if you read a post I made to my blog last week, you may have noticed that I recently started reading The Emperor’s Handbook by Marcus Aurelius, a Roman Emperor and a Stoic.  When writing about the ways that people ‘abuse their souls’ Aurelius decries those whose “actions and efforts have no apparent purpose and [who therefore]… operate at random and without consequence.”  All of life, according to Aurelius, ought to be lived with a goal in mind; without planning and conscious effort, people end up living random and meaningless lives.

Although I don’t agree with everything that Aurelius has to say about life, I do agree that planning is important to avoid wasting time and energy on scattered pursuits.  Knowing the value of consistent planning is a key part of why I am writing this set of blog posts about planning in Agile.  Unfortunately, I also know from experience that planning – especially for a team of people collaborating together – can be very challenging for a number of reasons.

  1. Planning is tough to do on a consistent basis because it requires being proactive and deliberate in the midst of busyness.  It is simply easier to be reactive or to rely on vague goals than it is to do the hard work of analyzing your alternatives, determining your priorities, and structuring your time and effort to pursue what matters most.  I like the idea of planning more than the discipline involved in doing it well, and so I often find that the busyness of my days swamps the efforts I make to establish and stick with a plan.
  2. When it comes to planning Agile software development, effective planning means understanding the market well enough to know what to start building when – and that too is difficult.  Reacting to the latest piece of feedback I received or simply going with my gut feeling is easier than gathering the data the product team needs to plan well.
  3. Creating and sticking with a plan for Agile development can also be challenging because it requires saying no (or at least not yet) to seemingly urgent requests.  As I’ve made clear in many previous blog posts I am not at all suggesting that solid Agile planning somehow means that you never say yes to new ideas that disrupt your prior plans; however, Agile planning is not the same thing as ‘pivoting’ every time you get a new piece of information from potential prospects.  A flexible product plan leaves room for appropriate responsiveness, but the hard work of planning is also crucial to avoid ‘operating at random’ by reacting to every apparently urgent request.
  4. Another reason that Agile planning can be tough is the simple fact that it takes time to plan, estimate, and prioritize and gathering the team to do this work regularly can sometimes feel like a distraction from the ‘real work’ of writing and testing code or selling the product.  Getting buy-in from the team about the fact that the work of planning is truly worth the effort can be challenging – especially if the team is used to being reactive or having only vague goals rather than specific and measurable plans.  This ties into yet another reason why Agile planning can be difficult.
  5. True Agile planning isn’t a ‘once and done’ activity like waterfall project planning; it involves daily interaction with both the development team and the market so that you can respond flexibly to new information.  Creating an initial plan is only the first step in Agile software development.  As the team works together and the product vision evolves, you will all learn new and important information that will shape ongoing planning – information about technology, about the true complexity of the product and features you are building, and about market demands for your product.  This collaborative learning means that Agile plans are always developing and so planning is a never-ending part of creating and enhancing a product.
  6. Finally, I find Agile planning can be difficult because it has to be a collaborative engagement.  Since no one person on the team has all of the information required to plan fully, Agile planning isn’t something I can simply sit in a room and do on my own or that I can squeeze into short blocks of free time I find during a week.  Multiple people – often the entire team – need to be actively involved in the planning process, which means planning has to be a deliberate activity.

For all of these reasons Agile planning is a difficult undertaking even when the whole organization understands and buys into this approach.  The firm where I work has been growing as an Agile shop for over seven years and we are still wrestling with how to improve in this area.  Even as I start this blog arc on planning our company is trying to initiate a deeper level of planning discipline that will no doubt inform the things I write about in the coming weeks and months.  No one wants to fritter away a life or career in random activity as Aurelius rightly observes, but with all of the obstacles to consistent Agile planning the fact is that in truth it’s not that simple.

An Agile plan to write about planning in Agile

As I refocus on trying to blog more regularly about life as a Product Owner in an Agile software development organization, I want to start out with a set of posts on the topic I planned to write about in the fall as I wound down my series on velocity.  I planned to start my next blog arc discussing planning in Agile development.  Over the past few months the daily demands of my actual job as Product Owner on two different development teams has meant that (as the Agile Manifesto encourages) I have been ‘responding to change’ rather than ‘following a plan’ and so I am only now launching this next arc.

Here’s my plan to write about planning: I want to share a series of ten posts about planning in Agile that start at the widest level and move into the weeds.  At this point the rough topics and titles for these posts are as follows:

  1. What’s so hard about Agile planning?
  2. The planning onion (or is it a flame?)
  3. Listening to the market – outside in
  4. Product roadmaps
  5. When worlds collide: functionality that impacts across products
  6. Sprint planning: how much is too much (or too little)?
  7. Incorporating new feedback between and within sprints
  8. Planning for today – would Kanban help?
  9. Context is crucial but planned work should be in small chunks
  10. Use the retrospective to improve future planning

However, this is an Agile plan to talk about Agile planning, and so it may evolve slightly as I write and get feedback – or it may pivot significantly if I learn something that redirects my thinking (talking about shifting plans to write about planning somehow has me thinking of this: http://www.youtube.com/watch?v=U_eZmEiyTo0).  I already know that somewhere in here (maybe as part of several posts and maybe as a separate post itself) I’ll want to talk about incorporating code refactoring and addressing technical debt into the sprint plans and the product backlog.  Given what I know now though these are the set of blogs I plan to write about planning over the next roughly 10 weeks.  I like planning and I like writing so you might think it would be easy to execute on these goals, but in truth it’s not that simple.

A life lived on purpose

So those of you who follow me on Twitter (and if you’re not following me please check out my tweets @asbiv) may have noticed that I recently started reading (and tweeting about) The Emperor’s Handbook by Marcus Aurelius, a Roman Emperor and a Stoic.  Lots of folks who think about leadership and character seem to have recently rediscovered these ancient writings and a friend of mine recommended them to me.  While I don’t share some of Aurelius’s basic premises as a Stoic, I am enjoying reading his meditations about how to live a good and meaningful life in the midst of strife (he died of exhaustion after thirteen years of warfare so he knew a fair amount about strife).  Many of his meditations are brief and so they fit well into twitter, but the other day I read a longer section that had too much meaty content for me to boil down to 140 characters.  It bears almost no relation to what I blog about here (hence the distinguishing tags on this post) but I decided I had to share it anyway.  Again, even in these words I find somethings about which I disagree with Aurelius; still there is good food for thought here.  Hope you enjoy both reading and reflecting on it.

[Book Two, Parts 16 and 17] A man’s soul abuses itself in a number of ways, first and foremost by becoming, as much as it can, a cancerous growth, a foreign body in the universe.  Complaining against the nature of things is a revolt against nature, which is made up of all the natures of its many parts.  Second, it does violence it itself when it scorns another man, or seeks to do him harm out of anger.  Third, it wrongs itself when it yields to pleasure or pain.  Fourth, when it wears a mask, and speaks or acts falsely on insincerely.  Fifth, whenever its actions and efforts have no apparent purpose and cause it to operate at random and without consequence, for even the slightest act should have some end in mind.  The end for all rational beings is to obey the reason and law of the one hallowed City and Republic.

What is man? His life a point in time, his substance a watery fluxion, his perceptions dim, his flesh food for worms, his soul a vortex, his destiny inscrutable, his fame doubtful.  In sum, the things of the flesh are a river, the things of the soul all dream and smoke; life is war and a posting abroad; posthumous fame ends in oblivion.

What then can guide us through this life?  Philosophy, only philosophy.  It preserves the inner spirit, keeping it free from blemish and abuse, master of all pleasures and pains, and prevents it from acting without purpose or with the intention to deceive, ensuring that we lack nothing, whatever others may do or not do.  It accepts the accidents of fate as flowing from the same source as we ourselves, and above all it waits for death contentedly, viewing it as nothing more than the natural dispersal of those elements composing every living thing.  If the constant transformation of one element into another is in no way dreadful, why should we fear the sudden dispersal and transformation of all our bodily elements?  This conforms with nature, and nothing natural is bad.

Again, whether or not you agree with this perspective, I hope you’ll find it worth pondering.  Because in truth, it’s not that simple.

How new ideas impact the sprint team

Over my last few posts (back in the fall) I talked about understanding and improving the velocity or productivity of a project team.  The idea of achieving a sustainable and rapid development pace can be complicated by many factors but hopefully the ideas I wrote about in the past few entries have been useful in elaborating my own perspective on the Agile principle of sustainable development.  With these posts on sustainable development velocity I am wrapping up my set of blog entries on what I called the topic of Agility and shifting my focus – for a little while anyway – to talking about Agile planning.  As something like a bridge between these two arcs, I wanted to write about a topic that comes up frequently for the teams I work with.  It is an issue that impacts both velocity and planning, and it concerns how a team responds to new ideas for features and enhancements.  In my experience the product owner plays a key role in helping the team handle these new ideas.

Ideas for what a development team might focus on for a given sprint can come from all kinds of places.  Some come from exploring and implementing new technologies that can improve performance and enhance user experience.  Some come from the sales and marketing team reflecting on what might make the product ‘easier to sell’ in the marketplace.  Some come from interactions with current clients and prospects, observing the problems they encounter in their daily work and dreaming up solutions to address pervasive and painful issues in the target market.  And some come from observing competitors and conducting win/loss analyses.  Any one of these can be a source for good ideas, and the Agile process is designed to be flexible and responsive when good ideas emerge.  But the fact is often the ideas that come to the development team – and especially those ideas that come from folks with limited connection to the whole development process – are not particularly good.  The ideas might be unformed, shaped by anecdotal evidence or opinion rather than true market insight.  If the team needs to respond to these ideas mid-sprint this can adversely impact their development velocity and impede their ability to plan effectively.

How should the product owner respond when folks want to offer new ideas for the team to work on?  Way back in March when I was writing about the top ten responsibilities of a strong product owner, I posted a piece on how the product owner must protect the team from ‘seagull’ managers who swoop in, dump stuff of questionable value on the team, and fly off again (see this link to read the original post: http://bit.ly/166n18u).  Here I want to share a similar metaphor that I first heard from Marty Cagan (who in turn attributes the image to Todd Jackson of Facebook) who contrasts ‘funnels’ with ‘umbrellas’ in an article he wrote: http://bit.ly/114KXlJ.

Some people view the role of the product owner as that of a funnel, collecting diverse ideas from multiple sources and funneling them to the development team to implement.  In this approach, all ideas are given equal weight and are slotted for development as soon as they fit – with little or no attempt to evaluate the merit or priority of each idea that flows to the team.  By contrast, in the umbrella model the product owner views her role as shielding the team from these kinds of disruptions.  Rather than allowing outside ideas to derail the team’s velocity mid-sprint and distract from the team’s planned and prioritized user stories, the effective product owner serves as an umbrella protecting the team.  That way the team can focus on getting its work done and the product owner can determine which if any ideas ought to be honed enough to bring to the team in future sprints.

For this model to work effectively, two important things must be kept in mind:

  1. Strong product management is crucial.  Unless the product manager (and the wider product development organization) shares the view that the product owner should function as an umbrella, the product owner will soon be regarded as an annoyance or even a hindrance to getting development work done.  If people across the company believe that every idea ought to be worked on as soon as possible then the product owner will be resented or marginalized as the development team gets pulled in different directions every time a new idea emerges.  For the product owner to shield the team from these disruptions, she needs the support and even the organizational cover provided by a strong product manager affirming the importance of vetting all ideas for true market impact before they get prioritized.
  2. A good Agile process promotes pivoting while avoiding the problem of churning. Serving as a protective umbrella for the team does not at all mean that the product owner blocks out all new ideas.  Part of the differentiating strength of Agile is the value it places on iterative planning and intelligent pivoting – when market research and sprint-by-sprint planning reveal that the product direction needs to shift then the team can adjust on the fly to pursue the best opportunities.  Protecting the team from churning through every new idea that someone mentions actually promotes the overall ability of the development team to pivot in the right ways.
    1. As sort of a sub-point here, one thing we find in my current company is that our quarterly ‘hack weeks’ (in which the ‘umbrella’ comes down a bit and the business and technology teams collaborate to explore new potential ideas) provide a great outlet for initial investigation of ideas and approaches that might be further developed later on.  I’ve written about the value of these innovation exercises before (check out the post here: http://wp.me/p2BePD-2q) and we regularly learn valuable things from spending time playing with new technologies and tools.

As with so many things about Agile development, serving as an effective umbrella for the sprint team requires balance, forcing the product owner to block out distractions without shutting out good – and even disruptive – ideas.  Steady development velocity is difficult to achieve and impossible to sustain if the team is continually interrupted by fresh distractions that keep folks churning through one disconnected idea after another.  Good planning fueled by market intelligence and open to new insights allows the team to find a rapid and sustainable rhythm protected from unhelpful interferences.  A strong product owner can play a helpful role as an umbrella supporting the teams’ efforts at both planning and responsiveness.  That’s a challenging balance to strike because, in truth, it’s not that simple.

Are you still out there?

It’s hard for me to believe that 2013 is over and a new year has begun.  The last half of 2013 flew by and kept me incredibly busy professionally.  We’ve been aggressively developing and marketing two products on which I have been working as the Product Owner, which has meant a fair amount of time both interacting with the market and digging deep with our developers.  This all means that my ‘day job’ has made it hard for me to find the time to blog in the past four months.

Hopefully that will change as we head into 2014.  I have a new posting that should be up later today that kind of ‘wraps up’ my previous thread on velocity (for those who can remember what I was writing about in the fall).  Now one of my New Year’s resolutions is to get back to blogging; I’ve had no shortage of ideas but finding the time to write coherently about them has been a challenge for the past few months.  My plan is to get back to sharing my ideas, experiences, and perspectives here on topics like behavior-driven development, Agile planning, Kanban, and more.  Hopefully some of my thoughts will spur responses from some of you who read this, and as always I would love to hear from you.  Comment here, talk to me on twitter (@asbiv), or let me know what else I should be reading and thinking about.  I know I still have plenty to learn about Agile development and being an effective Product Owner because, in truth, it’s not that simple.