Nov 23, 2014 - The Agile Impedance Mismatch

Comments

Agile development works and works well!

However, there’s a problem: At the highest level, the Agile team’s promise is: Let the stories trickle in and we will deliver! But, because of the fluid nature of “requirements”, there is no deliverable defined up front. In the absence of a concrete objective, the success of an Agile effort becomes difficult to access. Therefore, this approach does not sit too well with the project sponsors, and understandably so! They want to know, in no uncertain terms, what they are paying for! “Define the deliverable, then we’ll pay!”

So, while there is immense value in not doing a BDUF, build incrementally, put the results in front of the stakeholders and solicit feedback, the Agile process has to sit within a larger framework that defines milestones and releases.

In this post, I will attempt to describe how I recently addressed this mismatch and how we can estimate a project in an Agile landscape.

First, we need to describe the various entities that exist in this space:

As we can see, the green box encloses all the stuff that is near and dear to any Agilista: Features are made up of one or more stories. Several Stories are developed in a Sprint. Also, Stories follow the INVEST principle and typically span the full tech stack. Stories are further divided into Tasks that are carried out by one or more developers and do not have to span the stack.

Ok, that was the easy part. The part where traditional project management kicks in is in the beige box. Releases comprise one or more Milestones. And Milestones comprise of one or more Features.

The impedance mismatch is seen where we try to marry the Feature Set to a Release. In the constantly changing world of Agile, it is entirely possible that all features are not known at the time the Release is defined. Or, at least, it is a constantly evolving feature set. Remember, no BDUF! So how can we predict the success of a release when we don’t even know the effort that should go into a release? How do we address this mismatch? How can we make the bean counters happy and yet let development be agile?

Here is how I addressed this on a recent project.

First… a picture that speaks the proverbial 1K words:

Let’s assume a fictitious Hospital Information System. We will start with the grey column on the left that defines the broad areas that are the high-level modules that such a system can conceivably have. Then we will work our way rightward, from the strategic first 4 columns to the more tactical purple and orange Story and Sprint ones.

Budget Estimate

To define a budget for the project, it will make sense to record the effort involved in each of the broad areas of the project. This is shown in the left-most (grey) column. The numbers that we come up for this effort could be from past experience or from peer teams working on similar functionality. In other words it’s a swag or a very high level estimate. This estimate can be used to predict the project cost and time.

Because the very purpose of this step is to give an idea of the whole project, it is necessary to take a swag at all the broad areas in all at once. This is unlike the other steps that are estimated incrementally as we will see.

So this estimate will keep the sponsor happy… but not for long!

Bounded Contexts Definition

This, in my opinion is the key ingredient in Agile estimation. Bounded Contexts are a term defined by proponents of Domain Driven Design, and it is something that I’ve come to believe is a very effective way to deliver on the Agile promise. Here we have to determine what would comprise a bounded context, then engage subject matter experts to discuss the entities that would need to be in that bounded context and keep the discussion contained.

An example will consolidate the idea:

In the Hospital Information System example, we can certainly see that a Patient is very central to every part of the system. Therefore, if we were to start discussing the Patient entity and it’s relationships, without a pre-defined boundary, the discussion could very easily bleed (no pun intended :) ) into the following areas:

  1. Patient Doctor interaction - What doctors does this patient usually use?
  2. Patient Geographic Affinity - Where the patient lives (climate, food, flora, fauna, bugs)?
  3. Patient Escalation - How, problems with this patient need to be escalated?
  4. Patient Security - How to ensure that the patient can see his own data and no one else’s?

If we do not define these bounded contexts up front, there is every chance that the discussion will lose focus and every entity that touches the Patient entity will be brought into the discussion. So we have to pick and choose ONE of the above as a potential bounded context so that we can only design that aspect. When, later, we have some time, more importantly, the subject matter experts have the time, we can discuss other bounded contexts. In true agile fashion, we build our solution bounded-context by bounded-context.

Therefore we see (in the figure above) that we have defined five bounded contexts only for Patient Management and Registration and not for Vitals Monitoring, Bed Registration etc. Those will be added when development is already underway on the first bounded context.

Of the five bounded contexts identified, let’s pick the first: Patient Security. In the discussions that surround this context the following should emerge:

  1. The entities that are pertinent for this context: PATIENT, ROLE, PERMISSION.
  2. The relationship between these entities: PATIENT has zero or more ROLEs and each ROLE has may have one or more PERMISSIONs.
  3. Basic UI interactions in terms of screen mockups.

Feature Definition and Estimation by Tech Leads

This is the next step that uses the output of the context meetings (outlined above for the Patient Security context). Now that we know the entities, their relationships and some broad UI screens, we can start defining features that can be delivered for this bounded context.

In the figure we see that for the Patient Preferences and Security Contexts, we can define features like Self-Registration, Change Account Settings etc..

During Feature definition we should keep in mind that features are delivered with Releases. So the feature name should define something that is complete in itself. “Patient Management” is probably too broad a name because it will likely be ongoing even after a Release is cut. But something like “Self-Registration” is something that probably wont be worked on across releases. Once a user can self-register, there’s not much else you can do under that heading.

However…

The question always comes… what if you want to improve the self-registration process? What if you wanted to add a “Login with Facebook” as an option to self-registration in a subsequent release. Then would we want to resurrect the “Self-Registration” feature. Probably not. This is because features are defined within a milestone/release and show up in the release notes. So most likely the name you will need to give to the “Login with Facebook” feature is “Oauth Support” (or “Third Party Login” if you are not too technical) and add “Login with Facebook” as a story.

Features that are defined at this stage are estimated based on experience of a senior developer/architect who is familiar with the tech stack. This estimate is expected to be more accurate than the swag that was taken in the budgeting swim lane (grey). Therefore it can potentially be less or more than the budget estimate.

Release Planning at Feature Level

Now that we have feature level estimates, and we know resources available to us, it is possible to define a release.

In the picture we see two releases defined, one named Mar 31 and another named Dec 31. The Mar 31 release comprises 3 features carried over from the blue column. (In a real life situation, one would definitely expect to see many more features per release).

This is the stage you would probably involve Marketing and ensure that the features being released are in line with the product road map and other marketing collateral.

This step sets priorities and allows us to determine what stories to start writing first. On the flip side, the features that make it in the next release will decide what bounded contexts should be defined. So this step serves as a bridge over to the tactical side of the fence.

Story Grooming and Developer Estimation

This step looks at the features that are required to be released (from the green swim lane). Those features are then broken down into stories following the INVEST principle.

These stories are placed in the prioritized backlog and moved up or down to facilitate incremental development. As the current sprint winds down, developers should start grooming these stories for the subsequent sprint.

The grooming process should be carried over a few days (at least) so that developers are given a chance to collaborate with the story author. Once groomed, the developers are expected to estimate the stories in either story points or time effort.

Groomed stories are prioritized in the prioritized backlog.

Sprint Planning

Once the stories are prioritized and estimated, depending on the velocity of the previous sprints, and the capacity available, an appropriate number of stories are moved to the subsequent sprint.

Progress Tracking

We have seen how we can estimate work at three levels:

  1. Estimates by peer group when the budget is defined. This is a high level estimate for purposes of budgeting
  2. Estimates by the tech lead/architect when features are defined. This is for the purpose of release planning.
  3. Estimates by developers when stories are defined. This is for the purpose of sprint planning.

These estimates need to be recorded in a project management tool (like Microsoft Project).

Looking at the picture above, we can see that the developer estimate is the most accurate; then comes the tech lead estimate and finally the budget estimate that was made in the beginning of the project.

The developer estimates are not too far in the future; they are estimated when the stories are groomed and therefore not available for bounded contexts and features that are not yet defined.

The next best estimates are those that are given by the tech lead when the bounded contexts are broken into features. This occurrence precedes story grooming and therefore is available over a longer period of time.

The estimates to fall back on in the absence of other estimates are always the budget estimates.

In the picture, we see that the arrows keep moving downwards and therefore over time, as bounded contexts are defined and stories are groomed, estimates will keep improving.

The only thing that is left to do is to pull these estimates into Microsoft Project. In the event you are using JIRA Agile, there is this plugin that will do this for you. All that is left to do is to define milestones and releases and group features against them. Remember, all you have to pull into Microsoft Project are the Features, not the stories.

In Microsoft Project, you can create a baseline at a point where you are comfortable with the position of the black arrow. From that point onward, you can track against that baseline.

What is not mentioned

Building an application bounded-context by bounded-context does cause churn, of that, there is no denying. One key enabler for such incremental development would be the existence a robust testing policy and of course, Continuous Integration. But I’ll save that for another post.

Conclusion

We first identified the artifacts that can be tracked using the Agile process. Then we defined a process to plan a release and a sprint. Finally we defined a process to estimate the effort over time and measure that progress against a base line.

Now the agilistas are happy and so are the bean counters! Stake holders can see incremental results, developers don’t have to deal with stale requirements and peace and joy reigns all around!

Sep 24, 2014 - Spring One - Dallas

Comments

I’ve just returned from a 3 day trip to Spring One @ Dallas where I met and heard from several of the smart people who fuel the Spring Framework.

I could only attend 10% of the break out sessions because of parallel tracks. But there was a buzz in the air and in the NFJS sessions and if I were to draw a tag cloud of those conversations, here is what it would look like:

Yeah.. SpringBoot and Spring XD ruled! The keynote was on these two projects too!

But in the end and after attending a couple of sessions, I concluded that Spring Boot is an CoC project that does the following:

Helps you to bootstrap your project by asking the user what he wants to build (webapp, JMS, JPA etc) Deploys the app in what ever (default) container depending on the type of app (Tomcat, Rabbit, Jetty etc) To me this seems to be a souped up combination of Maven archetypes and Spring Roo.

But what about Grails, you may ask. Is that not also a CoC framework that does just that: scaffold out your app based on a series of questions?

It does, but the difference is that the Grails user is attempting to build a webapp, which falls under the definition of application development (appdev). Spring Boot, OTOH, results in faster startups and deployments! And while it’s true that appdevs do indulge in startups and deployments, I’ll be surprised if they do more than a couple of startups and deployment a year! The folks who do startups and deploys all the time are the folks who are developing Proofs of Concepts (PoC), much like the speakers at this conference! No wonder, all speakers were swearing by the virtues of Spring Boot!

And so, instead of serving the appdev community, the good folks at Spring have based the keynote on a product that serves the needs of speakers at the conference! And consultants!

And so I feel that by focussing on Spring Boot, the Spring community has become a self-serving community!

The other big ticket item was Spring XD:

I am sure that a lot of work has gone into Spring XD development and I am, by no means trivializing it. It is a great effort, but it is more of a reference implementation of an app well done! I cannot imagine that any corporation will be willing to build out an application that uses all the technologies that Spring XD brings together, and not look under the covers! And if you did take the trouble to look under the covers, then you may as well configure the absolutely excellent projects which hold up Spring XD, like Spring Integration, Spring Batch, Spring Data, Spring AMQP, Spring Security etc. And pick and choose what parts of Spring XD it would like to incorporate in your own application.

Moving on, the next big thing was MicroServices. Now, like a lot of old concepts which Spring has brought to the masses (like Coding to Interfaces, Inversion of Control, Aspects), the time for MicroServices is here.

With the advent of Reactive Programming paradigms, it makes total sense to slice the pie vertically. Where traditionally, we were layering horizontally around cross cutting concerns like presentation logic, data access logic, persistence etc, now we need to build micro services around bounded contexts. The way these microservices communicate and authenticate with each other is what a lot of talks covered.

And the last topic that was given a lot of air-time was Groovy.

Spring has been pushing Groovy for about a decade now and it serves well to provide support (via DSLs) in many areas; the two main ones being build systems (gradle) and Web development (Grails). But now with the advent of Java 8 and Lambda expression support, the value proposition of Groovy seems to have lost a a little bit of steam.

But… the main advantage, IMO in Groovy and other loosely typed languages (Scala, Ruby, Javascript etc) is their adhoc nature and their ability to quickly build DSLs. Java with all its fanfare and ceremony is for the old guys like me!

But…. Like I said, these are first impressions based on only the 10% I attended and the buzz I caught. I haven’t watched the videos yet but when I do, I may change my mind.

Sep 21, 2014 - In Memorium

Like lost friends, I’d like to enumerate (from memory) the fondest of my posts over the past 10 years! This is because of my devastating loss!

  1. Angular EE - Angular from a Different Angle. This was, by far my most visited and commented on post! I doubt I will ever be able to reproduce it but the main focus of this post was trying to apply server side robustness on the more loosey-goosey client side development. The corresponding work that prompted this post can be seen here and here.
  2. Black box testing in Legacy applications: This post talked about how to use the maven-nddbunit plugin to test legacy application by testing the result of datasets generated via the plugin and tested using Spring’s Testing Framework and Transactional support. This is being used by several developers
  3. White-Box testing using Mocks: This post talked about how white box testing and mocking results in modular code. It gave examples of using PowerMock, Mockito and drwew out the distinction between spies and mocks.
  4. Object Oriented Programming vs Functional Programming : How Object Oriented Programing is used where the domain is ever changing (like business objects) and how Functional Programming is meant for a low number of domain objects to model (like scientific apps or databases).
  5. Rails vs Grails : This was an extremely popular post that described the differences in approach for Rails and Grails developers. My and my commenters conclusion was that Rails is web-centric whereas Grails is domain centric. Since domains are typically persisted on the server, by extension, Grails becomes server centric!
  6. GWT based posts that talk about domain models being used in the client (javascript) but they are > 4 years old. …