We all know the manifesto but why do so few know the principles?

Agile manifesto:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan[6]

Principles behind the Agile Manifesto

We follow these principles:

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
  • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  • Business people and developers must work together daily throughout the project.
  • Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  • The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  • Working software is the primary measure of progress.
  • Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  • Continuous attention to technical excellence and good design enhances agility.
  • Simplicity—the art of maximizing the amount of work not done—is essential.
  • The best architectures, requirements, and designs emerge from self-organizing teams.
  • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly. [6]

Oxford Assignment

I recently completed an Agile module for my Oxford masters and though I’d share the essay with you all.

Please compare and contrast the different adoption paths taken by the BBC on the iPlayer project and Sapient on their Project with the Office of Naval research. 

There are a lot of myths that prevent companies adopting Agile, including claims of lack of planning, quality, documentation, predictability and control.  In reality Agile has a number of techniques to diminish the issues surrounding traditional methodologies and one study showed Agile initiatives actually fail in “two-thirds of cases due to a failure to integrate the right people or to teach a team-based culture.”[9] The other reasons are also people related such as executive sponsorship and training.

Clearly people are important and the common denominator between Agile methodologies, the Agile Manifesto, states “individuals and interactions over processes and tools”. This means that process must not be forgotten but must take second place to people interactions. “A common misconception is that Agile is about a new process – no; the difference are in the values and philosophy” [4]. Jeff Patton stated “Agile development is a culture, not a process.”[13] “We would rather use an undocumented process with good interactions than a documented process with hostile ones”. [4] It should be noted, however, that Agile is not a “Silver Bullet” and is often oversold. A comparison of Agile methodologies is provided in Appendix C

Two differing cases studies, and further interviews with the authors, were used to see if a “one size fits all” approach could ever be agreed upon.

Company

The companies could not have been more different.

The BBC study describes nine teams divided by functional areas reporting into a, non Agile, central team to deliver the iPlayer application. In my interview, Mike Lowery stated that the BBC, a publicly funded company, had a non-compete rule. If another company were to develop a similar or competing project the BBC would have to stop. They also didn’t have a financial budget. People were paid to be there for the year so it was less critical what they did, in fact the BBC sought to create jobs [1]

“Sapient is a global services company that helps clients transform in the areas of business, marketing, and technology.”[7] In my interview, Andy Takat confirmed that Sapient were ideally placed, and heavily incentivised, to identify the optimal agile adoption strategy’s due to their massive and diverse client base, projects, technologies, budgets and systems. [2]

Project 

Agile is often used to deliver early to customers but the BBC built 3 different versions of the iPlayer. Mike states the first version was very amateur. The Second used an external design agency with big design up front but suffered poor user feedback.  Only the third version was a success when Agile techniques were used. [1] Clearly they were not “failing” fast.

Sapient’s customer was the US Navy’s Office of Naval Research project “to develop an entirely new set of naval logistics command and control capabilities that can be rapidly transitioned to the operating forces”. [8]. Given that this was post September the 11th this was a life and death project. There was no option to fail or not deliver which is often incorrectly cited as a reason not to use Agile.

Like the companies, the projects were very different. The BBC were free to use a “lightweight” process as opposed to the more prescriptive patterns used by Sapient’s critical project.

Training

On joining the BBC, Mike was given the “black book” and was then sent on the scrum master course with ken Schwaber. Mike Cohn performed an estimation course for them. Rachel Davies was his mentor and co-author Liz Sedley was a project manager, which made a “massive difference”. As a scrummaster Mike then coached his team. [1]

Sapient have a single consistent induction for all staff but different business areas customise their detailed ramp up approaches based on the work they undertake which varies tremendously. Sapient also have an open online community based on Jive software that allows everyone to contribute new ideas and patterns. [2]

Given Agile is a mindset, the BBC approach should be preferred but given the global consistency, maturity and criticality of Sapient’s approach this better suited them.

Management

The BBC management did not listen to the scrum masters and hired a building site project manager who soon realised he was out of his depth and left. Mike’s team would often be criticised for not being able to plan, due to the lack of a Gantt chart, but his team delivered on time whilst the core team were regularly late despite using a Gantt chart. This indicates an immaturity which allowed Mike to be experimental. [1]

Sapient have a very mature management team, heavily focussing on customer involvement and are keen to be seen as facilitators and not the owners. Sapient state that a “Dictatorial approach is almost always doomed” “The customer whose requirements are snubbed will no longer support the initiative”. [8]

Sapient focused on “Customer collaboration over contract negotiation” [6] with customer involvement, strong stakeholder buy in and timely decision making. They are however very prescriptive which is not always beneficial.

Customer and Requirement gathering

A main theme with both companies was their challenge with customer representation. In the BBC it was scaling the Product Owner, with Sapient it was accessing a customer who would change and be sent to hostile situations. The customer is a critical part of any Agile team to ensure software solves actual business problems and prioritises work correctly. Used correctly the customer can make decisions which reduce effort for example by removing functionality which will never be used from the backlog. There are obstacles, however, such as managing their existing full time job and only lending junior staff.

 The BBC  report states that when the Product owner (customer representative in Scrum) was not able to provide adequate input the team went ahead without them and saw this as an improvement for a while. This is at odds with Agile thinking.  This may be because it was a technical project and the technical staff, who are also likely end users, thought they knew as much as the Product Owner.

Sapient’s customer was the US Navy who were at war. Whilst the BBC continued without the customer Sapient refused to allow this. The report stated that “Collaboration… is perhaps the most critical aspect of making our clients successful.” [8] embodying “Customer collaboration over contract negotiation.”[6] Sapient built trust by gathering requirements face to face in a truly collaborative, iterative, cross functional  manner with people ignoring the job titles to ensure the work is completed such as minute taking to establish client solution ownership . Time-boxing prevented wasting time on finding perfect solutions. The results were then replayed like retrospectives in light one pager documents, embodying “Working software over comprehensive documentation”[6]. They also used visual models and aligned terminology. Work was prioritised by benefit using a force ranking pattern to quantify them then investigated in priority order as with a backlog and iteration planning. They did however have limited time to up front to gather requirements and may have benefited from test driven requirements, using a framework such as Fitnesse, to reduce ambiguity. Patterns could have been too prescriptive but Andy stated that Sapient realised the limitations and enforced “use right one at the right time”[2]

Roles

The BBC filled all the Scrum roles with scrum teams, scrum masters and product owners. The BBC also used coaches in advisory roles due to some practises being unintuitive coming from waterfall. Mike defended the large number of scrum teams as synchronising the sprints meant they were able to create a deliverable in the first sprint. Seven of the teams were already using Scrum+XP [1] As scrum master, Mike ensured scrum values and practises were enforced. He therefore voiced concerns about the Product Owner who could not fulfil his role. Two product owners, against Scrum recommendations, had benefits but they still had too much work and did not align so the teams ignored them. A Development producer (based on a Product owner) was assigned but lost the big picture. They returned to a single product owner again but kept in team product owners. [5] In hindsight this is logical as it mirrors the scrum of scrum roles for the scrum master.

 The Sapient requirements team was cross functional with The Project Manager, Technical Architect and developers. “For the workshops, they took on different roles: lead facilitator, two facilitators and two note takers. Sapient also had a steering committee of shareholders conducting regular meetings to checkpoint the progress of the deployment work. Given the Project Manager, shareholder reviews and facilitated workshops this indicates a Prince2/DSDM influence, appropriate where rigour is required, but that roles were allowed to change.

Methodology and adoption

The need for Agile differed between the two companies. As such they adopted different styles.

A lack of communication and collaboration encouraged the BBC to adopt scrum. “It was not hailed as the future, and Scrum was even not mentioned… Rather than applying practices by the book, teams at BBC embraced principles and that influenced their internal culture “[10]. The iPlayer project established the Agile working practises which worked well in isolation including synchronising the sprints to deliver working software, daily scrum, scrum of scrum, focus on impediments and cross functional needs. Scrum was a good choice given its management friendly approach and its simplicity [appendix C]. Whilst it requires the team to commit to deliverables which management did not appear to require, aligning releases from 9 teams did. Blending it with XP gave  additional engineering rigour but given the issues with the core team they may have been convinced to adopt DSDM which is the only one to have a traditional project manager and has more rigour, although it is unlikely the scrumteam would have accepted this. Kanban may also have helped them spot where the bottleneck with the Product owner was occurring. The BBC therefore focused on technology and forgot the Product Owner and didn’t get buy in from the central team.

Sapient sought to “collect requirements … as early as possible” [8] which appears at odds with Agile but Sapient were, in effect, building their backlog. Sapient adopted traditional agile patters with strong proprietary “Patterns” to ensure that projects are right first time. [8] Andy stated that the Sapient sales team are mature to the point that they have a detailed process to capture basic information about companies up front. They have identified that different verticals have greater levels of adoption of agile practises but each project always has needs to customise (from terminology, reporting frequency to wholesale changes to methodologies and processes). Sapient also have core reporting and risk management but there is a lot of flexibility. Sapient therefore have a small set of mandatory patterns with a number of other ones that teams use unless there’s a good reason not to. From a team perspective it’s best to keep a consistent set of rules so people don’t have to continually re-learn new ones. [2]

Sapient had a very mature model based on experience and a need to be right first time. The BBC had the luxury of continuous improvement.

Conclusion

Comparing the two cases, Agile was right for them for different reasons so they followed different adoption patterns. They both prove, however, that size, criticality, management support, geographical location, whilst important, are not blockers. In deciding which method to use it’s therefore important to consider the readiness and willingness of the company. What tools, roles and cultures exist? Most importantly what issues are they looking to address?

Could one adoption pattern work for all? Crystal and Sapient’s patterns appear to meet this in many cases but Alistair Cockburn wrote That job was to produce a set of policies and documents that could be used by all consultants worldwide, and I couldn’t do it. I still can’t.” Jim Highsmith and Alistair Cockburn both emphasise “ Being agile on a 100 person team is different than being agile on a 10 person team”  [12].

Like religion, there is still yet to be a single global agreement, so Agile, whilst more quantifiable, still relies on human choice. It took a while for Micheal Keeling to convince me to adopt Pair Programming based on insignificant data.[3] So, in closing, Mike Lowery responded to my question ”could one size could fit all” by replying “No, Context is king”.

Motivation

Do you remember that feeling as a child that you really could be anything you wanted? As a birthday gift I was recently given a tour of the Millennium Stadium (the home of Welsh Rugby) where I got to look round the players changing room, run onto the pitch and attend a press conference. As a massive Welsh Rugby fan I was thrilled but as a middle aged software manager it brought home just how unlikely it was that I would ever actually play for Wales.

In my sporting prime, I rowed at an international club level for Marlow rowing club. The contrast with my Rugby dreams was stark. By the time we got to the start line of each race we had already put a lot of time and effort into getting there. We knew what our goal was and we had a tight team where everyone could count on everyone else in the boat. We had proven ourselves and we were focused on winning – never just completing the race. Without goals we would not have got to this point. Without a training plan we would not have got to this point. Without check points along the way we would not have got to this point.

It’s so easy to blame others and to make excuses when we aren’t ”winning”, but that isn’t going to improve matters. So often staff blame poor managers. Having a great manager makes a huge difference but we all have experience of “bad” managers. Rather than complaining I’ve sought motivation from them. I was told I could not study electronic engineering, I was told that I could not study at Oxford, I was told I could not be a consultant. I wondered why, so deliberately did them all largely because I couldn’t see why not and was inspired to prove these people wrong and to learn about my strengths and weaknesses.

I was fortunate that these haphazard targets lead me to success so you should think about what you want to achieve. You only have yourself to blame if you’re not living the life you want and it’s down to you to change it – not your manager. Surround yourself with successful people. Be proud and respect those around you but don’t be in awe of them. Approach people you wouldn’t normally, such as a director and show genuine interest. Whatever you do give it your all, don’t just hide or settle for meritocracy – you’re better than that.

Life is tough at times so have something that reminds you of your motivation, perhaps a model car, pictures of your children or a competition flyer. Continuously learn, improve and ask questions. Set targets, make them realistic and achievable – get there in baby steps if necessary but get a plan that gets you there.

Be the best and don’t look back and wonder “what if”.

Communications

Keep written communications short and unambiguous. No exceptions.

Capture critical decisions, agreements and proof of escalations.
Establish metrics and use them to communicate what management want to know.

Most other comms will probably benefit from discussing directly with the audience, in which case, check what your audience want to learn and check you deliver that. Think of what you want to get out of the conversation and make the point clearly.

Simple.

Thinking outside the box

I’m working with a client at the moment on a Salesforce implementation. The last time I worked with Salesforce was about 6 years ago and at that time promoting work between environments was, putting it politely, suboptimal. Given the time which had passed I was keen to see what progress had been made and was amazed to find that unless the work can be performed through the metadata API then the work has to be performed manually. No, really. No rigour, no repeatability, manually. Seriously. What other enterprise system would get away with this. Worse still no one seems to have created their own solution to solve this. My client were using a well known specialist in this area and were being advised that there was not alternative. Within a few minutes I was proposing a couple of solutions – the first to use selenium to perform the changes and to repeat them consistently. Salesforce has a web front end so I should be able to navigate the DOM and find fields in a programmatic manner which I can create a “release note” to reproduce. Looks like I might have to find some time to create a harness and then sell it to a number of disillusioned software teams.

Simple win #2 Divide work into small deliverables

When projects are over a certain size, both in terms of people and elapsed time,  estimating the work, assessing progress and testing the work all become increasingly complex.

In general larger projects are very difficult to manage due to:

  • More uncertainty about the requirements still being relevant at the delivery date
  • Accepted tolerances can still be quite large as a 10% of a large number can also be large.
  • Overheads with communication channels.
  • Longer test period which hold up environments until all tests are completed.
  • Less focus to start with distant delivery dates.
  • Less known up front.

In order to counter these concerns projects should be divided into manageable “chunks”. This allows the customer to receive deliverables which provide business benefit earlier.

Within each release it’s ideal to prioritise work using MOSCOW (Must, Should, Could Won’t) so that essential work is delivered ahead of nice to have work. Difficult, risky and large work can then be prioritised within the must’s to address the risks as early as possible. Ideally you will also be working as part of a cross functional team (Analysts, Developers and testers) to further ensure the right work is developed at the right time.

Simple win #1 Daily meetings

Irrespective of the project methodology, doesn’t it make sense for people to regularly get together to agree on project progress and to identify and address issues as early as possible? Daily stand-ups/meetings allow the team to assemble and check that the project and individuals are all still aligned without consuming significant time. Try the following simple technique by gathering the team together once a day:

Each person in the team has 1 minute or less to describe:

  • What did they do in the last working day?
  • What do they plan to do in the next/current working day?
  • What obstacles are preventing them from achieving their deliverables?

By answering these questions the team focus on their tasks and also those of the rest of the team. This helps ensure that analysts, developers and testers know when to expect deliverables and ensures they realise that other people are waiting on them.

To be effective these meetings should occur at the same time everyday at a time which is convenient for all. To ensure promptness you might like to ask the last person in to start. If someone cannot make the meeting then it should still happen. Ideally you’re looking for the meetings to become like clock work.

The time of the meeting will obviously depend on when the meetings are held. I typically like holding them first thing – at a time when everyone is in – at the same time everyday at a time which is convenient for all. If the team is geographically distributed then timing becomes trickier as does ensuring that everyone can hear each other – crackly, quiet phone lines aren’t optimal although skype is usually useful and cheap. If international calls are barred there are cheap national numbers which allow you to dial internationally for a minimal cost.

You can also let stakeholders know what is being worked on so they can listen in. If they have any questions they can grab the relevant people after the meeting, rather than hold the entire team up.

Give it a try.

What’s in a name?

Sometimes when I’m introducing Agile to a company I wonder how seriously people are taking me when I use terms such as kanban, Scrum, dojo etc. I also become frustrated when I hear someone state they are adopting Agile and I reply by asking which methodology and receive a response along the lines of – “erm, err, just Agile”. A long time ago I used to be a member of the SUN advisory board meeting regularly to discuss SUNs strategy. During these meetings I’d often advise against the use of martial art “belts” to indicate an individual’s ability as it just didn’t appear professional compared to my engineering background. I’ve also worked with a company to deliver a “factory” model and this suffered from highly technical people being insulted by the connotations of unskilled workforces performing mind numbing tasks. The Agile term is strictly forbidden. All this got me thinking:

Shouldn’t we stop hiding behind buzzwords and instead ensure we’re just focusing on delivering best practise?

Agile Myths

Image

Having assisted numerous companies with adopting Agile I’ve heard a lot of concerns and myths. I’m always surprised how many times I hear the same myths quoted as facts and thought I should share my thoughts:


Myth: Implementing Agile is easy
Truth: Agile does have some basic concepts but implementing it properly and making it self maintaining requires analysis, effort, a mindset shift and absolutely everyone needs to be bought in.


Myth: Agile gives instant benefit
Truth: Benefits are often possible in the short term but be prepared for a certain amount of disruption and effort aligning to any new process.


Myth: Agile means no documentation
Truth: This one actually frustrates me the most. The Agile manifesto states “Working software over comprehensive documentation”. This means focus on working software but do not ignore documentation. Be extremely wary of anyone that states otherwise!


Myth: Agile means ‘hacking’
Truth: Agile encourages prototyping and early development but this needs to be aligned with the solution, testing and the customer. Just “hacking” is not going to deliver working software in an efficient manner.


Myth: Agile means poor quality
Truth: This is another frustrating quote. Agile is about building quality in from the start not about catching it late. Quality level needs to be confirmed and adhered to right from the start.


Myth: Agile is a silver bullet
Truth: Agile offers common sense benefits but it can’t solve everything alone. It still requires effort, analysis and alignment.


Myth: Agile is easy. We can just read a book
Truth: I’ve been asked into a few companies where they have used consultants or internal staff who have simply read a book on the subject. This can provide benefit but I tend to find that they miss the mindset shift of what Agile is about and therefore miss at least 50% of the benefit.


Myth: Agile is new/just a fad
Truth: One can argue that Ford in 1910 and Toyota in 1940 were the earliest form of Lean development. Agile is just about best practise so it can’t be a fad.


Myth: Traditional delivery never works
Truth: Some projects need to deliver against specific deliverables which are locked down at the start and must not change. In this case you may be better off with waterfall.


Myth: Agile replaces everything that has gone before
Truth: Agile learns from everything that has gone before by identifying best practise which delivers.


Myth: Agile means no planning – “Just do it”.
Truth: Agile prevents analysis paralysis and planning with unknowns but focusing on the immediate deliverables. These deliverables need to be clear and agreed or they will not be accepted.


Myth: Agile doesn’t support Jnr members
Truth: I would always favour a small group of highly talented individuals but we need to bring new people on and some tasks would better suit Junior members of staff. Just ensure they have the tools to do the work and they are supported and trusted.


Myth: Agile = Scrum.
Truth: There are many types of Agile. Scrum is just one. Each company should look at the different types (e.g. XP, DSDM..) and choose whatever works for them.

Agile introduction

I need to declare full disclosure up front, if you were to cut me open you would find Agile written through me like a slightly overweight, middle aged piece of seaside rock (hopefully this reference won’t be lost on my non UK readers). With this in mind my early years as a developer were spent frustrated that projects were taking too long, test were engaged too late and the business weren’t engaged at all. I appreciated that a process was critical and we needed to agree what we were to deliver and provide checks against progress but did it really need to take so long and be so disjointed? Over the years I learnt more about the classic problems with large “big bang” (aka waterfall) releases:

  •  In the numerous months between the customer requesting work and receiving it, their requirements had changed due to the speed of change in the real world. This wasn’t the client’s fault they couldn’t have predicted the next 6+ months with perfect accuracy.
  • During this time customers often had very little progress updates and if they did the indicators weren’t always reliable.
  • Dev felt they were too remote from the customer and feared making decisions or had to wait a long time for them to be made.
  • Test acted as game keepers trying to catch the developers out to settle the age old question of who’s better, developers or testers.
  • With a long project plan very little pressure existed at the start. Issues were found and added given the lack of pressure to hit deadlines. Panic only set in towards the end where it became clear that we wouldn’t achieve what had been committed to. We then had the choice of reducing the test duration or dropping functionality and sometimes both.
  • Often things would go wrong and it felt like some people were spending more time building their defence as to why it wasn’t their fault rather than working on a solution

You can therefore imagine my absolute joy when I was introduced to Agile. Oddly it wasn’t alien but simply collected ideas that I had seen working well on previous projects. I need to be clear that there are a number of Agile methods which we’ll be looking at in later discussions but for now a few forgive me a few generalisations. Agile focuses on:

  • Satisfying the customer through early and continuous delivery of valuable software rather than waiting till all deliverables are completed.
  • Work is performed in regular “chunks” or iterations and working software is the primary measure of progress.
  • Business people, developers, testers and analysts work together throughout the project to ensure the project remains valid and provides useful software.
  • Teams are given the environments and support they need to get the job done.
  • The team performing the work are involved with providing consistent estimates to divide the work into iterations.
  • Requirements are captured up front but we only add detail to the ones we’re due to work on in the immediate future.
  • The team are empowered to make the right decisions
  • Quality and validation are built into the process not an afterthought waiting to strike
  • The most efficient and effective method of conveying information to and within a development team is face to face conversation.
  • Any changes to requirements can be supported as long as we all agree and the impact is agreed.
  • At regular intervals the team reflects on how to become more effective, then tunes and adjusts accordingly.

Let’s be clear that Agile is not a silver bullet and won’t be appropriate for all scenarios but now that I’ve given a brief reason to consider Agile my next blogs will discuss, are you ready for Agile, what are the Agile methodologies, myths and facts of Agile, optimal techniques for introducing Agile and optimal development environments.