How to get the most from Agile by understanding the theories behind it

The power of Agile is that it leverages human behaviour as a means to achieve a goal. By understanding the theories behind the human behaviour you can better utilise them and make the most of Agile. This article will contain a consolidated link to all of the articles that I write under that title.

The Sprint

In everyday life, we observe that for every plan and corresponding action there is a reaction, such as applying pressure on the accelerator of a car will result in the car going faster. The closer together that those two events occur, the easier it is to learn the outcome. For example pressure on the accelerator will cause the car to increase in speed.

In light of this, let’s consider how Scrum works, see the diagram below (Scrum Sprint Cycle). The Scrum Team undertake Sprint Planning (planning to accelerate); they Test & Develop the plan (actually accelerate); they Demo that it was done (speed accelerated) and they review accordingly for the next Sprint (they know the level of pressure to apply to the accelerator in order to reach a certain speed).

Education theories such as Kolb, can be applied to Scrum to give us a better understanding of the process of learning within projects and how to maximise this learning of the Development Team. According to Kolb’s Experiential Learning Cycle (ELC), effective learning can be seen when the learner progresses through the cycle, we can see similarities to the Scrum Sprint Cycle where the Development Team learns as they progress through a Sprint. Activities are undertaken, observed, discussed and understood within this cycle or Sprint and then the cycle or a new Sprint continues. 

Considering our earlier outline of the Scrum Sprint Cycle and as we can see in the above diagram: the Scrum team plan [Sprint Planning] an activity; they do [Test & Develop] it; they check [Demo] that it was done and they act [Retrospective] accordingly for the next Sprint. This plan-do-check-act process is key to Kolb’s Experiential Learning Cycle (ELC).

I have observed that the correct application of this theory to Scrum is the quickest path to increased velocity and productivity.

The Team

I want you to imagine the following scenario; As a naturally creative curious person, who loves to solve problems, you are presented with a set of different ideas and asked to choose your favourite one. You are to make it real for the rest of the world. You know that if you get stuck your friends will help you out. They, like you, have done this before. I suggest that this person is eager to complete the task. This is the world of Agile. A world where you are self (intrinsic) motivated to complete tasks.

It is the polar opposite to the situation where an autocratic manager gives you a task and tells you to complete it by a set date in order to earn your pay-check. The external (extrinsic) motivation will cause the job to be completed, but the pride, ownership and satisfaction is certainly lacking.

Intrinsic Motivation Theory demonstrates how a persons desire to do something is much stronger when it is felt that it is their desire and they were not dictated to do so. In Scrum, when team members select their own User Stories they create an internal Intrinsic motivation to complete that User Story as best they can. The User Story comes complete with its ‘Definition of Done’ (Principle #7) which creates an implied contract between the developer and the team, that the User Story will be delivered back conforming to that definition.

Scrum Ceremonies

The Scrum Ceremonies are:

• The Daily Scrum (Stand-up) Meeting
• The Planning Ceremony
• The Sprint Review / Demonstration Meeting
• The Sprint Retrospective Meeting

The Daily Scrum Meeting

The Daily Scrum meeting is short, focused and provides just enough information for the Scrum Team to update each other and progress towards their goals. It is the metronome that sets the rhythm of the Sprint cadence. Only the Scrum Team speak. The ScrumMaster takes away impediments and unblocks future User Stories.

The Daily Scrum meeting centralizes communications and cuts down on the paths of communication thereby minimizing the impact of Metcalfe’s Law, which highlights an n(n-1)/2 level of communication in teams. This means that there are 45 communication paths when 10 people are involved. Where ongoing communication is required and email doesn’t cut it, centralized, single source of truth, communication platforms such as Slack & HipChat have become popular, as they offer a single update to all relevant team members.

The practice of Nudge Theory (Thaler-Sunstein/Kahneman-Tversky) applies to the Daily Stand-up on many levels. Firstly the Three Questions are framed (choice architecture) to promote productivity; they are all focused on progress. Secondly, the meeting is short (timeboxed to 15 minutes) and focused; reinforcing productivity. Additionally, Nudge Theory can be indirectly introduced via nudge heuristics from a person playing the role of a Choice Architect (warning: use responsibly).

The daily questions are the nudge that frame productivity

Where a team member may come unprepared or lacking in progress, Social Conformity and the Pygmalion Effect comes into play such that the team member is persuaded to perform at least at the baseline performance level of the team. The team provide each other with positive reinforcement to achieve non-forced compliance for the betterment the group. The reciprocity bias further promotes the behavior. Coupled with the small team size, this eliminates the Free Rider Problem

The Planning Ceremony

To successfully complete a Planning Ceremony you need to select the appropriate & completed User Stories and allocate them to the next Sprint. Simple! Not quite. Lets run through the applicable theories and practices to optimize these activities.

‘select the appropriate’ – This selection is based on your Sprint Vision, which is based on your Product Vision. Therefore before any Planning meeting proceeds you need to first set the projects strategic direction by using a variety of tools such as the Lean Canvas, Vision Board and Product Roadmap. Next, you need to prioritize the resulting Themes, Epics and User Stories (in that order) using methods such as the MoSCoW method. It is recommended that you then identify the Weighted Shortest Job First (WSJF) based on the Cost of Delay (Donald Reinertsen).

‘completed User Stories’ – Being mindful to avoid the Garbage In, Garbage Out scenario, all of the User Stories and their associated Acceptance Criteria should be specific and measurable. It is best to employ the INVEST acronym in relation to User Stories. User Stories will be discussed in more detail in a future article.

‘allocate them to the next Sprint’ – Sprint capacity (Bucket Theory) is determined and User Stories are sized by playing Planning Poker (The purpose of which is to take advantage of Crowd Theory while minimizing the impact of Convergence Theory whereby team members blindly size a User Story before being influenced and converging on a number). Within the Sprint, it is important to determine whether User Stories are dependent on each other in order to verify that they can all be completed within a single Sprint. The Critical Path method focuses on tasks and looks to create a critical sequence of events that must take place. Ideally because your Generalizing Specialists can handle a variety of tasks you will not need to also consider the resource focused Critical chain project management that was derived from the Theory of Constraints.

So now that that is all done, what could go wrong?

Plenty. But Scrum is designed to naturally mitigate these issues. Naturally you will observe that the following theories have a tendency to sometimes arise.

• Planning Fallacy (Kahneman & Tversky) highlights the optimism bias and how that results in underestimation.
• Prospect Theory (Kahneman & Tversky) highlights where things can go wrong. Just be aware of this.
• Self Justification Theory highlights why we keep on making the same mistakes.
• Brooks Law. Beware the mythical man month!
• Parkinson’s Law says that ‘work expands so as to fill the time available for its completion’ To avoid this problem, submit more User Stories for consideration to the Development Team for an upcoming Sprint then they could possibility undertake. Let them choose.

There are many theories why planning should be left until the last minute. As noted on Dr. Jeff Sutherland’s (Co-creator of Scrum) personal website; The Uncertainty Principle in Software Engineering (Ziv) states that specifications will never be fully understood and that uncertainty is inherent and inevitable. Multiple others have shown this to be true; Humphrey’s Law, Lehman’s 8 laws of software evolution, Wicked problems definition, Conway’s Law, Wegner’s Lemma, Langdon’s Lemma, Hofstadter’s Law, as well as the The Cone of Uncertainty shown here.

The Sprint Review / Demonstration Meeting

This ceremony offers the Product Owner another opportunity to inspect the Product as per their view of the Minimal Viable Product (MVP). Usually Stakeholders are invited to this ceremony and this is a chance to discuss with them, the direction in which the Product is going and how the market is likely to react to the MVP. Iterative development affords this value-driven approach.

The Product Owner should facilitate the Sprint Review and play the central role in demonstrating the output of the Sprint to the group. This act further ingrains the Ownership Bias (endowment effect or divestiture aversion) in the Product Owner. 

Take advantage of the Endowment Effect to offset the impact of Agency Theory

If tension is sensed during this Ceremony, then this is because Agency Theory is at play on two levels, at one level between the Product Owner (principle) and the Development Team (agent) and also between the Stakeholders (principle) and the Product Owner (agent). Agency Theory occurs when one party can make decisions on behalf of another, such as the Product Owner on behalf of the Stakeholders and the Development Team on behalf of the Product Owner. If either of these agents are not acting in accordance with the principles wishes, then tension arises.

The Sprint Retrospective Meeting

The Sprint Retrospective is an opportunity for the Scrum Team to inspect itself and create a plan for improvements to be enacted during the next Sprint.

Noting from my last article that the Sprint Retrospective is the time to reflect and learn. It is important to reflect on the previous Retrospective and identify whether the previous learnings were utilized. There are a number of techniques that can be used to better facilitate the Sprint Retrospective, with scrum.org offering suggestions. I favor the Starfish method (Stop, Start, Keep, More, Less) with an anonymous voting mechanism (as per ideaboardz.com) to get as much honest feedback as possible.

While the Retrospective forms part of Kolb’s Experiential Learning Cycle (ELC), the process within the Retrospective itself goes deeper to allow for Double Loop Learning. As one of the Scrum Alliance founders; Esther Derby puts it;

Single-loop learning asks, “How can we do what we are doing better.”

Double-loop learning asks, “Why do we think this is the right thing to do” and involves scrutinizing values, thinking, and assumptions

The Sprint Ceremonies involve the whole team and this draws criticism, but I hope that I am starting to demonstrate in this series of articles that Agile draws on the best of human behavior. When it is not misused (eg: Scrumbut, Fragile) but instead understood and utilized correctly you will experience more solid, performant and pleasing results, both within your daily work life and in the project output.

Scrum Roles

• Organizational Behavior
• Senior management,
• The Product Owner
• The Scrum Master
• The Team

It is important to note that Agile & Scrum are not simply a set of practices to be implemented according to a rigid set of guidelines or checklist. Successful Agile implementation is best described as a state of mind or mindset. The theories observed here are the manifestations of that mindset. You can use them to grow the desirable behaviors that springs forward from Agile.

Organizational Behavior

“Understanding human needs is half the job of meeting them” – Adlai Stevenson

Regardless of our role within an organization, there are general Organizational Theories that relate to everyone and therefore must be understood, namely; Maslow’s Hierarchy of Needs, or ERG Theory, Taylors Principles of Scientific Management, Theory X and Y (Douglas McGregor) Organizational Structure and Job Characteristic Theory (Hackman & Oldham) that ‘proposed a model of five “core” job characteristics (i.e. skill variety, task identity, task significance, autonomy, and feedback) that affect five work-related outcomes (i.e. motivation, satisfaction, performance, and absenteeism and turnover) through three psychological states (i.e. experienced meaningfulness, experienced responsibility, and knowledge of results)’. You may note that this is a super-set of the Motivation Theory (Pink). PERMA Theory (Seligman) also known as ‘Well-Being Theory’ has been gaining interest in the context of a work environment as it digs deeper into a persons measure of well-being.

Addressing these basic needs goes far in creating a high performance culture within an organization.

Cross-cultural organizations benefit from broad rules of behaviors based upon Values, Expectations and Ad-Hoc Rules (SW-ICCM), as show by Xibao Zhang in his thesis. Hofstede’s Cultural Dimensions Theory lists the six dimensions upon which societal norms are woven into the fabric of our being & therefore effect culture within organizations.

Senior Management

Senior management, sitting outside of Scrum teams need to set a clear strategic direction for the company, enabling employees to embrace the organization through a clear set of company values. A HBR study sums it up well where they discuss the primary management practices—strategy, execution, culture, and structure. A Cummings & Worley study lays out how such change can be effected in six steps. Simon Sinek simplifies this in his Golden Circle Theory.

Within the context of Agile, traditional organizational cultures and structures would be required to reorganize if large scale Agile projects are adopted across the enterprise which are best suited to Agile Frameworks such as SAFe, Less or Dad. This would typically be the case in a business transformational project, or large scale projects with over 50 participants. 

The frameworks listed above could be more readily adopted in the new generation of organizations that have Radical Management(SM), Holacracy, Cynefin, Management 3.0 or Teal Organizations at their core. 

The Product Owner

The Product Owner acts as an Agent (Agency Theory) on behalf of the project Stakeholders. They own, prepare and prioritize the Product Backlog, incrementally feeding User Stories to the Development Team based on the Sprint Vision. Together with the Development Team they jointly form the Scrum Team. 

One of the key duties of the Product Owner is to protect the Development Team from outside influences, and then inspects, validates & Quality Assures their Sprint Output. Their other key duty is to deliver a Return on Investment (ROI) for the Project Sponsors & Stakeholders; customers and business units such as Marketing, IT, Customer Support, Business Strategy etc.

Unskilled Product Owners represent one of the biggest impediments to Agile

Given the diverse set of skills that the Product Owner must command, it is not surprising to note that unskilled Product Owners represent one of the biggest impediments to Agile adoption according to 49% of respondents to a Forrester survey. They need to keep an open mind, not fall prey to Confirmation Bias in their duties, and watch out for some of these specific pitfalls noted by agile.org.

Many college courses are devoted to Management & Leadership and so my purpose here is not to comprehensively cover this coursework in this article, but instead, I would like to point out the Management Theories and that most impact the Product Owner. I should first point out that the Product Owner is not a manager, in fact if a Product Owner were to issue autocratic diktats then we can certainly assume that the project is doomed. Instead the Product Owner needs to trust the Development Team to deliver upon their promises. This trust will be reciprocated as per the Swift Trust Theory (Meyerson).

Studious managers who fulfil the role of the Product Owner, may have noticed their focus shifting from literature on ‘management’ to ‘leadership’, this is due to the nature of the role of the Product Owner. This HBR article covers the transitional topic well.

The Product Owner needs to lead the Scrum team. As such, the Product Owner should understand their Leadership Style and should position themselves between a 5 – 7 on Tannenbaum-Schmidt Continuum depending on the project, to adequately lead the project.

“When dealing with people, remember you are not dealing with creatures of logic, but with creatures of emotion” -Dale Carnegie

The Product Owner should understand their Base of Power (French and Raven) and be very aware of the legitimacy (or lack thereof) of your ‘power’. They should be aware of the traits of leaders & five practices of exemplary leadership, as the Implicit Leadership Theory (Lord) informs us that we already have preconceived notions of what a leader looks like. Most of all, however, they need to focus on building upon Emotional Intelligence (EQ Theory) (along with the 99 other Qs) and recognize their own strengths and limitations before they fall prey to the Peter Principle whereby they stop performing effectively, and “rise to the level of their incompetence”.

In both the project that they are engaged in and the engaging product that they are creating, it is worthwhile for the Product Owner to be aware of the Peak End Rule (Fredrickson & Kahneman) to create memorable experiences by actively looking for the peak (enjoyment/engagement/interaction) and ensuring that the experience ends well.

The Scrum Master

The Team

The Development Team are cross-functional, self-organizing and self-managing (empowered).

When a group of individuals join together to solve as common problem as a team, they naturally progress through a four stage process as captured in Group Development Theory (Tuckman). These stages are Forming, Storming, Norming and Performing, where Performing equates to a strong velocity in terms of Agile. Adding an additional team member resets the stages and the team must go through the process again. Based on this theory, it is clear to see that adding a new team member will make the team temporarily less productive & therefore this should be avoided as much as possible. Of these stages, I’d like to single out Storming in order to highlight its importance. Motivation may be sapped during the Storming phase as ‘conflict and tension are resolved’ and a natural leader arises (This may not always be the Scrum Master!). Working styles, personality traits and undefined roles are all sources of conflict. While it is recommended that the Product Owner allow the team to self organize, it is possible that the team can become stuck in the Storming stage, in which case, the Product Owner may wish to adapt their leadership style [more detailed] to best suit the phase in order to assist the team.

It is only when the team are in the Performing stage can the team perform optimally. This usually happens after several Sprint Cycles (after several opportunities for learning) and is called hyper-velocity in Agile. Flow Theory (Csikszentmihalyi) equates to the expression of hyper-velocity when the Scrum team are fully immersed in their tasks and can be informally described as ‘in the zone’. Interrupting any of the team members during the Sprint while hyper-velocity is occurring can impact their current User Story by increasing its duration by anything between 20% to 50%. This is the process of Context Switching Theory (Weinberg) and it strengthens the necessity of the practice of not multi-tasking within the Sprint & observing the definition of Done / Done before moving onto another User Story. It highlights the fact that persons outside of the Development Team should only ask team members questions at the designated times (eg: Grooming / Refinement) or to a designated person such as the Product Owner or Scrum Master, in order to allow for hyper-velocity to occur.

We have already seen from the first article, that the team are about to engage in Kolb’s Experiential Learning Cycle (ELC) as part of the Scrum Cycle, but it should be noted that each team member will learn in a different way. The Honey & Mumford Learning Style Theory classifies people as either one of four types of learning styles; Activist, Theorist; Pragmatist and Reflector. ELC incorporates the Honey & Mumford four Learning Styles of;

·      Activists are those people who learn by doing.

·      Theorists learn by understanding the theory behind the actions. 

·      Pragmatists learn by putting that learning into practice in the real world. 

·      Reflectors learn by observing and thinking about what has happened

In summary, Scrum accommodates all learning styles and so everyone benefits. People sitting outside of the Scrum Team can also benefit in a process called diffusion chain, as stated in Observational Learning Theory. Coupled with the impact of the Intrinsic Motivation Theory it becomes clear that the Team have everything they need to determine their own success (Self Determination Theory (Deci & Ryan))

As this article focused on people more than processes, it was always going to be the most complex and likely for omission of a relevant theory. However, I hope that it has informed you of new theories, reminded you of old ones or enlightened you to realize that actions that you undertake have a theory behind them and perhaps could be optimized.

The Scrum Backlog

The Backlog is the name given to the collection of features, tasks and ideas bundled together as a collection of unique short descriptions called User Stories. The goal of the Backlog is to identify, from it, a Minimal Viable Product (MVP) to bring to the market. This is done via the Product Vision and the individual Sprint Goals set by the Sprint Vision, as they iteratively lead to the initial Product Go-Live. Below, I have applied Simon Sinek’s Golden Circle Theory to the Scrum Product Vision triangle and set it on the foundation of the Agile Testing Pillars (described later). The Golden Circle questions;

The Product Backlog is not just simply prioritized once (using the MoSCoW method, Cost of Delay (Reinertsen), Risk Based Scrum Method & noting the Pareto Principle) and then implemented as envisioned at the start of the project. It is added to, deleted from, modified, reprioritized, chunks are broken off & allocated to Sprints, while the remainder of the Backlog may undertake a redefinition of what it means to be marked as complete. It is governed by Chaos Theory. The ever-evolving unpredictable Backlog equates to Chaos. The Sprints bring about Stability (Order / Disorder Chaos). Chaos is reintroduced again through the Feedback offered by the Grooming / Refinement and Planning sessions – outside of the core activities within the Sprint. The Butterfly Effect can be seen in the way that small changes to the initial conditions (e.g.: Definition of Done & Garbage In, Garbage Out) lead to drastic changes in the result.

Within the Sprint Backlog, the Theory of Constraints (TOC) comes into play. The theory states that a chain is no stronger than its weakest link. Scrum has already identified and strengthened the weak links; Late integration of code is mitigated by continuous integration. Late testing is replaced by Test Driven Development. Late bug detection is offset by Pair Programming and Inspection & Adaptation. Queuing Theory, plays its part within the Sprint, as does Little’s Law in reducing Work In Progress (WIP) levels to reduce average cycle time. However, some note that this Law is overstated.

As an aside, I have found it worthwhile representing the Sprint Themes & Epics in a colourized Pivot Bar graph. For me, this serves two purposes.

1.   Goal Mapping; Clear mapping of Themes & Epics to the Sprint Goals

2.   Stakeholder mapping; Clear mapping of Themes, Epics & Sprint Goals to Stakeholders within the Business, beyond the Product Owner.

User Story

Every idea (features, tasks, requirement) can be entertained as a User Story. It is captured, defined, categorized, viewed through the lens of a persona and the value that it creates is defined. Each User Story is specific, or SMART / INVEST, each one represents a Minimal Marketable Feature and each one is prioritized and earmarked to a Sprint based on the Product Vision by the Product Owner.  As with any task, User Stories can be written efficiently (top tips) and inefficiently (common mistakes to avoid).

Each User Story has a number of attributes that have associated theories behind them;

The Definition of Done is an extremely powerful tool as it acts as a Social Contract on behalf of the Scrum Team members. It defines the criteria by which the Intrinsically Motivated team member will deliver the User Story back to the Scrum Team. While the Acceptance Criteria defines the boundaries of that User Story.

The specter of Technical Debt is ever present. Technical Debt arises when code that is quick and easy to implement is used instead of applying the best solution; quickly leading to bugs & defects. Ward Cunningham observed that there is an increased effort for code changes as design entropy occurs, or put another way, it takes longer to fix the problem later than it does initially. This is confirmed by Lehmans Law which states that the older the code, the longer it will take to maintain it.

If bugs do make it into the Product, then the Broken Window Theory demonstrates the benefits of removing bugs as soon as they are identified as opposed to deferring them until later, as this will lead to a gradual degradation of the overall Product, unless they are actively rectified.

Great Agile Products stand upon the firm foundations of test strategy such as the 3 Agile Testing Pillars of:

• Automation and Tools (further details below)
• Software Testing (functional, non functional, exploratory, release based)
• Team Practices (Pair programming, code reviews, 3 amigos conversations)

This concludes my exploration and analysis of the underlying theories behind Agile, with a particular emphasis on Scrum. In time, I’m sure I’ll find that the series wasn’t exhaustive, but I hope that you have enjoyed reading / skimming some of it and maybe learned something along the way.