I’ve often wondered where the idea of self-organizing teams originated and why the approach is considered essential to Agile methodology (see also post on Nonaka & Takeuchi). The idea is stated as the 11th Principle supporting the Agile Manifesto: “The best architectures, requirements, and designs emerge from self-organizing teams.” The principle is enigmatic in that it doesn’t define the term nor provide any clues as to what drew the authors of the Manifesto to believe in its veracity.
The term “self-organizing” originated in the field of cybernetics in the 1940’s, spread into business management in the 1960’s and other fields like general system theory, complex adaptive systems, economics, and psychology in the following decades. Self-organization is defined in the Wikipedia as:
A process where some form of overall order arises from local interactions between parts of an initially disordered system, also called spontaneous order. The process can be spontaneous when sufficient energy is available, not needing control by any external agent. It is often triggered by seemingly random fluctuations, amplified by positive feedback. The resulting organization is wholly decentralized, distributed over all the components of the system. As such, the organization is typically robust and able to survive or self-repair substantial perturbation.
The contexts in which it has been used, and this definition, relate to large numbers of independent agents exhibiting undirected group behavior. Examples include phenomena like murmuration of starlings, schooling of fish, crystal formation, critical mass, herd behavior, and groupthink. What isn’t clear from this definition and these examples is how they relate to software development teams, which are orders of magnitude smaller and engage in directed group behavior.
My search begins with the authors of the Agile Manifesto and their publications prior to its 2001 release. Working back in time from the February 2001 formation of the Agile Alliance, the first author I considered was Jim Highsmith and his book Adaptive Software Development. In the book, Highsmith focuses significant attention on self-organization as a principle and its application to software project management. In this approach he is influenced by the writing of John Holland about complex adaptive systems and Ralph Stacey who applied complexity theory to managing organizations. With those influences, he defines self-organization as “the tendency for living things to work together for some common purpose in the absence of some central organizing force.” (p. 11) Among the properties of self-organization, he especially prizes emergence which he defines a property of complex adaptive systems that creates some greater property of the whole (systems behavior) from the interaction of the parts (self-organizing agent behavior).” (p. 12) Highsmith portrays emergence as the key ingredient to a team’s ability to adapt to complex and dynamic situations to successfully deliver software products.
At the same time, he writes that his “view on project team governance is that so-called self-directed teams, in fact, have no leadership. At times in the past, infatuation with empowerment may have led practitioners to assume that the need for leadership and leader decision-making no longer exists. Maybe there is a place on some projects for self-directed teams, but not on an extreme [complex and dynamic] project.” (p. 214)
In a blog No More Self-Organizing Teams in 2007, Highsmith continued his critique and acknowledged a significant issue with “self-organization” being misinterpreted and used to justify anarchistic, leaderless team organizations. He disapproves of both and advocates for use of the phrase “light-touch management” instead, which he defines as “light in terms of decision making, it is heavy in articulating goals, facilitating interactions, improving team dynamics, supporting collaboration, and encouraging experimentation and innovation.”
In the end, I can’t decide if Highsmith is the originator of the 11th principle. While he certainly researched and thought deeply about self-organization and emergence as elements of managing software projects, his definitions diverge from those in the prior literature and ultimately, he abandoned the term as a negative influence on Agile practice. But if he wasn’t the primary source, who was? More on that at a future date.
