An Agile Approach to System Documentation

While a popular conception is that Agile methodology opposes the creation of documentation, most mature Agilists disagree, including the originator of the Scrum approach.  In Sutherland’s OOPSLA ’95 paper, the first to define Scrum, he describes a robust design and documentation phase called the ‘pre-game’ occurring prior to the beginning of sprints and additional documentation at the conclusion of sprints prior to release.  Scott Ambler, author of Disciplined Agile Delivery makes an exhaustive case for why and when Agile practitioners should create documents.  The following briefly summarizes the case and best practices he recommends and applies them to projects managed by my team (see also my specific discussion of project charters).

Taking an Agile stance on documentation requires consideration of the 5 W’s: 

• Why – Documentation should be the product (other than code) that meets requirements of users’ stories and those stories should be captured and prioritized with other stories.  These stories are expected to support all stakeholders rather than just system end-users.
• When – As late as possible; after delivery of the software itself where possible.  Documents for immediate use like user training and operations should be considered during the code development.  However, with the expectation of gradual knowledge loss as the resources move on to other work, documentation must be done before the information is forgotten.
• What – Lowest cost forms of the information that satisfy the needs; should avoid bulky templates with boilerplate in favor easily constructed, consumed and maintained forms.  Visual diagrams and models are an obvious approach, as are data-based and folksonomy approaches which can be maintained in parallel with the code.
• Where – Kept in a central location accessible to all stakeholders
• Who – Knowledge creators should capture as they go then ‘productize’ the knowledge as documents at the project stage where it is possible and/or necessary.  Productizing can be done by separate resources with analysis/documentation skills.

User Stories for system documentation – The following examples taken from projects my teams have managed illustrate the demonstrable value provided by system documentation for different product stakeholders.

While many Agilists seem to assume that products made using Agile approaches are simple, end-user transactional systems, many times our products are intended to support complex back-office operations and may be replacements for existing legacy systems and processes.  Since legacy system replacement often requires an MVP that is larger than an otherwise ideal Agile project, our project durations can be longer as well. Also, we find that we may have significant time gaps between projects. In that context:

• As a business operations user of a system, I need to learn how to use the system to perform a business process:
  • Operational teams relying on software tools for their business processes need up-to-date documentation on system behavior and its application to their business.  When processes change as the result of system changes users must be retrained to follow the updated process using the new system features and may need ongoing access to reference documentation when system usage questions arise.
• As a system support resource, I need to resolve issues raised by system users:
  • End user support is typically provided by resources that were not on the original product development team and so need reference documentation to allow them to support users, resolving their questions and identifying system issues.  While this need is true for most software products, it is especially true where the system supports complex processes requiring support teams to support highly complex situations with end users.
• As a member of a project team, I need to understand work that has been completed in earlier projects creating earlier versions of the product or supporting the same users.
  • Project members may not remember all facets of a project at a later time when work resumes and may need to refresh their memories of assumptions, constraints and decisions made during design, build and implementation.
  • Reuse of test cases and their expected results contributes to efficiency and can provide the baseline for regression testing.  To achieve this productivity use of the test cases and results must be fully captured by the previous project.
• As a functional manager, I need to be able to manage resource allocation and enable resource transitions with minimal loss of productivity.
  • While some Agilists assume intact teams throughout the delivery of a product, reality often does not cooperate.  Resource managers must have the information within the project necessary to cope with resource turnover and transitions. 
  • When the team lacks sufficient resources or the full range of skills needed to deliver the product, then resource managers need to be able to use resource augmentation via cross-team collaboration.  Examples include:
    • Integration with other enterprise systems with dedicates resource pools.
    • Making enterprise-impacting architecture decisions which may either be centralized or require collaboration with teams owning other systems.
  • As well, when the team and the broader organization lack sufficient resources or the full range of skills needed to deliver the product, then resource managers need to be able to use resource augmentation via outsourcing or vendor partnerships.  Supporting the required knowledge transfer between internal and external team members requires durable knowledge sources.

While the specific user stories may vary for your organization, the general idea that Agile teams need to consider users beyond the typical end user is critical and yields a host of users’ stories like these that demonstrate the need for suitable documentation of the system.  Additional user roles to consider include back-office business operations, end-user support, functional managers of product development team members, and new team members learning the system and its implementation.