Agile and Iterative Software Methodology for Application Support

Long lives the Support!
Basic principle of the agile methodology is to make the software process such that, there is negligible support requirement in post implementation stages of software life cycle. A bug free software means complete customer satisfaction. If software works the way is defined in requirements, it is bug free. Somebody wise said, there is no logic without a bug.

�Since human beings themselves are not fully debugged yet, there will be bugs in your code no matter what you do.� We work to minimize the bugs in the software we ship, but they�ll always be there.
- Chris Mason (via Anatomy of a Software Bug)

Is it even possible to write a bug free software? Yes. There are a lot of theories put forth by learners for defining a Bug in software. Some Ontologists   like to term bugs as defects. All seems to relative, but if you ask me, if we need support, a bug free software is yet to be written. Virtually, there is no software ever made which does not have break-fix requirements, or maintenance requirement. As long as the software lives support lives.

Why do we need a change?

If we have support needs, we need a correct and optimized way managing that need. After working in technology support for more than 3 years now, I believe, there isn't really a good software methodology for Support. There should be some way to manage and use support time effectively. Diminishing support time and usable software are really the main objectives of Support. Effectiveness of software development methodology is inversely proportional to support time. If you are a support engineer, you will have some idea of incidents based on the development life cycle. Though you have good understanding of base technology and good development, it really a tough to estimate the time needed for support for a new system. All support individuals provide a huge estimate for support based on some gut feeling. At year-end either they run out of budget or they waste the estimated hours.

The support is always incident based. We cannot plan for the unplanned. The incidents just happen, we do not control the way they happened, when they happen. This hampers the ability for planning time for those in a software shop. So finally we have to get a new support methodology, which is effective enough and provide a better value to the customer/client. We need some way to identify majority of loopholes in support techniques. They are the primary hour consumers.

Support management needs to keep the support resources attached to their daily support activities. They get bored of continuing the activities over the time. Team needs rotation in support activities. The methodology focused on support resources and working software can display great payoff.

That means, do we have to plan for some unmanaged support tasks of support to generate a support model to manage the incidents? Planning for unplanned activities? Answer is YES. We need to utilize the available time with support consultants to provide more value driven activities. At end of support cycle we should be able to provide the value to the customer. The value not only for incident based activities, but utilizing between ticket time. The work completed by the consultant during that free time is purely a bonus.

Let's see what we do as typical activities in Application Support?
Actually, what restricts us in coupling the infrastructure support and application support. Usually the application support consultant and infrastructure support consultant should always work together to resolve majorities of issues. Usually, it is difficult to find a consultant with skill set of both infrastructure and code maintenance.

Support is a very time critical and problem ownership activity; consultant needs to have a good attitude to engineer the resolution to an incident.

Engineering a resolution means

• Defining a problem
• Finding and analyzing the symptoms on interfaces and also inside application
• Replicating that in test and/or development environments
• Isolating the problem condition to code/environmental snippets.
• Analyzing probable fix dependencies.
• Finding and creating a fix in development/test environments.
• Regression Testing for adverse effects.
• Deploying the solution with correct and well tested back out procedure.
• Updating the system documentation for the changes made.
     

All the support activities can be generalized and classified as follows.

• Emergency Support:
  • Activities that are typically those which are termed as incident based activities.
  • They involve resolution and/or break-fix condition of Severe/High/Medium priority issues.
  • This can be due to the code level conditions, bad data conditions, dependency system failures and virus or security threat conditions requiring emergency patch updates or system upgrades.
  • If a change management system is used they are typically rated as Emergency/Incidental Change.
  • Usually they are call based, it can user generated call ticket or system generated/automation generated call ticket based on their severity.
• Application Enhancements:  These are typically code changes due to but not limited to following reasons
  • Request due to the functionality enhancements.
  • Performance enhancements in application.
  • For improving the application supportability.
• Application Maintenance : These are typically the code changes due to but not limited to following reasons
  • Changes in application for incidents slated as very low priority.
  • Changes in system, which are found as problems by the support, and need to move those to add value.
  • Application garbage cleanup like removal and/or archival of log files, restart the application to get new settings effect or to resolve memory leak issues.
  • Product version upgrades, operating system upgrades, system service packs, security updates.
  • Dependency system maintenance.
  • Updating system documentation.
  • Usually the code changes are made via a Maintenance move, if we use change management systems, these changes are typically rated a Normal/General Maintenance change.
• Application consultancy :
  • This is when other interfacing applications require help to resolve incidents with their application.
  • Any help needed from development project team regarding the application.
  • Participating in disaster recovery trial runs.
  • Participating in user training activities.
  • User requests some help while system usage.
  • These are the places where we do not make any changes to the system but support individual provides the system knowledge to others.

Here is small document I found on internet, defines the process models for support.

XP in a maintenance environment (also known as extreme maintenance) is very common. According to Kent Beck, �Maintenance is really the normal state of an XP project. (Cohen, et al. [2003])� The project evolves over time because of frequent
iterations. The first iteration can be considered the initial release. Therefore, all following iterations are, by definition, the maintenance stage of the development cycle.
via . Dept of Comp Sci, Tuft Univ.

Let's see if we can apply the Agile and Iterative Practices to make support better.

What is Agile and Iterative Practice Principles and Manifesto talk about Support?

In reality, NOTHING..... Adopting and then tailoring a software process to meet your team�s needs is an important and difficult decision, one that is critical to your project�s success. There is a myriad of choices, ranging from prescriptive processes such as the Rational Unified Process (RUP) and the OPEN process to agile software processes such as eXtreme Programming (XP), SCRUM, and Dynamic System Development Methodology (DSDM).  All these processes are descriptive enough to adapt to project development life cycle. We need to fabricate those for the needs of support. These cannot be directly applied for the technology support, due to the uncertainty of work. The manifesto of agile states as follows, this applies as is to the support .

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Once we know the ideology behind this practice, it is easier to follow and modify to fit the requirements. Usually changing the waterfall attitude towards software is difficult and requires time to adapt to this new process. Support should never find this problem, as there is currently no software methodology in place. Usually organizations follow a simple methodology based on processes and procedures evolved over the time. These processes and procedures are not backed by any research. They are usually not known or understood for the why these processes are made, but just followed as a formality. These processes are meaningless without their original context.

The process should be flexible enough to implement and modify to changing requirements. No matter what you follow, it should always provide a measurable value to Customer. Most of software development cycles follow a single procedure for providing benefit to sponsor. They forget to provide attention towards Supportability of application. That factor is going to yield higher benefit to customer over the years. Project tem has to make customer/sponsor understand better is software supportability, lesser is the cost for maintenance, and lesser is the systems running cost. Agile Principles for Software development, which we need to modify based on the requirements for support, but keeping the philosophy same.

1. Our highest priority is to satisfy the customer through early delivery of valuable resolution and working software.
2. Welcome changing requirements, even late in resolution. Agile processes harness change for the customer's competitive advantage.
3. Keep software working, frequent monitoring and scheduled audits keep system usable. The audit interval can be decided by the criticality of system uptime or business loss due to downtime. Monitoring tools are must for effective support, if you do not have some kind of automation for monitoring, it is wastage of a support engineer's skill set.
4. Business people and support engineer must work together throughout the incident resolution. This assures a faster and effective resolution. Regression testing by clients can improve the quality of resolution.
5. Maintain the systems around motivated individuals. Give them the environment and support they need, and trust them to get the defect fixed.
6. The most efficient and effective method of conveying information to and within a team is face-to-face conversation.
7. Working software is the primary measure of progress. For example, support health can be measured by number of incidents and changes in production system.
8. Agile processes promote sustainable support skills. The sponsors, engineers/consultants, and users should be able to maintain a constant pace indefinitely engineering a resolution. Longer is the time between incidents, longer is the time required for resolution. So promote the periodic system checks and/or knowledge updates even though there are no system breakdowns. I make an analogy to military exercises; armed forces do not wait for a war to harness their ability to win the war.
9. Continuous attention to technical excellence and good design enhances agility.
10. Simplicity--the art of maximizing the amount of work not done--is essential. Simple is always best.
11. The best practices, attitude towards the problem ownership, business understanding and technical excellence emerge from self-organizing teams.
12. Application software should provide value in changing requirements. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.  

Paired Support

Support structure should not be tied to a person. Support engineers join and leave as time progresses, but systems don't. We know some systems used in aviation, insurance and baking industries are supposed to have a software life of around half a decade or more. Support organizations need to build a process around the system to sustain the support knowledge with changing support engineers. There will always be an increase in problem resolution time with changing support person.
Most of the organizations follow "Eat you own dog food" principle. I am very sure development team can provide best support, but as the size of organization grows, this model becomes less effective. There is no resolution engineering in development team. A resolution engineering activity requires a different attitude and skill for efficient support.
Team that has interchangeable support engineers is the best model for support. There are no specific roles. Paired support does not mean to create fixed pairs of on call schedules, individuals in pairs must be rotated. By rule, junior drives the pair, so that he learns fast. Everybody can rotate support activities and they can share the responsibilities.  Paired support means two resources engineer a resolution together. Paired support leads to person independence, allows resources to avail opportunities other than being on call. Paired support makes support knowledge in persistent in team.

Time Boxed Support

Support is a continuous activity, does not come as time-boxed process. We can divide it for sake of planning called iterations, where we can plan application enhancements. Support activities can also be planned, especially; break-fix changes can be planned through their code fix and regression test cycles. Upgrades can be managed to avoid the impact on other systems. Customer is required attendee for iteration planning meeting (IPM). Customer drives the priority for the support tasks.

Sometimes, systems are so huge, it is really difficult to decide who is customer. Anybody who is uses the system, has clear understanding of business, and can represent his whole community can be a customer. When customer is present in IPM, it adds the accountability, and customer knows cost of each small fix cost. Customer can make wise judgment on cost of fix and value it adds to his business. Customer exactly knows what he is paying for is more important to create customer satisfaction.

Iterations allow to change estimates and priorities on the basis of history and customer impact at some manageable intervals. These iterations can be larger based on size of support need for application, or number of applications supported in team. Iteration planning is mainly for getting better on support estimates, and allowing window to plan for unplanned support.

Support Metrics : Plan & Learn

Plan as much as you can, but planning is most difficult part of support. Recurring support activities can definitely be planned. System changes can also be planned. Make marginal planning for unplanned. Keep the record of all the activities. These activities might sound as overhead, but they can yield great results when support metrics matures. Support metrics not only tracks the amount of work done, but also helps in prediction of expected work.

Support time is addition of dependency wait, analysis, problem recreation, resolution development, deployment, and testing times. Ability to track down those activities and relate to percentage of total time can build efficient metrics for support predictions. These tracking needs should not turn on engineers as overheads. We tracked the support requests by problem tickets.

These matrices can build good estimation base for new applications and defining technology based support tasks. More important is how we intelligently we use that metrics. This metrics should also account for complexity of the system. After maturity, calculate the support momentum, organizational latency due to dependencies. Support momentum is average time for resolution, this will reduce with maturity of team. The number of incidents must decrease over time, that means team capacity rises significantly. Decreased momentum means but not limited to gain in team efficiency, increased system knowledge, added value to customer, and grown capacity for more work.

More granular tracking provides ability to identify support automation opportunities, means opportunity to reduce support cost. As per my manager Matthew Feldt, any activity repeated for third time, needs serious consideration for automation.

    So plan... plan .. plan and then learn to plan better.

Team Communication

Team communication is very important in support, if a pair is working on problem; it is very likely that someone else in team has already seen this problem. Daily standup is very simple and easy way of communication. Team should be together, at one location, at maximum of 7 feet between two members, that provides more opportunities for communication. This builds team spirit and increases knowledge sharing. Other ideas behind everyday support stand up meeting are to promote team leadership. Team knows, what is better for application and team, manager remains a silent facilitator. Manager helps in building the environment for team for efficient work; manager creates a cocoon for team to grow in leadership and self-sufficiency.
Work is planned and tracked in iteration. Customer/Client is always present in the standup meeting, and this helps in prioritizing the support work. Customer prioritizes work in IPM, but they can change priorities in daily standup also based on need. Typically support problems are unknown at the time of IPM. So customer plays vital role in daily standup meetings. Customer also gets involved in support roles from providing great insight from business.

Documentation and Support

Agile says, create documentation minimal possible. Document what is required, not what can be documented. Support does not have specific needs for documentation, there are no typical documentation stages defined. Support needs some tracking documentation. Support demands for problem tracking system, every support person must update that problem ticket with problem description, and also resolution steps. This makes the support for new persons easier; they can just search in the problem database for problem in hand. If we can embed this practice in daily activities, this can be effectively used as knowledge management tool.

Testing and Support

Support principle says, minimum changes possible to fix a problem , because we are not sure how many things will break with the change. Test driven development is a proven technique to generate quality software. I hate when somebody does not test and tries to push the code from one environment to other. This practice is awfully dangerous for support. Every piece of code has to be tested thoroughly before pushing it. Testing practices are defined for all different needs, like unit testing, regression testing, stress testing, longevity testing, and so on so forth. Actually, every developer has to develop test cases for his program, they ensure the coding changes made in support do not break anything else. Usually, automated tests are required so that, for each changes support engineer can run the TestSuite, and feel comfortable. There are numerous TestFrameworks available for formulating test cases on different platforms and technologies.

Large organizations with separate support and development wings, has quality processes defined for production turnover. These processes does talk about the test results, but they never have requirement for test scripts. Development team has to handover the unit test scripts created by developers, before development. Agile methodologies proclaim test driven development approach to deliver quality software. Test driven Support is also key to Customer Satisfaction in support. Test driven development can make the support as a test driven practice. Off course this can never be a substitute for functional, regression, stress testing required by application complexity, client approval.

What do we do with efficiency gain?

Efficiency gain has to be returned back to client in terms of returning support budgeted cost, or providing value for the entire budget we planned for. Every application is build with some estimation of age. After some time it becomes obsolete on technology or on business use, so we design a new process in agile development methodology for keeping the system upgrades coming in, to keep it working and keep it usable in same team.

This efficiency gain can further be increased by having support engineers to develop better and efficient test cases, that can save time in making code fix in future. Moreover, efficiency gained in support can be returned back b utilizing engineers, to make development towards changing face of software.

Steps to success :
There are no special steps to success, they are same as they are for eXtreme Programming. All of those driven by a single question, what value does this activity brings to customer?".  In next blog, I shall write more about the steps we use to make this a success in a Maintenance shop.

This is all I can gather and write in single writing iteration. Willing to continue again, I will write more in subsequent blogs.

References :

• eXtreme Maintenance
• wiki - wikipedia, most useful and free encyclopedia on Internet.
• Using XP In A Maintenance Environment
• eXtreme Programming Explained by Kent Beck
• Agile and Iterative Development: A Manager's Guide by Craig Larman
• eXtreme Programming