Debugging Plan

For a project, we need a bunch of planning documents already:

• Development Plan
• Test Plan

The development plan covers initial project development-- how we will proceed with creating the system? The test plan ideally lists out the actual tests which will be performed. However, there's no plan in place which covers where projects spend most of their time-- debugging. We need a Debugging Plan.

You might object. “Stephen,” you might say (since we're on a first-name basis by this point), “you can't predict what will happen during debugging! Who knows what defects might appear? You might as well have a 'win the lottery plan' while you're at it!”¹ But I would point out that two of your existing documents know exactly where all of your bugs will be, and how they'll be found: your software architecture and your test plan.

Your test plan can tell you how your bugs will be found. A properly-written test plan calls out a list of tests which will be applied to the finished project.² If any of those tests fail, then that's obviously a bug which will require diagnosing and repairing. Basically, a test plan is a list of risks. If you're a manager, and someone hands you a list of things which can go wrong with your project, you're pretty negligent if you don't come up with a plan for dealing with them. However, most software engineering projects think it's enough to come up with a plan for finding bugs, instead of the equally important matter of actually solving them.

Your architecture documents, on the other hand, can tell you where your bugs will be. (If they're not in one of your system components, they're not your bugs!) Now, programmers aren't very good at predicting where bugs will fall. (Other than saying “not in my component.”) However, we don't need to predict that; in fact, we can assume that there will be bugs in each component. What we need is a plan for debugging problems found in each component and in the connections between components. Thinking about the types of debugging which will be required in advance allows you to plan your debugging environment and do some initial prototyping to ensure that your plan will work. Spending some time brainstorming about the kinds of defects which could appear in each component probably helps to avoid them in the first place.

Bringing in QA at this point would be a huge plus, since they frequently have the most
experience in focusing on bugs. Figuring out in advance the kinds of information which will be required to find and remove the bugs will save a great deal of wrangling later. “OK, so if test A on function reverseTheWord fails, I will need to know exactly which word you fed it, plus the state of the WordReversalEngine.” QA can then update the Test Plan procedure to make it easier to get developers the information required to fix the bug.

It's important to keep the debugging plan (or “defect mitigation plan”, or whatever) up-to-date as the project proceeds. Weekly defect reviews should focus on both open bugs and ones recently closed to determine where they were introduced. If the bug was introduced because of a faulty procedure or practice, can developers use that information to predict where future bugs will appear, and fix them proactively? If a component is particularly buggy, developers should take the time to review the design and code for fundamental issues. Is there requirements agreement between the test plan and the actual code? Is there some kind of pervasive misunderstanding? Were the requirements unclear? Were the unit tests adequate, and should they have caught this defect at creation time?

The project manager should keep a record of the code quality of different developers and use it in planning. In my experience, debugging time is relatively fixed as a percentage of total project time for a particular developer. With the assistance of their manager, a developer should be able to experiment with taking more development time for higher initial quality.

In the end, debugging is the most unpredictable part of the software development process, and the place where a project is most likely to stumble into the quicksand of complete failure. It's also the worst place to fail; when we're in the debugging phase, the company has invested substantial time and resources into the project already, and often has promised a release to customers. Any effort to reduce debugging time will be richly rewarded.

(Author's note: after writing this entry, I spoke to an experienced QA manager who told me that he always insists that the development plan contain what I've described here. It doesn't matter to me whether the debugging plan is a stand-alone document or part of the development plan, just so long as it's written, read, and followed.)

1Well, as it happens, I do have a “win the lottery” plan. It involves never entering it in the first place. Or inventing a time machine. Whichever comes first.

2Once SCAV is developed, it'll be “just a view”, like everything else.