MoveinBlue tech

Dec 18 2012 ∞

Reporting Problems, Part 2: Tooling

(Image source: unknown.)

In the first part of this series we saw an introduction to user problems. In this second and last part we will give a few guidelines for creating a tool to manage a large number of problems.

The Perfect Tool

A reporting tool should report user problems, whatever they are. It should not assume that everything is a “bug unless proved otherwise”. It shouldn’t either put the burden on the reporter that there is a problem: it is too easy to ask the user for confirmation by providing infinite details. Instead, such a tool should be completely agnostic and just let users state their problems.

Some problems may be caused by a missing feature in the software. It is a good practice to let users request new features using the reporting tool, since they are supposed to solve user problems, after all. It is important to be able to mark those problems as “missing features” so the user knows what to expect.

Process Issues

By far, the most important event in the life of a report is the triaging: the report manager has to decide if the report should be taken care of now or later, and assign someone to work on it when the right moment comes. A blocking problem (something which prevents people from doing their job properly) would be given the maximum priority, while cosmetic issues will probably be relegated to valleys of inactivity.

It is important to keep everyone informed about what happens with the report. And by “everyone” we mean “everyone involved, and anyone who wants to keep informed”; their job will probably be affected by the problem. So it is a good idea to keep a notification list (where people may add or remove their address) and use it to mail status updates.

By the way, people involved in the report should be able to add comments to it: the developer may want to ask more information, and the reporter will probably be eager to provide it. Keeping a list of comments associated with the report is also a good idea.

Reversibility is, as always, very important. Any open report can be closed and viceversa.

The Missing Fields

Bug trackers have usually lots of fields. The bugzilla demo has seven fields in its basic interface, and at least 20 fields in the advanced mode (more can be added). Problems in bug management can always be solved by more fields, or so the team imagines.

In the real world reports get usually bloated very quick; the administrative burden for managing them increases with each new field. It is often better to keep optional information in free text fields.

There should be three distinct areas in each report: one for the reporter, another for the triager and a third for the assigned developer. All of them should be visible if someone is interested (and has the necessary clearance in matters of security), but they can be collapsed by default to simplify the user experience.

Priority, Severity and Urgency

The report manager has to decide when a problem will be dealt with, and to do it he or she needs to set a priority. The user should never have the last word, as all users will mark everything as “priority -1000” if needed, even if it is in fact a trivial issue.

It is very easy to get lost in a lot of different concepts that quantify how many people are affected, how severe the problem is and when it will be dealt with. But in fact just one field is enough: an approximate priority with an approximate value of high, medium or low is usually enough.

Prioritization should be done using a different tool, which is just a queue of work given to someone. There is no need to use high, medium or low priorities; just state which problem is the first to be tackled and those that go next. The ideal queue will get a developer busy for a few days before they come back for more; if they get stuck in one problem they can start with the second, and so on.

Traceability

Usually we will keep our code in a repository. While we cannot keep our reports also in the same repo, there are elaborate systems that keep track of which code changes solve which issues, and viceversa.

Unfortunately it is not that easy. Reports can correlate with code changes, what we call “commits”: changes that have been committed (or uploaded) to the repository, and viceversa. But this means that developers must enter manually a commit identifier in every report that resulted in code changes, and a report identifier in every commit that is the result of a report. Lots of work for the eventual case where we may need to look up which commit came with which report.

Wouldn’t it be great to keep reports in the same place as the code, then? This approach has been tried and failed multiple times. Reports are just not easy to store in repos.

All this work may not even be convenient. In my (admittedly limited) experience it is enough to state succintly which report is solved by which commit, so it can be searched later.

Reporting Software Problems, Part 1: Process

(Image source: Popular science monthly.)

Software has bugs; this is a widely known secret about software development. How do we deal with this? The first step is to open communication channels between users and developers. In this first part I will look into how problems should be reported; in a second part I will discuss the principles for a good problem reporting tool.

Managing Bugs

A malfunction in a software system is found; what we usually call a bug. The developers should know about it and probably correct it; what do we do with it? Our first idea is to report it to the team in person, but they may reply: “Not so fast! There are procedures for that.” Why so much arrogance? Simple: in a medium-to-large project there are probably tens or hundreds of malfunctions, people tend to report the same issue multiple times, and the development team has to manage all that information. Not all bugs can be corrected immediately: some are more important than others, and some can even be postponed as they cause only minor visual problems.

That is why bug trackers were invented: they allowed developers to keep track of every individual malfunction, including: who reported it, what developer is responsible for it and in what state it is at the moment. They are the perfect tool for a group of specialized testers and the development team to communicate; and they help management evaluate the situation.

Mozilla developers, knowing the problem of releasing software to the public, created a tool called Bugzilla. Proprietary software developers had long had ticketing systems, change request managers and other tracking tools, but this was the first major bug tracking system released as Free software.

But before we even reach the state of bug tracking, there is an issue to solve: what is a bug? To the software developer a “bug” is a mistake introduced in the code of a software system. There are other terms in use: error, defect, crash. But we are not going to fight about words, are we?

Issues, Problems and Bugs

It is not easy to make bug reporters communicate with people responsible of fixing bugs. First there is a language barrier: one is supposedly a domain expert (meaning that they know what to do with the software), while the other is a software developer, and they do not necessarily share a common language to describe problems. What looks like wrong output to one might be a problem with the requirements to the other; what one identifies as a plain bug, the other may discard as simply a misconfiguration.

A common obstacle when reporting problems in software is miscommunication between the reporter and the software developer. The obstacle often arises because non-technical people know our little secret: that software has bugs. So they tend to attribute all of their problems in life to software bugs, when to the developer they may be caused by many other things.

Note that a bug is only considered as such if the software doesn’t do what it is supposed to do; but different people may have different ideas about it. Communication is not always perfect; any mistake in the specification (if someone took the trouble to write it down) will look like a software bug to the user, but will just be a wrong requirement to the developer. At other times features are not even implemented, so they won’t work; but the engineer will refuse to acknowledge a bug in code that hasn’t been written.

But you know what? The customer is always right, and the user is our customer. If any user takes the trouble to report a problem, then there probably is a problem, at least for that particular user. It might be ignorance by the user on how the software works; an error in the specification, or a missing feature; misconfiguration of the system or requirements not met; or perhaps a good old software bug. But whatever the cause, any reported problem should be taken seriously and solved if possible.

Solving Problems

How to solve the problem depends a lot of what the problem is, obviously. But we can outline a process to manage the report: to process it until it is solved.

First, someone needs to take responsibility for the report; we will call this figure the ”report manager”, although it doesn’t need to be a figure of authority. The problem may be diverted to user support to be identified, and hopefully solved by teaching the reporter how to use the software. Otherwise the report passes to the next stage.

As we have seen the problem may be caused by many different things: requirements, configuration, user documentation, etcetera. The manager must decide if the problem can be solved by systems admins, requirements engineers or any other team, and refer the report to them. All the possibilities must be discarded first; some companies have elaborate escalation procedures, others just let the report manager decide. Several layers of support are common once an organization grows above a certain level. Whatever workflow is most appropriate in your situation is fine; just be sure to think about it and improve it if needed.

Finally, if nobody else has found the cause the development team has to take care of the report. If the problem really is caused by a development product (a software bug, or an error in the manuals perhaps) then it should eventually be assigned to someone who can solve it.

Types of Problems

When we have more than a handful of reports we will probably want to divide them in groups. One obvious way is to identify and remove duplicates. Apart from leaving just one report for each problem, the most significant categories all refer to the underlying problem.

Keep in mind that “problems” should refer to repeatable conditions, not one-time faults. Specifying that “I could not access the system from 2:00 to 4:00 last night” is no good, since there is no way to reproduce the problem and find the root cause.

So, how can we categorize reported problems? The most productive types divide problems into groups according to their origin as we saw above: software bug, documentation issue, configuration error, and so on. Within software bugs it can be useful to distinguish between a malfunction and a visual problem, for instance if you are dealing with a web application where the HTML layout can be botched up but be completely functional.

There are a couple of special types often thrown into the mix: a “missing feature” is something which has not been implemented and should, while “wish list” identifies a desirable feature which has not been planned yet for inclusion.

There are many other ways to categorize bugs: severity, priority, number of users affected, subsystem where the problem arises, etcetera. Users should be able to enter those values that make sense to them; the rest should be entered by the report manager. As we saw before users may be challenged to find the correct category for a problem, e.g. mistaking missing features as bugs; the report manager should review the problem type assigned by users, if any.