About a million years ago, I wrote my doctoral dissertation about how to talk to computer system users. I videotaped 50 systems analysts talking to 50 different real estate brokers discussing the requirements for a system. I did formal protocol analysis on each of the tapes. Then I started testing a bunch of hypotheses:
The syntax of the question does not matter.
Most conventional wisdom is to ask open-ended questions since these provide better feedback. This “wisdom” is false. All of the evidence for that advice comes from “adversarial interviews” such as police or legal interrogations. Since users want to tell the analyst what they would like to see in the system being built, the questions simply open up topics for discussion.
The more feedback you give a user, the more information they will give you.
Every time a user tells you something, parrot it back to make sure they feel understood. Every few minutes you should have a whole part of the conversation that starts with “What I hear you saying is that…”
People-oriented professionals are better analysts.
There are many psychology inventories that attempt to measure this phenomenon. I gave several of them to the participants. Analysts who are more “people-oriented” get more information.
But none of these factors really explained the HUGE differences between good analysts and poor ones. A good analyst will extract about ten times as much information from a user in the same amount of time as a poor analyst. After I graduated, I got some really good advice from Dr. Iris Vessey when we were both teaching at Penn State. She told me to go home and watch all of the tapes to see if I could discern what made the good analysts different. That’s when I saw the big difference between a good analyst and a poor one.
Good analysts come to the table with an organized set of topics in their heads. Think of it as a tree of topics and subtopics. The analyst traverses that tree asking questions about each topic and subtopic. The tree dynamically changes and expands over the course of the interview.
At the end of the exploration of each topic, good analysts ask: “Is there anything else you want to say about X?” After they hear about several subtopics, they ask if there are more, : “You told me that what is really important about this topic is X1, X2, X3 and X4. Is there anything else we need to keep in mind?”
You can think of the first kind of question as a “vertical terminator” of the tree navigation and the second kind of question as a “horizontal terminator”.
At the end of the process, the analyst has constructed a very detailed tree of information.
EVERY really good analyst used this technique. When I talked to them about it, some were aware of what they were doing, but most just looked at me like I was dense and said something like… “Well how else would you do it?” It just came so naturally to them that they didn’t even think about it.
I provided training about these discoveries to groups of analysts for a while. It is a funny thing to train people to do. The really good analysts already do it naturally, so they think that I am totally wasting their time. The really bad analysts feel as though I am trying to make them do something that is artificial.
I would think that the prospect of doing your job ten times more efficiently would be a good thing… Maybe they all bill hourly.
Maybe they shouldn’t be analysts.