Spoken word audits

If what you’d really like to know is what analysts say to clients on the phone or in person, when nobody is listening and vendor clients won’t hear them, Lighthouse’s spoken word audits are the solution.

At firms like Gartner, an analyst must use the firm’s official position as a starting point, but can diverge from it to reach an unexpected conclusion. At others, even this much adherence to the “party line” isn’t required if the analyst feels a user’s situation calls for a different approach. Spoken word audits let you become the “fly on the wall?”

A “spoken word audit,”  uses a vendor client’s analyst inquiry time to “play user” with analysts to see how they respond, what they recommend in scenarios you define ahead of time.

To do a spoken word audit, Lighthouse’s approach rests on a four-step method.

• A. Define the scenarios clearly. A brief description isn’t enough. You need that to give the analyst as the conversation starts, of course, but work it out at least one level deeper to respond to clarification questions without fumbling or making up things that don’t make sense. You may need help from your internal subject matter experts to do this.
• B. Develop a structured form for recording answers. For each competitor, we record points in favor, points against, which points matter in this scenario, why they matter, and the overall conclusion. If you’re looking for how well the analyst understands your positioning, independently of who the other vendors might be, we put down specific points you’d like the analyst to reflect and what he or she should say about them. The Lighthouse approach also notes if the analyst brought them up without prompting, mentioned them after a general hint, discussed them in response to a direct question or didn’t know about them at all.
• C. Do the first audit to develop a baseline. The information is important, to be sure, but what’s really important will be how this information changes over time. If you don’t expect to be able to do a second audit, the value of the first is reduced.
• D. Repeat the process six to twelve months later with the same firms, whenever possible with the same individual analysts. Less than six months is too short an interval for anything you do to have an impact. More than twelve can let problems go on too long without corrective action.

As a side benefit, even if your measurement-hungry management doesn’t see meaningful results from the first audit, they’ll see that you appreciate the value of measurements. That’s likely to keep them satisfied for a while and to prevent them from coming up with their own, less suitable, ways to measure what you do.

To find out more, contact us to arrange a discussion with Efrem Mallach, our research fellow.