Re: Enterprise search - defining standard measures and a universal KPI #metrics #search


Tom Barfield
 

Lee - here are three of my favorite (and imperfect) approaches to measuring search. I agree with other thoughts in the thread - it is difficult to parse out the cause/effect on metrics - influence of the search engine vs. content quality, alignment between user needs and the scope of the search..

Average time spent in search for a session
This is imperfect in that we don't always know when a session ends or if the end user is performing multiple searches. Frequently, users don't close the search window - causing the timer to run until the server times out. Still, I find there is value here - particularly if you can see changes over time.  For instance, when we implemented a slide preview capability we found that the total time spent in search reduced by about 20%.

Average result clicked
The closer this gets to 1 the better.  In one situation, over the course of about a year we reduced this from an average of 16 to 8. Our end users were more likely to stay on the first page of results. The imperfection - some users use search to gather - and are happy to go digging deep into results.  One person going down to page 100 and clicking a result could influence this average.  BTW - another impact of the search preview capability mentioned above was that the average result click was reduced.  Our hypothesis (never fully tested) was that most users make decisions based on the title of the document.  With the search preview displayed they had more to see that caused the user to click on items that they would have otherwise missed. 

End user Smiley/sad face rating
We started collecting these and focused on the poor ratings.  We collected the search string and items clicked on associated with the rating.  This enabled us to do research to determine if it was a content issue or search engine issue.  Over time we saw the number of sad face ratings reduced.

There are no standard targets with any of these.  For us it was a matter of measuring, taking action, measuring to see some sort of change.  If we saw changes in the metrics when we weren't actively making improvements, we would do some research to understand potential causes of the changes in the metrics.

All in all, this feels far more art than science.

Tom

Join main@SIKM.groups.io to automatically receive all group messages.