Questions about analyzing search usage... #metrics #search


Murray Jennex
 

I understand why you are doing what you are but I want to give a strong warning first, research (mine included) is showing that the amount of use of a KMS (including searches) is not the best measure.  Two things too ponder: what really matters is the value of the use, just because someone searches frequently doesn't mean they are a power user, they may be retrieving low value or low impact knowledge, while someone that uses the system only once may retrieve something that has tremendous impact/value.  My research showed no correlation between frequency of use and importance of that use.  Second, the more important measure is Intent to Use.  This measure looks at the users to see if they intend to use the system when they have a problem that could be helped by the system.  Intent to Use looks at perceived value, usefullness of he system, and ease of use.  If Intent to Use is high it means your users see the system has value and will use it when appropriate.  Intent to Use will remain high as long as subsequent use yields benefit.  Thanks...murray jennex
 
In a message dated 3/30/2009 2:41:24 P.M. Pacific Daylight Time, cerrone@... writes:

 

To help provide an indicator of engagement with our KM program, we are looking to further analyze usage of the search system integrated with our KM program.

Leveraging the authentication information from our single sign-on environment, we can understand usage down to individuals.  Though our goal is not to analyze search behavior at the individual level, we are interested in looking at a group of users (say a department or a region) and representing their search usage demographics in categories such as "occasional user", "average user", "top users".

Realizing that much of this will depend on the type of content, the type of users, the business, the culture, etc...are there any good pointers to benchmarks and/or studies to help us think through these categorizations of usage (ie, occasional, average, top) as well as what defines them (ie, average is >= 5 searches per week)?

Thanks in advance!
- Dave

Dave Cerrone
Knowledge Management Operations Leader
GE Energy
T  1-518-385-0196
M 1-518-605-6539
cerrone@...
GE imagination at work

 



A Good Credit Score is 700 or Above. See yours in just 2 easy steps!


Dave Cerrone
 

 

To help provide an indicator of engagement with our KM program, we are looking to further analyze usage of the search system integrated with our KM program.

Leveraging the authentication information from our single sign-on environment, we can understand usage down to individuals.  Though our goal is not to analyze search behavior at the individual level, we are interested in looking at a group of users (say a department or a region) and representing their search usage demographics in categories such as "occasional user", "average user", "top users".

Realizing that much of this will depend on the type of content, the type of users, the business, the culture, etc...are there any good pointers to benchmarks and/or studies to help us think through these categorizations of usage (ie, occasional, average, top) as well as what defines them (ie, average is >= 5 searches per week)?

Thanks in advance!
- Dave

Dave Cerrone
Knowledge Management Operations Leader
GE Energy
T  1-518-385-0196
M 1-518-605-6539
cerrone@...
GE imagination at work

 


Dave Cerrone
 

 

Murray - Thanks for the reply.

While I agree that analyzing search trends may not provide an absolute indicator of value to the user, I don't know that I agree it doesn't provide any insights...especially over time.  One might consider that if the trends of usage increases, or at least sustains at a reasonable plateau (relative to the size of the user base), over time, then you might infer that the users are returning because they are finding some level of value from the search.  Worst case, you might infer that the users still a) have an unmet need, and b) have some level of confidence that the search will provide value to them over other options.  If you eliminate these two inferences, it would seem difficult to explain increasing usage, or at least sustained usage, of the search over time.

The "intent to use" measure that you note is interesting, as possible addition to our KM metrics.  What's not clear to me, however, is the methods and approaches to measure this.  Can you provide any more detail and/or examples?

Thanks in advance for any further comments and information.

- Dave

 


--- In sikmleaders@..., murphjen@... wrote:
>
> I understand why you are doing what you are but I want to give a strong
> warning first, research (mine included) is showing that the amount of use of a
> KMS (including searches) is not the best measure. Two things too ponder: what
> really matters is the value of the use, just because someone searches
> frequently doesn't mean they are a power user, they may be retrieving low value or
> low impact knowledge, while someone that uses the system only once may
> retrieve something that has tremendous impact/value. My research showed no
> correlation between frequency of use and importance of that use. Second, the more
> important measure is Intent to Use. This measure looks at the users to see if
> they intend to use the system when they have a problem that could be helped
> by the system. Intent to Use looks at perceived value, usefullness of he
> system, and ease of use. If Intent to Use is high it means your users see the
> system has value and will use it when appropriate. Intent to Use will remain
> high as long as subsequent use yields benefit. Thanks...murray jennex
>
>
> In a message dated 3/30/2009 2:41:24 P.M. Pacific Daylight Time,
> cerrone@... writes:
>
> To help provide an indicator of engagement with our KM program, we are
> looking to further analyze usage of the search system integrated with our KM
> program.
> Leveraging the authentication information from our single sign-on
> environment, we can understand usage down to individuals. Though our goal is not to
> analyze search behavior at the individual level, we are interested in looking
> at a group of users (say a department or a region) and representing their
> search usage demographics in categories such as "occasional user", "average
> user", "top users".
> Realizing that much of this will depend on the type of content, the type of
> users, the business, the culture, etc...are there any good pointers to
> benchmarks and/or studies to help us think through these categorizations of usage
> (ie, occasional, average, top) as well as what defines them (ie, average is >=
> 5 searches per week)?
> Thanks in advance!
> - Dave
> Dave Cerrone
> Knowledge Management Operations Leader
> GE Energy
> T 1-518-385-0196
> M 1-518-605-6539
> _cerrone@..._ (mailto:cerrone@...)
> GE imagination at work
>
>
>
>
>
> **************A Good Credit Score is 700 or Above. See yours in just 2 easy
> steps!
> (http://pr.atwola.com/promoclk/100126575x1220439616x1201372437/aol?redir=http:%2F%2Fwww.freecreditreport.com%2Fpm%2Fdefault.aspx%3Fsc%3D668072%26hmpgID
> %3D62%26bcd%3DfebemailfooterNO62)
>


Murray Jennex
 

I agree that you have to measure use, I'm just cautioning against giving actual use too much credence, it is not the prime metric.  To explain more I'm attaching my article on KM Use as a Measure of Success and also my paper on KM Success Model.  The KM Use paper discusses use as a measure in great detail and includes an evidence based approach to stating why I believe actual use is not the prime metric.  The KM Success Model includes discussion on the Perceived Benefit Model which is what I use to measure Intent to Use.  I use Perceived Benefit Model as it is based on the Theory of Unplanned Action based on the Theory of Reasoned Action
 
In a message dated 3/31/2009 7:38:19 P.M. Pacific Daylight Time, cerrone@... writes:

 

Murray - Thanks for the reply.

While I agree that analyzing search trends may not provide an absolute indicator of value to the user, I don't know that I agree it doesn't provide any insights...especially over time.  One might consider that if the trends of usage increases, or at least sustains at a reasonable plateau (relative to the size of the user base), over time, then you might infer that the users are returning because they are finding some level of value from the search.  Worst case, you might infer that the users still a) have an unmet need, and b) have some level of confidence that the search will provide value to them over other options.  If you eliminate these two inferences, it would seem difficult to explain increasing usage, or at least sustained usage, of the search over time.

The "intent to use" measure that you note is interesting, as possible addition to our KM metrics.  What's not clear to me, however, is the methods and approaches to measure this.  Can you provide any more detail and/or examples?

Thanks in advance for any further comments and information.

- Dave

 


--- In sikmleaders@..., murphjen@... wrote:
>
> I understand why you are doing what you are but I want to give a strong
> warning first, research (mine included) is showing that the amount of use of a
> KMS (including searches) is not the best measure. Two things too ponder: what
> really matters is the value of the use, just because someone searches
> frequently doesn't mean they are a power user, they may be retrieving low value or
> low impact knowledge, while someone that uses the system only once may
> retrieve something that has tremendous impact/value. My research showed no
> correlation between frequency of use and importance of that use. Second, the more
> important measure is Intent to Use. This measure looks at the users to see if
> they intend to use the system when they have a problem that could be helped
> by the system. Intent to Use looks at perceived value, usefullness of he
> system, and ease of use. If Intent to Use is high it means your users see the
> system has value and will use it when appropriate. Intent to Use will remain
> high as long as subsequent use yields benefit. Thanks...murray jennex
>
>
> In a message dated 3/30/2009 2:41:24 P.M. Pacific Daylight Time,
> cerrone@... writes:
>
> To help provide an indicator of engagement with our KM program, we are
> looking to further analyze usage of the search system integrated with our KM
> program.
> Leveraging the authentication information from our single sign-on
> environment, we can understand usage down to individuals. Though our goal is not to
> analyze search behavior at the individual level, we are interested in looking
> at a group of users (say a department or a region) and representing their
> search usage demographics in categories such as "occasional user", "average
> user", "top users".
> Realizing that m uch of this will depend on the type of content, the type of
> users, the business, the culture, etc...are there any good pointers to
> benchmarks and/or studies to help us think through these categorizations of usage
> (ie, occasional, average, top) as well as what defines them (ie, average is >=
> 5 searches per week)?
> Thanks in advance!
> - Dave
> Dave Cerrone
> Knowledge Management Operations Leader
> GE Energy
> T 1-518-385-0196
> M 1-518-605-6539
> _cerrone@..._ (mailto:cerrone@...)
> GE imagination at work
>
>
>
>
>
> **************A Good Credit Score is 700 or Above. See yours in just 2 easy
> steps!
> (http://pr.atwola.com/promoclk/100126575x1220439616x1201372437/aol?redir=http:%2F%2Fwww.freecreditreport.com%2Fpm%2Fdefault.aspx%3Fsc%3D668072%26hmpgID
> %3D62%26bcd%3DfebemailfooterNO62)
>



New Low Prices on Dell Laptops - Starting at $399