Adoption Rate #metrics


jacobwatts <jacob.watts@...>
 

Hello, everyone. As a silent member for a few months now I thought
I'd briefly introduce myself (and thereby my situation) before posing
my first question to the community.

I am a senior program manager for Cisco WebEx (formerly WebEx
Communications) within the account management (service)
organization. Internally I am referred to as the Knowledge Base
Manager, and I am responsible for the success of our organization's
implementation of a knowledge base. The implementation began just
over 2 years ago, and went live around mid-2007. The KB is used for
process documentation as well as typical product-related content
(usability questions and problem resolution).

One of the measurements I look at to evaluate usage (and consider a
barometer of success) is total unique users per month. If people
don't find what they're looking for, they don't come back. If they
continue to return, I feel we can safely conclude that it ~is~
working for them more often than not, so a positive trend in this
area indicates a successful implementation. (When I say successful
implementation, I mean "we're on the right track," not "we're
done.") Here's where my question comes from: through reading the
threads in this forum I ran into Andrew Gent's blog, where he
states "Frequently, an adoption rate as low as 20% is considered
success for a KM initiative."
(http://incrediblydull.blogspot.com/2008/09/alternatives-to-
collaboration.html) Not to dispute that this is his experience, but
this does seem extremely low to me – although it does sound like he
is referring more towards community-based collaboration efforts than
a systemic implementation like mine. Regardless, it did raise again
the questions in my mind which I thought would be best answered by
this group:

With a KM systems implementation (whether KB or other CMS), what
adoption rate do you consider a success? What adoption rates have
you seen in your experience?

The assumption here is that we agree on how I'm measuring the
adoption rate, which could be a conversation all on its own. Thanks
in advance for any insight you have on this subject.

-Jacob Watts


Andrew Gent
 

Hi Jacob,

Great question. Unfortunately, I think the answer is the inevitable "it depends".

What does it depend on? It depends on what your target audience is and what their expected usage pattern would be. While I was at HP, our monthly usage (i.e. unique users) for SharePoint reached 100K (that was 2/3 of the total company population) before we stopped counting. Our original target had been 50% of our specific organization a month.

On the other hand, our target for a catalog of past projects was around 1 or 2 thousand users a month.

Why the difference? We expect SharePoint users (for self-serve collaboration spaces) to use the space at least once a week -- in many cases it is actually more than once a day. We also expected up to 80% of the organization to participate in projects. The project catalog, on the other hand, is usually used only when people are starting projects and looking for previous examples or when they get stuck and are looking for assistance from people with similar experience. Some fuzzy logic about how many new projects started a year and who was most likely to need the information (project managers, architects, and project leads), we came up with a target of 10-15% of the organization per month.

So in your case, it would depend on who and how often you expect people to need the documentation.

--Andrew Gent

P.S. My example of 20% was from a few collaboration technology implementations I knew of.


From: jacobwatts
To: sikmleaders@...
Sent: Thursday, January 8, 2009 1:09:25 PM
Subject: [sikmleaders] Adoption Rate

Hello, everyone. As a silent member for a few months now I thought
I'd briefly introduce myself (and thereby my situation) before posing
my first question to the community.

I am a senior program manager for Cisco WebEx (formerly WebEx
Communications) within the account management (service)
organization. Internally I am referred to as the Knowledge Base
Manager, and I am responsible for the success of our organization' s
implementation of a knowledge base. The implementation began just
over 2 years ago, and went live around mid-2007. The KB is used for
process documentation as well as typical product-related content
(usability questions and problem resolution).

One of the measurements I look at to evaluate usage (and consider a
barometer of success) is total unique users per month. If people
don't find what they're looking for, they don't come back. If they
continue to return, I feel we can safely conclude that it ~is~
working for them more often than not, so a positive trend in this
area indicates a successful implementation. (When I say successful
implementation, I mean "we're on the right track," not "we're
done.") Here's where my question comes from: through reading the
threads in this forum I ran into Andrew Gent's blog, where he
states "Frequently, an adoption rate as low as 20% is considered
success for a KM initiative."
(http://incrediblydu ll.blogspot. com/2008/ 09/alternatives- to-
collaboration. html) Not to dispute that this is his experience, but
this does seem extremely low to me – although it does sound like he
is referring more towards community-based collaboration efforts than
a systemic implementation like mine. Regardless, it did raise again
the questions in my mind which I thought would be best answered by
this group:

With a KM systems implementation (whether KB or other CMS), what
adoption rate do you consider a success? What adoption rates have
you seen in your experience?

The assumption here is that we agree on how I'm measuring the
adoption rate, which could be a conversation all on its own. Thanks
in advance for any insight you have on this subject.

-Jacob Watts



john_mcquary <john.mcquary@...>
 

At Fluor, we measure our knowledge communities as being successful if
they are delivering results on their objectives, however we do maintain
numerous statistics including unique users per month. In any typical
month, about 55% of our 28,000+ members will go into the system at
least once. Over any three month period we see just over 70% unique
logins. As of Jan 8 2009 10:39AM PST, there have been 26.2% unique
users this month.

- John


jacobwatts <jacob.watts@...>
 

Thank you for your response Andrew. Your stat for a community
implementation makes more sense. Now that I'm thinking about it, we
do have a grassroots CoP with about a 20% membership rate and
probably half of them are regularly active in the community.

For our implementation, I believe we should be at a 95%+ adoption
rate on a monthly basis, as we operate in an extremely complex
environment. In addition to the volume of processes we have
internally and with other internal organizations, we offer a wide
variety of products that are on various platforms, making it
impossible for anyone to know everything WebEx, regardless of
tenure. I can't imagine that any individual could go an entire month
without having at least one opportunity to use the system.
Realistically we could probably say our organization should be at a
95% adoption rate on a ~weekly basis, but changing the everyday
workflow and habits of a largely tenured organization doesn't come
easy, so at the moment a monthly measurement seems to make sense.
Our last 9 months we've averaged over 70% adoption, which I
consider 'not bad' under the circumstances, but would like to see
improved upon. From a management perspective I've asked managers not
to focus on enforcing usage through metrics to avoid turning the
system into a button-mashing contest, so I feel the adoption we are
seeing is genuine.

I am definitely interested in hearing other experiences on the
subject, and would like to see what kind of consensus the group comes
to.


Murray Jennex
 

My colleagues and I have been researching KM success for several years with the express purpose of identifying:
 
a success model
a set of critical success factors
a definition of KM success
a set of KM success measures
a set of KM success KPIs
 
I will be happy to share our findings with the group in a few weeks but will first ask for a short confirmation survey (we are finalizing our results and I would like to see what this group thinks of them so want your opinion before sharing all the docs - this is so we don't bias you, not because we are trying to entice you)
 
I did want to make one comment on use measures:  I don't think they mean much as long as you can show that your KM/KMS is getting some use.  What I have found to be a much better predictor of KM success is the intent to use of our knowledge workers.  What intent to use means is that the knowledge worker intends to go to the system when they have a question.  Why this is a good predictor is due to the use of the Perceived Benefit Model based off of the Theory of Unplanned Action (and what this all really means is that the organization and the knowledge worker see value in the system so intend to use when its needed).  I've published these results in the article citation below (I can send you a file if you can't find it if you let me know):
 

Jennex, M.E., (2008).  “Exploring System Use as a Measure of Knowledge Management Success,” Journal of Organizational and End User Computing, 20(1), pp. 50-63.

 
The bottom line is amount of actual use doesn't mean much, the quality of use and using it when appropriate are much better indicators of success and Perceived Benefit is a way of measuring if a user will use the system when appropriate.
 
Thanks...murray jennex
 
Murray E. Jennex, Ph.D., P.E., CISSP
San Diego State University
Editor in Chief International Journal of Knowledge Management
Co-editor in Chief International Journal of Information Systems for Crisis Response and Management
 
 
In a message dated 1/8/2009 10:52:56 A.M. Pacific Standard Time, john.mcquary@... writes:
At Fluor, we measure our knowledge communities as being successful if
they are delivering results on their objectives, however we do maintain
numerous statistics including unique users per month.  In any typical
month, about 55% of our 28,000+ members will go into the system at
least once.  Over any three month period we see just over 70% unique
logins.  As of Jan 8 2009 10:39AM PST, there have been 26.2% unique
users this month.

- John


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/sikmleaders/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/sikmleaders/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:sikmleaders-digest@...
    mailto:sikmleaders-fullfeatured@...

<*> To unsubscribe from this group, send an email to:
    sikmleaders-unsubscribe@...

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/