KM Metrics Needed #value #metrics


Neil Olonoff
 

Hi all,

I've just completed an interim version of my KM strategy plan for Army
Medicine, and need some industry metrics to benchmark against. If
anyone has a source or link to KM metrics (for example, Community of
Practice support, content management, best practices, collaboration)
please let me know.

In addition, if anyone in government would like to read and review the
plan, please drop me an email.

thanks very much,


Neil Olonoff


Stan Garfield
 

I suggest starting by identifying three key objectives and then measuring how well they are being achieved.  See "Identifying KM objectives" at https://sites.google.com/site/stangarfield/IK_May07_Masterclass_Identifyingobje.pdf


Metrics and reporting: capturing operational indicators and producing reports to communicate performance against goals, areas for improvement, and progress toward the desired state


There is a wide spectrum of opinion about the importance of measuring knowledge management activities.  Some believe that it is essential, and want to collect data and create reports on a long list of detailed metrics.  Others believe that this is a waste of time, and that effort is better spent on enabling knowledge flow.

Three different kinds of KM metrics are typically captured and reported.  Goal-oriented measurements which directly relate to employee goals allow assessment against those goals.  Operational metrics are based on data captured by KM systems, and include such details as web page hits, uploads, and downloads; threaded discussion subscribers, posts, and replies; repository submissions, searches, and retrievals; and document ratings, wiki entries, and blog posts.  Business impact metrics attempt to determine the return on investment (ROI) of KM initiatives, and include costs saved, costs avoided, incremental revenue, improved quality, increased customer satisfaction and retention, new business attracted, increased market share, revenue from innovations, and revenue from inventions.

 

Collecting and reporting on goal-oriented measurements ensures that the organization is aware of how it is performing and that individuals can be held accountable for achieving their goals.  Reports should be produced and distributed every month to track progress, reinforce good performance, and encourage improvements where needed.  Reporting metrics by group within the organization, for example, regions of the world or countries within a region, allows each group to compare its performance against other groups, and create a friendly competition to excel.  Reporting metrics by individual may be limited by data privacy laws, and if allowed, transmitted confidentially to the manager for use in performance coaching and appraisals.

 

Operational metrics can be helpful in analyzing how the KM infrastructure is being used, who is using it, and identifying areas for improvement.  However, there is only so much which can be inferred from page hits, uploads, and downloads.  These metrics don't indicate the value of any of these activities.  If a user visits a web page, they may not find what they need there.  If a document is uploaded, it may not be of any use to anyone.  If a document is downloaded, it may not be reused.  Follow these rules when deciding on which operational metrics to collect and report:  Keep the time and effort required to a minimum, automating as much of the collection, extraction, and production as possible.  Ask your KM team about which metrics will help them the most.  Focus on a few key metrics which relate to your overall objectives.  Use the metrics to improve the KM environment and test for this in user surveys.  Communicate the metrics regularly so that they influence behavior.

 

Business impact metrics are potentially useful in justifying the expense of a KM program, in garnering management support, and in communicating the value of spending time on KM activities.  Anecdotes and success stories can be collected and converted into numerical values.  Data can be captured in contribution repositories and incentive points systems about the value of learning, sharing, reusing, collaborating, and innovating.  Processes can be created or modified to ask participants about the business impact of KM tasks.  But there are few definitive ways to prove that a particular business indicator was solely influenced by KM.  There are usually multiple reasons for a specific business result, and KM may be one of those reasons.

 

Some firms have conducted one-time surveys to prove the case for KM.  For example, Caterpillar commissioned a one-time study (http://www.kmdynamics.com/newspowers.html ) by an independent consulting firm to identify the benefits and ROI for two established communities of practice: Joints and Fasteners and Dealer Service Training.  The results were:

  • Qualitative ROI:  Productivity (up 40%), Cost (reduced 25%), Speed (up 15%), Quality (up 4%)
  • Tangible ROI:  200% for internal CoPs; 700% for external CoPs

Based on these results, the Caterpillar KM program was justified, and has been supported ever since.  There is no need for ongoing collection and reporting of ROI, since it has been done once.

 

If there is a way for you to collect business impact metrics, then do so.  They have more significance than operational metrics.  But follow the same guidelines about limiting the effort involved to a reasonable amount.

 

Collecting and reporting on the measurements used in your KM program will help you to communicate progress, motivate people to improve their performance, and reassure management of the value of the initiative.  Keep the effort required to do so in the right balance with other projects, look for ways to continue to streamline the process, and review the reporting process annually to keep it relevant to the current state.

 

Also see:

From the Communities Manifesto https://docs.google.com/View?id=ddj598qm_44fx54rbg5

 

Goals should be set for communities and progress against those goals should be measured and reported.  Unhealthy communities should either be nurtured back to health or retired.  There are five ways to measure the success of a communities program: PATCH - Participation, Anecdotes, Tools, Coverage, Health

 

  1. Participation: % of target population which is a member of at least one community
  2. Anecdotes: % of communities displaying the following on their sites:
    • Testimonials by community members on the value of participation
    • Stories about the usefulness of the community
    • Posts thanking other members for their help
  3. Tools: % of communities having all five key tools (see below)
  4. Coverage: % of desired topics covered by at least one community
  5. Health: % of communities meeting these criteria:
    • At least one post to a threaded discussion board per week
    • At least one newsletter or blog post per month
    • At least one conference call, webinar, or face-to-face meeting per quarter
    • At least 50 members
    • At least 10 members participating in each event


Neil Olonoff
 

Thanks Stan - 

Great stuff!

Neil 


Neil Olonoff 




On Wed, Oct 19, 2011 at 10:23 PM, StanGarfield <stangarfield@...> wrote:
 

I suggest starting by identifying three key objectives and then measuring how well they are being achieved.  See "Identifying KM objectives" at https://sites.google.com/site/stangarfield/IK_May07_Masterclass_Identifyingobje.pdf


Metrics and reporting: capturing operational indicators and producing reports to communicate performance against goals, areas for improvement, and progress toward the desired state


There is a wide spectrum of opinion about the importance of measuring knowledge management activities.  Some believe that it is essential, and want to collect data and create reports on a long list of detailed metrics.  Others believe that this is a waste of time, and that effort is better spent on enabling knowledge flow.

Three different kinds of KM metrics are typically captured and reported.  Goal-oriented measurements which directly relate to employee goals allow assessment against those goals.  Operational metrics are based on data captured by KM systems, and include such details as web page hits, uploads, and downloads; threaded discussion subscribers, posts, and replies; repository submissions, searches, and retrievals; and document ratings, wiki entries, and blog posts.  Business impact metrics attempt to determine the return on investment (ROI) of KM initiatives, and include costs saved, costs avoided, incremental revenue, improved quality, increased customer satisfaction and retention, new business attracted, increased market share, revenue from innovations, and revenue from inventions.

 

Collecting and reporting on goal-oriented measurements ensures that the organization is aware of how it is performing and that individuals can be held accountable for achieving their goals.  Reports should be produced and distributed every month to track progress, reinforce good performance, and encourage improvements where needed.  Reporting metrics by group within the organization, for example, regions of the world or countries within a region, allows each group to compare its performance against other groups, and create a friendly competition to excel.  Reporting metrics by individual may be limited by data privacy laws, and if allowed, transmitted confidentially to the manager for use in performance coaching and appraisals.

 

Operational metrics can be helpful in analyzing how the KM infrastructure is being used, who is using it, and identifying areas for improvement.  However, there is only so much which can be inferred from page hits, uploads, and downloads.  These metrics don't indicate the value of any of these activities.  If a user visits a web page, they may not find what they need there.  If a document is uploaded, it may not be of any use to anyone.  If a document is downloaded, it may not be reused.  Follow these rules when deciding on which operational metrics to collect and report:  Keep the time and effort required to a minimum, automating as much of the collection, extraction, and production as possible.  Ask your KM team about which metrics will help them the most.  Focus on a few key metrics which relate to your overall objectives.  Use the metrics to improve the KM environment and test for this in user surveys.  Communicate the metrics regularly so that they influence behavior.

 

Business impact metrics are potentially useful in justifying the expense of a KM program, in garnering management support, and in communicating the value of spending time on KM activities.  Anecdotes and success stories can be collected and converted into numerical values.  Data can be captured in contribution repositories and incentive points systems about the value of learning, sharing, reusing, collaborating, and innovating.  Processes can be created or modified to ask participants about the business impact of KM tasks.  But there are few definitive ways to prove that a particular business indicator was solely influenced by KM.  There are usually multiple reasons for a specific business result, and KM may be one of those reasons.

 

Some firms have conducted one-time surveys to prove the case for KM.  For example, Caterpillar commissioned a one-time study (http://www.kmdynamics.com/newspowers.html ) by an independent consulting firm to identify the benefits and ROI for two established communities of practice: Joints and Fasteners and Dealer Service Training.  The results were:

  • Qualitative ROI:  Productivity (up 40%), Cost (reduced 25%), Speed (up 15%), Quality (up 4%)
  • Tangible ROI:  200% for internal CoPs; 700% for external CoPs

Based on these results, the Caterpillar KM program was justified, and has been supported ever since.  There is no need for ongoing collection and reporting of ROI, since it has been done once.

 

If there is a way for you to collect business impact metrics, then do so.  They have more significance than operational metrics.  But follow the same guidelines about limiting the effort involved to a reasonable amount.

 

Collecting and reporting on the measurements used in your KM program will help you to communicate progress, motivate people to improve their performance, and reassure management of the value of the initiative.  Keep the effort required to do so in the right balance with other projects, look for ways to continue to streamline the process, and review the reporting process annually to keep it relevant to the current state.

 

Also see:

From the Communities Manifesto https://docs.google.com/View?id=ddj598qm_44fx54rbg5

 

Goals should be set for communities and progress against those goals should be measured and reported.  Unhealthy communities should either be nurtured back to health or retired.  There are five ways to measure the success of a communities program: PATCH - Participation, Anecdotes, Tools, Coverage, Health

 

  1. Participation: % of target population which is a member of at least one community
  2. Anecdotes: % of communities displaying the following on their sites:
    • Testimonials by community members on the value of participation
    • Stories about the usefulness of the community
    • Posts thanking other members for their help
  3. Tools: % of communities having all five key tools (see below)
  4. Coverage: % of desired topics covered by at least one community
  5. Health: % of communities meeting these criteria:
    • At least one post to a threaded discussion board per week
    • At least one newsletter or blog post per month
    • At least one conference call, webinar, or face-to-face meeting per quarter
    • At least 50 members
    • At least 10 members participating in each event



Greg <gregory.reid@...>
 

HI Neil,

Metics can be broken into a few areas:

o Enterprise metrics for KM. High level metrics where KM supports critical processes. Examples of this are ROI, Market Share, etc. These can be difficult to define because so many other variables go into them. Also a bit difficult to get past the CFO.

o Process metrics for KM. These are the metrics where you see some value from KM, meeting specific business, process or organization objectives. For example, KM for call centers reduces average handling time by 30 seconds (which is a big number when you 25,000 calls per day...)

o Management metrics for KM. These are the ones that the KM group manages to. For example, recent/valuable contributions, number of readers, Search engin accuracy, etc. There are a number of these.

We regularly do this for clients and I have a number of items that could be helpful to you. If you're interested, give me a call. 617.744.9504

Good luck!

Greg

--- In sikmleaders@yahoogroups.com, Neil Olonoff <olonoff@...> wrote:

Hi all,

I've just completed an interim version of my KM strategy plan for Army
Medicine, and need some industry metrics to benchmark against. If
anyone has a source or link to KM metrics (for example, Community of
Practice support, content management, best practices, collaboration)
please let me know.

In addition, if anyone in government would like to read and review the
plan, please drop me an email.

thanks very much,


Neil Olonoff


Tom Short <tman9999@...>
 

Neil -

Here is a good article highlighting some metrics and approaches at Accenture:
http://www.accenture.com/us-en/Pages/insight-cio-accenture-employee-collaboration.aspx?mkt_tok=3RkMMJWWfF9wsRolua7BZKXonjHpfsX56ukoXqOg38431UFwdcjKPmjr1YEJTsd0dvycMRAVFZl5nQJdDfOSfZRP6A%3D%3D

I have developed a variety of different scorecards/measurement frameworks for various KM and related initiatives. One of my favorite metrics for CoPs is based on social network analysis (SNA). It's a longer-term metric that is based on measuring the change in density and scope of connections within a CoP. You use SNA to create a baseline for a CoP, then conduct an SNA again six months to a year later. Changes in density, strength scores, and breadth become possible indicators of CoP effectiveness/activity. As a control you can go a step further and conduct an SNA for a similar group of employees who are not engaged in a CoP.

Good luck with your effort - are you going to share what you learn??

-Tom

--- In sikmleaders@yahoogroups.com, Neil Olonoff <olonoff@...> wrote:

Hi all,

I've just completed an interim version of my KM strategy plan for Army
Medicine, and need some industry metrics to benchmark against. If
anyone has a source or link to KM metrics (for example, Community of
Practice support, content management, best practices, collaboration)
please let me know.

In addition, if anyone in government would like to read and review the
plan, please drop me an email.

thanks very much,


Neil Olonoff


Neil Olonoff
 

Thanks Tom, 

good stuff. Will I share what I learn? 

Well, sure! 

regards, 

Neil 


Neil Olonoff 




On Thu, Oct 20, 2011 at 11:13 AM, Tom Short <tman9999@...> wrote:
 

Neil -

Here is a good article highlighting some metrics and approaches at Accenture:
http://www.accenture.com/us-en/Pages/insight-cio-accenture-employee-collaboration.aspx?mkt_tok=3RkMMJWWfF9wsRolua7BZKXonjHpfsX56ukoXqOg38431UFwdcjKPmjr1YEJTsd0dvycMRAVFZl5nQJdDfOSfZRP6A%3D%3D

I have developed a variety of different scorecards/measurement frameworks for various KM and related initiatives. One of my favorite metrics for CoPs is based on social network analysis (SNA). It's a longer-term metric that is based on measuring the change in density and scope of connections within a CoP. You use SNA to create a baseline for a CoP, then conduct an SNA again six months to a year later. Changes in density, strength scores, and breadth become possible indicators of CoP effectiveness/activity. As a control you can go a step further and conduct an SNA for a similar group of employees who are not engaged in a CoP.

Good luck with your effort - are you going to share what you learn??

-Tom



--- In sikmleaders@..., Neil Olonoff >
> Hi all,
>
> I've just completed an interim version of my KM strategy plan for Army
> Medicine, and need some industry metrics to benchmark against. If
> anyone has a source or link to KM metrics (for example, Community of
> Practice support, content management, best practices, collaboration)
> please let me know.
>
> In addition, if anyone in government would like to read and review the
> plan, please drop me an email.
>
> thanks very much,
>
>
> Neil Olonoff
>



katepugh <katepugh@...>
 

Hi, Neil
 
I'm pleased to see all of these great examples of measurement. 
 
I think you are also asking for benchmarks? For example how actual km project teams perform, meet milestones, spend, etc., relative to other companies' teams in the same industry (or of the same size)? 
 
You might want to check out the Intranet Benchmarking Forum. www.ibforum.com (I am a benchmarker and talked a bit about them in Tuesday's SIKM Leaders' conf call.  I'd be happy to explain the benchmarks: Communication and Culture, Strategy and Governance, Usability and Design, Metrics and Performance.)
 
IBForum has had and currently has many government and NGO type members, so you might be able to benchmark against them.
 
Kate 
 
Katrina Pugh
AlignConsulting
Columbia University Information and Knowledge Strategy
Author of Sharing Hidden Know-How (Jossey-Bass/Wiley, 2011)
 
617-967-3910 (mobile)
 
 

In a message dated 10/20/11 11:17:09 Eastern Daylight Time, olonoff@... writes:
 

Thanks Tom, 


good stuff. Will I share what I learn? 

Well, sure! 

regards, 

Neil 


Neil Olonoff 




On Thu, Oct 20, 2011 at 11:13 AM, Tom Short <tman9999@...> wrote:
 

Neil -

Here is a good article highlighting some metrics and approaches at Accenture:
http://www.accenture.com/us-en/Pages/insight-cio-accenture-employee-collaboration.aspx?mkt_tok=3RkMMJWWfF9wsRolua7BZKXonjHpfsX56ukoXqOg38431UFwdcjKPmjr1YEJTsd0dvycMRAVFZl5nQJdDfOSfZRP6A%3D%3D

I have developed a variety of different scorecards/measurement frameworks fo r various KM and related initiatives. One of my favorite metrics for CoPs is based on social network analysis (SNA). It's a longer-term metric that is based on measuring the change in density and scope of connections within a CoP. You use SNA to create a baseline for a CoP, then conduct an SNA again six months to a year later. Changes in density, strength scores, and breadth become possible indicators of CoP effectiveness/activity. As a control you can go a step further and conduct an SNA for a similar group of employees who are not engaged in a CoP.

Good luck with your effort - are you going to share what you learn??

-Tom



--- In sikmleaders@..., Neil Olonoff wrote:
>
> Hi all,
>
> I've just completed an interim version of my KM strategy plan for Army
> Medicine, and need some industry metrics to benchmark against. If
> anyone has a source or link to KM metrics (for example, Community of
> Practice support, content management, best practices, collaboration)
> please let me know.
>
> In addition, if anyone in government would like to read and review the
> plan, please drop me an email.
>
> thanks very much,
>
>
> Neil Olonoff
>