Metrics and reporting: capturing operational indicators and producing reports to communicate performance against goals, areas for improvement, and progress toward the desired state
There is a wide spectrum of opinion about the importance of measuring knowledge management activities. Some believe that it is essential, and want to collect data and create reports on a long list of detailed metrics. Others believe that this is a waste of time, and that effort is better spent on enabling knowledge flow.
Three different kinds of KM metrics are typically captured and reported. Goal-oriented measurements which directly relate to employee goals allow assessment against those goals. Operational metrics are based on data captured by KM systems, and include such details as web page hits, uploads, and downloads; threaded discussion subscribers, posts, and replies; repository submissions, searches, and retrievals; and document ratings, wiki entries, and blog posts. Business impact metrics attempt to determine the return on investment (ROI) of KM initiatives, and include costs saved, costs avoided, incremental revenue, improved quality, increased customer satisfaction and retention, new business attracted, increased market share, revenue from innovations, and revenue from inventions.
Collecting and reporting on goal-oriented measurements ensures that the organization is aware of how it is performing and that individuals can be held accountable for achieving their goals. Reports should be produced and distributed every month to track progress, reinforce good performance, and encourage improvements where needed. Reporting metrics by group within the organization, for example, regions of the world or countries within a region, allows each group to compare its performance against other groups, and create a friendly competition to excel. Reporting metrics by individual may be limited by data privacy laws, and if allowed, transmitted confidentially to the manager for use in performance coaching and appraisals.
Operational metrics can be helpful in analyzing how the KM infrastructure is being used, who is using it, and identifying areas for improvement. However, there is only so much which can be inferred from page hits, uploads, and downloads. These metrics don't indicate the value of any of these activities. If a user visits a web page, they may not find what they need there. If a document is uploaded, it may not be of any use to anyone. If a document is downloaded, it may not be reused. Follow these rules when deciding on which operational metrics to collect and report: Keep the time and effort required to a minimum, automating as much of the collection, extraction, and production as possible. Ask your KM team about which metrics will help them the most. Focus on a few key metrics which relate to your overall objectives. Use the metrics to improve the KM environment and test for this in user surveys. Communicate the metrics regularly so that they influence behavior.
Business impact metrics are potentially useful in justifying the expense of a KM program, in garnering management support, and in communicating the value of spending time on KM activities. Anecdotes and success stories can be collected and converted into numerical values. Data can be captured in contribution repositories and incentive points systems about the value of learning, sharing, reusing, collaborating, and innovating. Processes can be created or modified to ask participants about the business impact of KM tasks. But there are few definitive ways to prove that a particular business indicator was solely influenced by KM. There are usually multiple reasons for a specific business result, and KM may be one of those reasons.
Some firms have conducted one-time surveys to prove the case for KM. For example, Caterpillar commissioned a one-time study (http://www.kmdynamics.com/newspowers.html ) by an independent consulting firm to identify the benefits and ROI for two established communities of practice: Joints and Fasteners and Dealer Service Training. The results were:
- Qualitative ROI: Productivity (up 40%), Cost (reduced 25%), Speed (up 15%), Quality (up 4%)
- Tangible ROI: 200% for internal CoPs; 700% for external CoPs
Based on these results, the Caterpillar KM program was justified, and has been supported ever since. There is no need for ongoing collection and reporting of ROI, since it has been done once.
If there is a way for you to collect business impact metrics, then do so. They have more significance than operational metrics. But follow the same guidelines about limiting the effort involved to a reasonable amount.
Collecting and reporting on the measurements used in your KM program will help you to communicate progress, motivate people to improve their performance, and reassure management of the value of the initiative. Keep the effort required to do so in the right balance with other projects, look for ways to continue to streamline the process, and review the reporting process annually to keep it relevant to the current state.
From the Communities Manifesto https://docs.google.com/View?id=ddj598qm_44fx54rbg5
Goals should be set for communities and progress against those goals should be measured and reported. Unhealthy communities should either be nurtured back to health or retired. There are five ways to measure the success of a communities program: PATCH - Participation, Anecdotes, Tools, Coverage, Health
- Participation: % of target population which is a member of at least one community
- Anecdotes: % of communities displaying the following on their sites:
Tools: % of communities having all five key tools (see below)
Coverage: % of desired topics covered by at least one community
Health: % of communities meeting these criteria:
- Testimonials by community members on the value of participation
- Stories about the usefulness of the community
- Posts thanking other members for their help
- At least one post to a threaded discussion board per week
- At least one newsletter or blog post per month
- At least one conference call, webinar, or face-to-face meeting per quarter
- At least 50 members
- At least 10 members participating in each event