Measure and baseline the current levels of agents' knowledge #Art-of-KM

Nikhil Katyal

Hi All - I am a Knowledge Manager with a leading customer services organisation in India. I have a question. We want to measure and baseline the current levels of agents' knowledge (understanding of processes, awareness of products and services, etc.). What is the best way to do it? Any guidance would really help. Thanks. Nikhil

Vinod Shenoy

Hi Nikhil,
This is a tough thing to do without understanding your situation better. But here is a very high level of what you want to start measuring:
  1. Top 10-20 question types that drive the most channel volume. I assume your primary channel is phone.
  2. Quality issues your Audit team identifies namely:
    1. Ability to comprehend the customer's question to provide a suitable response
    2. Completeness and accuracy of the response
  3. Average Handle Time (AHT), though I would caution against placing a high value to this initially. Quality issues will percolate into AHT but may or may not be caused by knowledge issues.
  4. Adoption (by how many) and Utilization (by how much) of your Knowledge Articles if you use a central Knowledge Base. If not, identify sources of knowledge your agents use (email, printed guides etc.). Remember that 20% of your articles will be used 80% of the time.
You can then determine insights into what Knowledge challenges your agents are facing in their attempts to successfully solve customers issues. Let me know if you have more questions or would like to discuss offline.

- Vinod

John Coles

Hi Nikhil,

Much like Vinod wrote, we would want to understand a bit more about the environment. And as ISO 30401 calls-out, we want to know about the Context of the Organization.

If the goals is to "measure and baseline the current levels of agent knowledge," then it sounds like you are leaning toward an agent competency assessment. There are numerous ways to do that, but I caution you to head in this direction. For instance ...

    • What is the Agent Attrition Rate? 
      • If you have a high turnover, then the Agent Knowledge is walking out the door. Thus, measure Agent Competency is Sisyphus effort :-)

  • Depending on the Contact Center, what is the ratio of "New" vs "Known" calls?
    • If the agents are getting a lot of rote, redundant contacts (ie.password reset, policy questions, non-complex product/service questions) then there is less reliance on having a good healthy, living KB.
    • Conversely, if the type of contacts are always evolving due to the release of new products or services or they lean more toward medium-to-high complexity, then it will be a challenge for build a strong, solid Agent Competency. In this case, it would be better to follow a solid KM Methodology such as Knowledge Centered Service (KCS).

There is definitely more to than that, but perhaps this helps you build a better approach. So we are definitely looking for more context. If you would like to get on a call, let me know.


Nikhil Katyal

Thank you Vinod and John. Very kind of you guys to take the time out and reply. So here’s more details. Agents handle customer interactions over phone, chat and email. The current Quality model mainly scores the quality of interaction (rightly so as well). We have no way to showcase how any agents’ knowledge of processes, products, business updates, etc. impacts their ability to handle interactions efficiently. So I want to baseline their knowledge/know-how and then come up with plan to improve that.

PS – agents are not expected to remember stuff. They should be using the KB for business updates and processes to be followed while handling interactions.

Vinod Shenoy

You will find the answer in your problem statement - The current Quality model mainly scores the quality of interaction with no way to showcase how any agents’ knowledge of processes, products, business updates, etc.

As I said before, your data source (i.e. Audit logs) should be able to provide what you need - 
  1. Agent's ability to comprehend the customer's question to provide a suitable response
  2. Completeness and accuracy of the agent's response

You can conduct a test of the agent's knowledge based on the top 20 calls to determine their familiarity with processes, products, business updates, etc. I suspect you are looking for a solution outside this top 20 and the only way to get answers is to:
  1. Collect the right data
  2. Mine and analyze the data
Given the complexity of having to change processes outside your direct control, I would recommend a short term focus of:
  1. Making it easier for agents to access the knowledge they need by reducing number of clicks to access content, ensuring content is up to date by working with SMEs, and making it easier for agents to submit corrections.
  2. Improve communication of updates to processes/products through TLs running daily stand ups and storing updates/talking points for reference on your KB
  3. Allow sufficient time for agents to familiarize themselves with the content more through daily reading time and refresher training
These steps will reinforce concepts for your agents - the what they need - and tide you till you are able to get the necessary data. After which, you will likely be tweaking these steps anyways - more or less tweaking depending on what your data tells you.

From experience, call centers have high turnover so knowing "how well" your agents understand all processes, products is outweighs knowing "what" agents need to be able to answer customers' questions about your products correctly and consistently.

Let me know how I can help you.

T J Elliott

Dear Nikhil,
The question that you're asking is one of assessment in my opinion. And the answer depends upon how granular you want your baseline to be. It would seem from your question that you have an idea of the constructs that are involved, but within each of those constructs -- understanding of processes, awareness of products and services, etc -- subconstructs. for example, I imagine you have more than one process and I imagine that each process has more than one component. An agent might be very familiar with three out of four components but completely ignorant of the last one.
In my experience, what might prove more valuable for you Is to engage the agents in a collective exercise that captures what they know. Don't worry about it on an individual basis so much an instead bring them together and groups and have them paint the picture of the necessary knowledge to do their job. Some of them will know more than others, but those others will gain by being in the same room with individuals who have greater knowledge or skill. In other words, the scaffolding effect will prevail.
Once you have this information, then you can take a look flying it for new employee learning and you can also determine what areas appear to be the weakest. In other words, what were the areas that you had to fill in for the group where no member appear to have full knowledge of a process or a product.
I hope that this is useful for you. I would be glad to comment further if you desire. I'm sure there are others on this list who have good ideas as well. It is perhaps a bit ironic that having been device president and chief learning officer at educational testing service that I appear to be turning you away from assessment, but I am in a position to know how arduous and expensive and activity that can be. Good luck