1. SIKM | Main
  2. Messages

Re: Amazon Ratings - One Thumb Down #ratings #prediction-markets

David Snowden <snowded@...>
 

Sorry not to have been active, or on the calls - too much travel
However this is a topic of considerable interest to us, and a part of our own software development and method research.
I think there are some key points that should be made
1 - Goodharts Law states "The minute a measure becomes a target it ceases to be a measure" and we can extensive evidence of this in government and industry
2 - Snowden's variation on that law is that "anything explicit will be gamed"
3 - It follows that it is key to any rating system, that the rating does not produce financial or status based rewards, if they do it will be gamed. (look at Bay, the ability to game Goggle searches etc. etc.)
4 - Bloggs are now being manipulated and also some other folksonomies.
5 - Any rating system needs to allow people to choose a filter in which they see ratings based on people who;s opinion they respect.
6 - artefacts cannot have absolute value (sorry to disagree with my former employer here), they have value in context.
7 - if we look at the three generations of understanding of data we can see (i) its all about the data, followed by (ii) its about the data with conversations (CoP, folksonomy etc.) and now (iii) data with conversations in models
8 - this third approach is at the heart of the work we are currently doing on horizon scanning, weak signal detection etc. in anti-terrorism, but which is also applicable (and is being applied) in more conventional KM settings.  It requires a switch from taxonomic structures and search mechanisms to ones based on serendipitous encounter within the context of need.
9 - The concept of corporate memory needs to start to mimic human memory which is pattern based.  An expert for example has over 40K patterns on their long term memory sequenced in frequency of use which are selected on a first fit basis (Klein and others).  Each of those patters is a complex mixture of data, experience, perspective.  By storing data-conversation-model combinations in context but without structure we can start to allow contextual discovery

Now a lot of that is cryptic, but the most important words are CONTEXT and SERENDIPITY.  We need to move away from storing, accessing and rating artefacts and start to look at corporate memory as a complex system in which patterns of meaning will emerge within models




Dave Snowden
Founder, The Cynefin Centre
www.cynefin.net


On 20 Jan 2006, at 23:42, Mark May wrote:

I really enjoy this topic because it has great potential to benefit our practitioners in the field and because it is a concept that everyone can envision, given the ubiquity of the Amazon experience.

My thinking starts with how one would propose to use rating/feedback data. I can see at least three possible uses - 1) Provide confidence or caution to people who are considering using an IC artifact; 2) Help order search results to put higher rated content above lower rated content; 3) Provide input to content managers either to promote well rated IC or improve or retire lower rated IC.

These are all worthwhile and valuable uses of ratings/feedback data. However for many of the reasons that people have cited below, I don't think that a five star rating system provides data that really is of value to meet these ends. In addition, most content is not rated at all by anyone or is not rated by enough people or it takes too long to get enough ratings to represent a consensus opinion.

Given these limitations of an Amazon-like system, the IBM team is trying something a bit different. First of all, we have deployed the standard five star system and allowed space for comments on all artifacts in the major repositories. We feel that users have become conditioned through other internet experiences to expect this kind of a feedback approach. However, we don't use that raw data by itself for KM purposes. We combine the rating with other data to impute a VALUE for that artifact. The other data includes number of times it was read, downloaded, forwarded to others and printed. These factors are weighted so that a download counts for 10 times the value as a read, for example. We also give significant extra weight to any comments since we think that the comments are much more valuable than the ratings.

We have deployed this approach and are actively calculating imputed values now. However, we have not yet begun to use the data. One of our highest priorities is to help people find stuff faster, so we are eager to use these imputed values to order search results. We also plan to make it available to content managers so that they can see what is accessed and "valued" most (and least) among their inventory of content. The jury is still out on how much our practitioners will actually benefit from this imputed value approach. We have some pilots planned for later this year to see how if it works as well in practice as we think it should.

Mark May
IBM

"Tom" <tombarfield@...>



To

sikmleaders@...

cc


Subject

[sikmleaders] Re: Amazon Ratings - One Thumb Down

Bruce I found your comments on Tuesday and in this note insightful.  
I also liked the insights I heard from Kent Greenes (I think).

In the next couple months I am going to be asking my team at
Accenture to develop our strategy in this area.  Here are some off
the cuff thoughts based on Tuesday's discussion and what Bruce and
Ravi shared in this discussion.

I wonder if we should consider moving away from trying to collect
feedback from everyone and instead try to get feedback form people
who feel very strongly about the content - either good or bad.  In
other words - if something warrants a 2,3 or 4 on a 5 point scale
then I don't really care as much about the feedback.

If I download a piece of content that turns out to be a big help to
me (score of 5 on a 5 point scale) I am probably more willing to
provide feedback saying thank you and recognizing that.  It would be
like saying I only want a rating on 5 star stuff.  

If I download something that I really find to be worthless (scale of
1 on a 5 point scale) I might be incented to provide feedback to
either improve it or get it out of the system so no one else has
deal with it.

Tom Barfield
Accenture


--- In sikmleaders@..., "Bruce Karney"
wrote:
>
> Hi all,
>
> In Tuesday's call, I made a comment about why I don't
think "Amazon-
> style ratings" are an effective KM strategy.  Let me explain
briefly
> why I believe that, and what I think is a better approach.
>
> Let me contrast two kinds of reviews or rating schemes.  These are
> not the only two kinds, but they represent the opposite ends of a
> spectrum.
>
> 1. Peer review, prior to publication: This is the standard used by
> scientists and academics.  In essence, drafts of articles are
> circulated to "experts" who offer advice and input prior to
> publication.  This input is used by the author (and perhaps the
> editor of the journal) to improve the work BEFORE it is exposed
> (published) to a wide audience.
>
> 2. Consumer review, after publication: Amazon, ePinions, and many
> similar rating and awards systems use this approach.  Because
these
> post-publication reviews cannot affect the published work, they
> are "criticism" in the literary sense.  In Amazon's case, no
> credentials are required to post a review, so the reviewers are
not
> peers of the authors.  Nobel prize winners and your local pizza
> delivery guy have an equal voice in Amazon-land (and the pizza guy
> probably has more free time).
>
> Being able to write one's own review is a satisfying thing for the
> reviewer, especially since it has only become possible to do this
in
> the last few years.  However, the only way Amazon reviews impact
the
> world at large is to pull more readers toward a book or push a few
> away.  Isn't it better, especially in a business context, to use
> techniques that IMPROVE THE QUALITY OF THE BOOK?
>
> That's what Peer Review is designed to do.  If business KM systems
> can't support pre-publication Peer Review, they should at the very
> least focus on post-publication Peer Review and document
improvement.
>
> I also mentioned that at HP, where I used to work, most document
> ratings were 4's or 5's on a scale of 1-5.  I have located a copy
of
> study I did on the topic earlier in the year, and would like to
> share my findings:
>
> For a sample of 57 "Knowledge Briefs," which are 6-12 page
technical
> documents desighned to inform and enlighten, there were 12,295
> downloads and only 53 ratings/reviews.  This is a ratio of 1
review
> per 232 downloads, and slightly less than one review per
document.  
>
> ALL ratings were either 4 or 5.  The 53 reviews were provided by
40
> different individuals, so the vast majority of people who
submitted
> a review submitted only one, meaning (perhaps) that they lacked a
> valid base for comparing the Knowledge Brief they were reviewing
to
> any other Brief.  The most reviews submitted by a single person
was
> 7, and the second-most was 3.
>
> I contend that if you were perusing a listing of Knowledge Briefs
on
> a given subject, all of which were either unrated or had ratings
> between 4.0 and 5.0, you would not have information that would
steer
> you towards best documents or away from poor documents.  You would
> believe that any of the documents could be worthwhile, inasmuch as
> none of them had low scores.  Therefore, the rating scheme
provides
> NO value to the prospective reader.  Worse yet, if there were a
> documented rated 1, 2 or 3, that rating would probably be a single
> individual's opinion because of the infrequency with which
Knowledge
> Briefs are rated at all.
>
> My conclusion: don't RATE documents, but create systems to provide
> detailed written feedback from readers to authors BEFORE
publication
> if possible, or AFTER publication if that's the best you can do.  
> Encourage COLLABORATION, not CRITICISM.
>
> Cheers,
> Bruce Karney
> http://km-experts.com
>







Yahoo! Groups Links

<*> To visit your group on the web, go to:
   http://groups.yahoo.com/group/sikmleaders/

<*> To unsubscribe from this group, send an email to:
   sikmleaders-unsubscribe@...

<*> Your use of Yahoo! Groups is subject to:
   http://docs.yahoo.com/info/terms/







SPONSORED LINKS
Knowledge management Consulting firms System integration
Computer security Computer training Computer internet

YAHOO! GROUPS LINKS




Join {main@SIKM.groups.io to automatically receive all group messages.