Shopping at LEGO or Amazon?
Please use our links: LEGO.com Amazon
As an Amazon Associate we earn from qualifying purchases.

Ratings revisited

snowymikesnowymike Member Posts: 4
Hello Brickset Gurus,

Love Brickset, and thanks Huw and everyone that helps make this site so wonderful.

One thing I've been confused about many times is the rating system, which I've discovered is explained in this thread: http://www.bricksetforum.com/discussion/9009/how-are-the-review-ratings-calculated. Basically a pretend 2.5 rating is included and used to calculate the average rating. A benefit of this approach is it makes the Top Rated query avoid treating a single 5 star review the same as fifty 5 star reviews. I understand the motivation behind not wanting models with limited reviews from overtaking, say, Black Seas Barracuda's 4.89 (weighted) average. And the weighted average is a handy way to treat fifty 5 star reviews as better than forty.

With the current model, a single 5 star review gives you a weighted average of 3.75, which will display 4 stars and show 3.75 on hover (or 3.75 on the detail page for the model). A single 4 star review gives you a weighted average of 3.25, which will display 3 stars and show 3.25 on hover (and detail page).

I'm concerned with the false impression this weighted average presents. Consider:
* As I write this, there is a single 2013 Creator model with more than 5 reviews, and most with reviews have 1 or 2. There are no 2014 Creator models with 5 reviews, and most items with a review have 1 review.
* It's when a model is first released (or otherwise still in stores) that ratings are most sought after. At least that's my impression for the majority of Lego consumers (who aren't buying old sets on Ebay), though certainly not all Brickset users fit this model. And that's precisely when the weighted average is least accurate and presents a false impression on what the reviewers actually thought of a model.

I often see a model that appeals to me with a 3 star rating (or 3.25 on hover) and think to myself that the reviewer must not have liked it. Sometimes that causes me to skip over a model. Other times I'll click to the review, and I might find glowing review feedback, just not quite good enough to warrant a 5 star rating. A 4 star review is much different than a 3 star review, and to see so many 1 review models coming in at 3 stars, incorrectly, I think is concerning.

I also think I must not be the only person that, before coming across the forum post, just assumed Brickset's rating system was broken. If I see a rating of 3.75 I think "OK, probably three 4's and one 3". But that's what a single 5 star rating gives you at Brickset. On a number of occasions I've attempted to work out the math and have given up, just assuming the rating was not to be trusted.

Are there other options that allow the Top Rated query (and related queries) to work OK that don't result in an incorrect average rating? Such as:
1) Require models to have at least X reviews to be included in Top Rated, maybe X = 10 (which the top 25 all have), and then sort by real average. That would work for Top Rated on its own, but not when combined with other query parameters.
2) Separately store (or compute) the real average and the weighted average, displaying the real average when I view a particular model, but using the weighted average to determine items to display (and in what order) in the Top Rated query.

I also considered making the Top Rated query first sort by real average and secondarily sort by number of reviews, but a problem there is a single 5 star review would be enough to bump Barracuda.

Other thoughts?

I'm not sure the right solution, but I did want to raise (as some past posters have) the concerns I see with the current system, which can cause users to possibly a) lose trust in Brickset (rating system appears broken as math doesn't add up), or b) not pursue models that are either new or have few reviews based on an incorrect assumption that they were rated lower than they actually were.

Thanks.

-mike

Comments

  • HuwHuw Brickset Towers, Hampshire, UKAdministrator Posts: 6,455
    Thanks for your long and thought-provoking post! It's a subject which is revisited by the community every so often. I don't have any answers (yet) but I look forward to seeing what suggestions others come up with.

    The raw data is, of course, all in the database and the adding of the balancing score is done in a query so if a solution is found it should be easy enough to implement.
  • snowymikesnowymike Member Posts: 4
    Thanks for the reply, Huw.

    If you've found the addition of the 2.5 rating works well for Top Rated and other similar sorts (like User Rating high > low), and if this balancing score is added via a query, what's the concern with my option 2 above? Continue to process Top Rated and related queries using your weighted average (weighted average determines sort order), but when actually showing the rating for a model use the actual average rating. That way a product with one 5 star review shows an accurate rating of 5, but won't appear at the top of the Top Rated and User Rating high > low type sorts.

    The only concern I can see if this change is made is that someone might wonder "If I'm sorting User Rating high > low, why is this item that tells me it has a 5 star average rating displayed below an item that has an average rating of 4.8". That's a potential source of confusion, but I guess to me less confusing than a single 5 star rating for a model displaying as a 4 (or 3.75) rating. I tend to expect that sites do some magic sorting with "Most Popular" and "Highest Rated" type queries, understanding that more votes actually does mean something. But I expect when viewing a particular model that I really see it's actual average rating.

    -mike
  • HuwHuw Brickset Towers, Hampshire, UKAdministrator Posts: 6,455
    Yes that could work, let me look into it...
  • OldfanOldfan Chicagoland, IL, USAMember Posts: 621
    Some sites with ratings have the "official" rating as the one with the weighted averages, but details on the thing being rated will show the actual averages as well. Boardgamegeek.com does something like this; the only issue is that seeing all the different numbers in different places can be confusing.

    (However, the bgg'ers take great pride in the overall comparative rankings of their favorite board games, while the set review rankings on this site seem not to be a big deal; at least, I don't see too many forum wars concerning "my favorite set has higher rankings than your favorite set" here. So, having different sets of ranking numbers on Brickset might not be too disruptive.)

    I couldn't care less about the "official" set rankings here, but I'm happy to offer suggestions anyway!
  • PeteMPeteM Gallifrey (near Bristol)Member Posts: 402
    It's not a solution, but I have a query saved which brings up all sets I own with 1 review or less. I'm (very) slowly working my way through them and providing decent-length reviews do that - even though the star rating may not help, anyone interested in the set can at least read the narrative and get a slightly better idea :)
  • BooTheMightyHamsterBooTheMightyHamster Northern edge of London, just before the dragons...Member Posts: 1,293
    @PeteM‌
    That's an excellent idea! Might have a look and see if I can fill in any gaps too.
  • rocaorocao Administrator Posts: 4,288
    @Huw : We discussed this in the past, but I think it would be good to have a quick review system where a user could just submit the 1-5 star ratings for the various features of the set without having to do a text review.

    As a follow-on, perhaps there could be a tool/widget that automatically displays sets marked as owned by the user that are in need of reviews (below a threshold). Didn't you code something similar when we crowd sourced images?

    The benefit is obviously that we'd have more ratings since the time commitment is dramatically less.

    The drawback perhaps is that fewer people would submit in-depth reviews. If that's the case maybe we incentive it somehow?
    dougtsmargot
  • snowymikesnowymike Member Posts: 4
    Huw - did you implement the change proposed on July 12th recently? I noticed today that models with a single 5 star review now display with a 5 star rating (the average rating), while the Top Rated query still functions as it did before (sorted by the weighted average). Looks great to me from the few models/queries I've checked out.
  • HuwHuw Brickset Towers, Hampshire, UKAdministrator Posts: 6,455
    Yes I did, back in October.
  • snowymikesnowymike Member Posts: 4
    Great, thanks!
Sign In or Register to comment.

Shopping at LEGO.com or Amazon?

Please use our links: LEGO.com Amazon

Recent discussions Categories Privacy Policy

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Brickset.com is a participant in the Amazon Services LLC Associates Program, the Amazon.com.ca, Inc. Associates Program and the Amazon EU Associates Programme, which are affiliate advertising programs designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.

As an Amazon Associate we earn from qualifying purchases.