Set Ratings

Eric

Hi, I always wanted to rate a set but not actually write a review about it. I'm not great and reviewing stuff, (I've always have trouble explaining movies to people without spoiling the ending :P ) so, I think that giving this option of rating a set out of 5 (or 10) without the perceived 'hassle' of writing a review would give a much more 'honest' opinion of any particular set.
What do we think?

SapmiSatan

I agree, but I think you should have to rate all the aspects of a set, like you do when you review, so people won't rate sets without first thinking it through. I guess that's what's good about rating when reviewing, you get more serious votes.

Eric

^ Very good point. You'd probably just get people stamping in 5 stars or 1 star depending on how it looks in the picture.

Eric

{no text}

RabbitWizzard

If you write a review, make the rating worth 20x than a rating without a review and that problem should be solved.

davee123

One thing I'd like (NERRRD!) is to be able to do this and then download the rating database anonymously. I remember doing some AI algorithms back in college that let you predict whether or not you'd like something based on other people who had similar tastes to your own. Something like "people like you, who liked X, but disliked Y, appeared to all like/dislike set Z".

Handy for all those out there who are always asking "which set should I buy?"

DaveE

brickmatic

I don't do reviews because it is time consuming. I think a rating system with different rateable aspects is a great idea.

Yellowcastle

^ I agree. Perhaps these could simply be weighted less and/or still include the various diff criteria.

dougts

I do some reviews, but sometimes it would be nice just to be able to rate a set among the different rating categories

rocao

Im a big proponent of this so that I can help give vintage sets meaningful ratings. The current situation of little to no reviews means they are often at or near the baseline average rating, which is lower than most of them _should_ be... IMO, of course :p

MinifigsMe

+1

I'd like to rate sets, but I'm far too lazy to write reviews. But I agree - having the rating in diff categories would work well. A bit like feedback on ebay

Halberd777

Yes I would say the same thing. I really don't want to have to review every lego set that I have in detail. If there was a thorough option of rating then the overall scores of lego sets I believe would be a little more realistic. Right now we just have a very aristocratic system just the view that have the time to review end up putting the scores on lego items.

vynsane

To play devil's advocate, leaving the ratings only to fully qualified reviews means that people are more likely to write a review - it could be argued that adding the ability to rate a set without writing a review could have a detrimental impact on the number of reviews written.

Yellowcastle

^ Good point, though I'm not sure I'm sold. :o)

rocao

^^ Most of the good reviews are written by people who take the time to provide a quality review, and the sense of pride apparent in this work would indicate that they are sufficiently motivated for the task and not simply performing a necessary evil in order to be able to rate a set.

vynsane

^ True - it's highly likely that allowing ratings sans review will actually only make an impact on low-quality reviews, while those who write quality reviews will continue to do so... So I rescind my devil's advocate stance and whole-heartedly support this measure ;)

davee123

My inclination (when I found out BrickSet had reviews) was to go through and review everything I owned (that's what I did on LUGNET), but when I found out I actually had to write something, I stopped immediately. I have time enough in my day to click through sets and rate them, but I do NOT have time to review everything I'd like to give a rating to.

Mostly, though, I have to admit I don't really find LEGO set reviews helpful, even when they're well written. I find I can get the jist of a LEGO set pretty easily just by looking at the picture. The only times I find reviews helpful is when they provide information that isn't immediately apparent from the pictures. But that information is typically buried in other text, which isn't saying anything that I didn't already know.

But I really do like seeing throngs of data. How does everyone ELSE feel about this set?

Typically, I find that people review the sets that they feel strongly about. 6080 King's Castle has 12 reviews, but 6029 Treasure Guard has only 2. 928 Galaxy Explorer has 32 reviews, but 6875 Futuron Hovercraft has only 1 (and I think is a much cooler set than the reviewer rated it!). 1620 doesn't even have any reviews.

Anyway, even though I appreciate the sentiment NOT to want to allow the unwashed masses to rate things (because they'll be sloppy or what-have-you), I think the alternative is basically that you have no data whatsoever. I'm often leery of things like ratings until there's a few dozen data points, which is... well... most of the sets on BrickSet.

DaveE

Yellowcastle

@davee123 - I would argue that some reviews are quite helpful and as an example I would point you to a recent great review of Heroica in this very forum. ;o)

davee123

Yeah, you're certainly right-- there are definitely some things where you NEED a written review, like the LEGO games! I should clarify that my point was more geared towards the typical sets.

DaveE

Halberd777

Well yes you guys and gals are right about the review process but do you really want how the lego set is rated to be rated by those who have enough time to put together a review. Even if the review is good that person may or may not share my opinion but I usually have just enough time to complain about issues in this forum and not write a review. Seriously though I cannot sit and review everything I own it would take me a very long time to write a thorough enough review. Oh well that is my own two sense.

davee123

but do you really want how the lego set is rated to be rated by those who have enough time to put together a review

I guess I don't see whether or not someone has free time for the task of reviewing as a good indicator of the value of their opinion regarding sets.

What I'd want would be vast hordes of data, with filters galore-- or, more realistically, available for download. As noted earlier, anonymous data regarding each person's relative votes and other related activity. IE, "person X rated set 1234 this way (and owns that set), rated set 5678 this way (and did not own the set), and also wrote a full review for set 2468".

Personally, I would want to tweak the data to my liking. If someone only rated a small handful of sets, then I'd want to weight their information lightly. If someone rated hundreds of sets, weight their votes heavily. If they only ever give things a 3 or a 4 rating (never a 1, 2, or 5), then maybe alter the way that they vote in the weighting, by making their votes more extreme. And maybe throw away the top 5% and bottom 5% of votes (like they sometimes do in the olympics) Etc, etc.

Currently, IMHO, I consider BrickSet to basically have zilch for rating data. I feel sorta like "well, TECHNICALLY BrickSet has set ratings, but not really". 10 votes or less is something I'd consider just throwing out as being too statistically variable. Even 20 votes is pretty slim. Maybe once you've got 50 votes or more, I'll be happy that it's starting to reflect genuine sentiment across the community. But most sets seem to have 10 or fewer reviews, with more recent sets typically having more (or so it seems).

DaveE

georgebjones

^I agree. Rating would add a valuable dataset to the database. Opening up that to the community would allow the views of the community to come through. How many people do you think have an opinion that do not take the time to write a review. I think the Ben-10 sets overall are rubbish, but have some amazing parts in them. However, I have never written a review, nor do I think I ever will, but I will take the time to rate the sets, if the option was there.

I also like the idea of weighting the reviews slightly heavier from people that own the set. True, someone could just say they own it, but they would be the exception, and if there were a couple hundred ratings, theirs wouldn't sway the ratings in the slightest.

-George

legoDad

I think having a link to comment on people's reviews would have some back and forth (hopefully)helpful debate and have the rating systems open for just ratings and not a full length review is great too.

Huw

I'm warming to this idea based on this discussion. Would you want to give the set one overall rating, or score it on the attributes used in the reviews at the moment?

Eric

I think attributes would be best, the more the better.
I'm thinking the same attributes as the current rating system, with the addition of 'First Impressions' and asking if you own the set.

davee123

I'd say let people do both-- IE:

Overall Rating: [1 2 3 4 5]
- Set Design: [1 2 3 4 5]
- Building Experience: [1 2 3 4 5]
- Parts: [1 2 3 4 5]
- Play Value: [1 2 3 4 5]
- Value for Money: [1 2 3 4 5]

So, you can rate it overall, and if you feel like it, you can delve into more detail.

Then, what you do behind-the-scenes to come up with the "overall rating" based on people's individual votes might be straightforward or (my preference) involve some intense logic in order to make it more meaningful. My first attempt would probably be something like:

* Multiply people's vote 'strength' by a weight between 0 and 1, depending on how many sets they've rated. So if you've rated, say, 100+ sets, your votes are counted fully (multiplied by 1), and if you've rated less than 10 sets, your vote gets multiplied by, say, 0.1

* Have a softener (default vote of 50% as LUGNET did) so that it requires many positive votes before something's highly rated. That way you don't have some bizarre Fabuland set (with a single "perfect" rating) absurdly high on the rating list

* Cut a vote strength by half if the person doesn't actually own the set

* Reduce the vote's weight if they didn't vote on all the sub-categories, but only the "overall" category

* Remove the top 5% and bottom 5% of votes overall (for every 20 votes, remove the highest and lowest)

Having been raised by two actuaries, I love analyzing data-- so my vote would be to either let us have all sorts of preferences on how we see the data presented (lots of work for you), OR do something standardized, but let people see (or download!) the raw data for their own consumption.

DaveE

vynsane

* Have a softener (default vote of 50% as LUGNET did) so that it requires many positive votes before something's highly rated. That way you don't have some bizarre Fabuland set (with a single "perfect" rating) absurdly high on the rating list

If I'm not mistaken, this already is the case here, as well.

Yellowcastle

Honestly, I think I'd like more subcategories, possibly in place of "overall rating.". I would think your overall rating for a set should simply be the aggregate of your various sub ratings. And those sub ratings could be weighted differently If we feel play value, for example, is more important than the build. I do think, though, that there are more criteria we could consider.

For example, I'm a big fan of film scores and there are those scores that work great in the movie but can't stand alone at all. I can think of a number of sets that might have been fine overall but were terribly unenjoyable if they were the only sets you had. Am I alone in feeling this way? That new mini Hogwarts set seems like a candidate for this.

No matter what, a ton of questions jump out at me. Can we even consider adding or changing criteria midstream? What then would happen to all the existing ratings? Could they somehow just be grandfathered in under different rules and would we even want that? Will the EB reviews carry any weight? Would expanding to a 10 scale be better?

Huw

Blimey Dave, that's a comprehensive list! Vynsane is right, the softener is already used, and it does work very well.

MinifigsMe

The subcategories are essential. And perhaps a way to say how much was paid. As the value for money rating relates to that.

atkinsar

I think this would be a great addition. Like other, I don't have the time to write comprehensive reviews but would love to rate the sets that I own. My vote would be to keep it simple, and just allow people to rate in exactly the same way as the reviewers do now (but without having to write a review of course).

Huw

And how do you think this should manifest itself in the UI?

Small pop-up window brought up when you click on something in the reviews column?

davee123

Honestly, I think I'd like more subcategories, possibly in place of "overall rating.". I would think your overall rating for a set should simply be the aggregate of your various sub ratings. And those sub ratings could be weighted differently If we feel play value, for example, is more important than the build.

I'd vote to keep "overall rating" a distinct category for people to rate, rather than making it an aggregate. Personally, I don't really care about "building experience" or "playability", so they don't factor heavily into how I personally rate sets. But a parent might care a lot more about those factors! And similarly, a MOC-builder might not care that much how well a set is designed, but care a LOT about which parts are included. By forcing "overall rating" to be an aggregate field, and not something separate, you're forcing my opinion to weight the sub-categories equally-- or, at least, according to a scheme that's pre-made.

Again, this is where I'd love to see vast swaths of options for customizing the view. If you, personally, as (say) a MOC builder, want to see what the aggregate looks like by ignoring the "playability" component, allow it. Etc.

And just to clarify, I'm all for making the "BrickSet Overall Rating" be an aggregate based on all the different people's votes across all the various categories-- but I think each individual person's "overall rating" shouldn't be an aggregate based on their sub-category votes.

Can we even consider adding or changing criteria midstream? What then would happen to all the existing ratings? Could they somehow just be grandfathered in under different rules and would we even want that? Will the EB reviews carry any weight? Would expanding to a 10 scale be better?

I can't speak to how much effort it would be, but I'd recommend keeping and grandfathering in the existing review data, but probably adjusting it to fit the newer model. Like, if new sub-categories are available, then just leave them blank for the reviewer.

I dunno about the 0-10 vs. 1-5 system. It might be a little much, but some people like giving the extra granularity.

And how do you think this should manifest itself in the UI?

My vote would be for a small "Rate" form within the page, probably right underneath the "write a review" link-- IE, as little bouncing to new pages as possible.

Ideally, it could even go in the "Sets I own" screen, although it'd probably be way too crammed given the current layout. I suppose, to encourage more people to vote, you could have a screen dedicated to rating the sets they own-- "Rate My Sets" or something. Could even filter things to the top that you haven't rated yet! That way, you can zip through and rate your sets pretty easily.

DaveE

atkinsar

^^ yes that would most agreeable, although I just know that as soon as it's possible, I won't rest until all my sets are rated!

Huw

Right O, I'll see what I can come up with...

MinifigsMe

Yay! Thanks @Huw :)

Halberd777

Huw brickset is definately your brainchild but I would like to thank you for being so open to evolving it into something we all know and love when you could just choose to keep it the same way. I would say Kudos to you if you recognize the term.

Huw

The site would be nothing without you, the visitors, so it pays to keep you happy :-)

Eric

Not to be pushy, but any news on where we are at?

Huw

I've had a go at it but it's proving to be not as easy as I had hoped to do it in the way I want to. So, it's back to the drawing board for now.

Datoda

Hi,

Love this website and forum a lot. The database is so useful and the people are all so friendly here.

One question about the "brick score" that goes with each set in the database: How is it determined? I don't see a relation between that score and the scores given by the reviewers of the set. For example, set 2519 has a brick score of 3.2, yet the only review there gave it at least 4 bricks in each of the five areas being evaluated. Your explanation would be appreciated. Thanks.

Huw

There's a weighting score, of 2.5, added which causes the number of ratings as well as the rating values themselves to influence the overall rating.

So, 1 score of 4

4 + 2.5 / 2 = 3.2

1 score of 5

5 + 2.5 / 2 = 3.75

5 scores of 5

5*5 + 2.5 / 6 = 4.5

100 scores of 5

100*5 + 2.5 / 101 = 4.97

It stops a single score of 5 giving the same rating as a set that has 100 scores of 5.

Clever eh? :)

Si_UKNZ

Id wondered about for a while, and had assumed its was some very complicated formula, as it always seems to appropriately weight the number of reviews.

Devilishly simple - genius!

Dato

@Surrey: I totally agree.
@Huw: Thanks for the explanation.

DrFrankenstein

I have been noticing that some of the set rating scores on the site don't seem to relate mathematically to the review scores for the set. For example the 3.9 ratings for sets 9392 and 8838 cannot possibly be derived from the review scores which are 4 and 5 only. So, I wonder if there is something that I am missing here? Perhaps the algorithm for calculating the review scores uses a more complex formula than simply averaging the review scores?

tk79

Here you go:

http://www.bricksetforum.com/discussion/comment/27120/#Comment_27120

Yellowcastle

^ + ^^ Please note that I've now merged these discussions to keep things from getting splintered. :o)

DrFrankenstein

Thanks, tk79. Very elegant simple algorithm indeed.