Rating the reviews: What do you look for in review quality?

Town

When you go to rate a review here on Brickset as being helpful or not to you, what things do you take into consideration?

My reviews are often marked 'unhelpful' and then subsequently hidden.

PaulTR

I'll share with you the things I use to rate reviews. Please note that I'm not saying these are the things wrong with YOUR reviews; I haven't even read any of your reviews, so I'm not sure what is wrong with them. I'm just noting things that I personally look for in a review :)

1. Proper Grammar:
If someone doesn't use proper grammar in their review and if they chuck out things like, "So yea, i really like this set. it Has cool figs and love the shooting funktion! 5 out of 5 starz!", then I view that as a negative thing. You don't have to go crazy and use big words; just use words that best describe your feelings on the set.

2. Organized:
I like a review that has seperate paragraphs for certain sections of the review. For example, you might want to make a section called "Look of the Set" and then make a seperate section called "Playability". This allows you to organize your thoughts better and it allows people to easily skip to which section they most care about. Just make sure you don't make the review too long: for me personally, I like a review that has an intro (1 or 2 sentences), around 3 to 4 paragraphs of the "meat" of the review (3 to 5 sentences per paragraph), and a conclusion (1 or 2 sentences). Also, I like it when people use the little "Pros and Cons" tool to make a summary of the best and worst parts of the sets. It really helps to summarize the whole review.

3. Descriptive:
I like a review that explains where you got the set, how much you payed for it, and what are your true thoughts on it. For me, saying "This set is good, I guess" isn't going to cut it. Explain what kinds of things this set is useful for: MOCing, playing with, displaying, re-sale for a profit, ect. Explain the functions, look of the set, cool pieces used, negative things about the set, things like that. Remember, you don't have to get crazy with your description; just mention the things that pop out to you and that might pop out to different types of people.

Well, that's what I use to rate a review on. I actually base my own reviews off of the style I described. If you want to check out my reviews for a reference, just head on over to the Bionicle Barrakii line: I reviewed every one of them, and even a few of the Mahri. I'm not saying my review style is the best; I'm just saying it might help you get some ideas. :)

bluemoose

I think it's easier to say what turns me off about a review ...
(1) Bad spelling & grammar
(2) Bad/sloppy layout
(3) half the review being about things that have got nothing really to do with the actual set ("I got this from my friend", "My dog chewed one of the minifigures", etc.)
(4) subjective statements without any supporting evidence ("I hate this set. It sucks")
(5) lists of things that are in the set (especially if listed under "pros")
(6) wasting space - putting in unnecessary blank lines between paragraphs, loads of extra blank lines at the end of the review, splitting paragraphs unnecessarily, etc.
(7) using signature lines, etc., at the end of a review
(8) ... ... ...

I could go on, but basically, for me, a review is about getting as much useful information over to the reader in as small a space as possible. Anything that isn't relevant to the reader in getting a better understanding of the set, to know if it is one they should be interested in, counts against the review.

Probably the biggest turn-offs for me is when people post review text that contains loads of hidden HTML; I often end up having to manually edit the raw HTML code in the post to make it look OK.

The aim of a review is to provide information, to help other people. If the person submitting the review can't be bothered to take the time & make the effort to create a review that is useful & intelligible, then, frankly, why should the reader take the time to read it?

PaulTR

^Bluemoose, you said it better than me; my post was more about how to make a good review. Your comment is much more helpful! :)

jatonaz

I think it's easier to say what turns me off about a review ...
(1) Bad spelling & grammar
(2) Bad/sloppy layout
(3) half the review being about things that have got nothing really to do with the actual set ("I got this from my friend", "My dog chewed one of the minifigures", etc.)
(4) subjective statements without any supporting evidence ("I hate this set. It sucks")
(5) lists of things that are in the set (especially if listed under "pros")
(6) wasting space - putting in unnecessary blank lines between paragraphs, loads of extra blank lines at the end of the review, splitting paragraphs unnecessarily, etc.
(7) using signature lines, etc., at the end of a review
(8) ... ... ...

I second your points Bluemoose. I want to ask, on the side, what kind of reviews do the review moderators turn down? Not to criticize or sound ungrateful but I have to say at least a third of the reviews I read hit six or seven points you listed, so I was wondering why a lot of them pass. Sometimes you can tell it's the language barrier but most of the time it's just...ranting.

Especially ones like "omg this set roocks I got it at XXX on sell for 10 bucks (my brother paid half so he gets to build it)so here are the pros and cons pro: two figurs!!!! cons none goo get it!!!!"

bluemoose

Those were my criteria for ranking a review ... although I do end up editing about half the reviews I 'approve' to address one or more of the above issues. For outright deletion, they have got to be pretty much entirely without any redeeming attribute. Obviously no one Mod or Admin reviews all the posted reviews, so there will be some variability between what gets approved & what doesn't.

Huw

We probably let through more than we should, but I do bear in mind when checking them that reviews written by kids may well be interesting to other kids, if not to us.

We do get something like 20-30 reviews a day and it can be hard to read them all. I usually just skim them, to be honest.

The best thing to do if you read rubbish ones is to mark them as unhelpful, then they will disappear altogether once 5 people have marked them as such.

Town

There's no great solution to this- as marking some reviews as unhelpful does have it's benefits (to get rid of the rubbish ones mainly), but I sometimes wonder if neutral to negative reviews get marked down unjustifiably. As if the person rating the review doesn't agree with you. Or sees the lower star rating and immediately hits the unhelpful button.

I tend to be slightly critical with my reviews, I mean, why not use the full scale of the slider from 1 to 5 stars? Giving a neutral to negative review doesn't mean I hate Lego or anything... Obviously I'm a fan of Lego given that I'm signed up to and reviewing a Lego set on a Lego site, but just because of that doesn't mean I should be blindly loyal to Lego and rate every set highly. Some average-star rated sets also helps to bring the star-average back down to earth (what good are hundreds of 4.5 star rated sets?).

I'm probably over-thinking this though. All it takes is for people to hit the 'See all reviews' link, right? And the stars-ratings in hidden reviews are still factored in...

legoclone

I agree with you Town. I like a review that is not afraid to constructively criticize the set, at least a little. The ones that don’t help me so much are the ones that are just gushing about everything on the set as if there has never been a greater Lego set ever. There should always be *something* that Lego maybe could have done better, something that they can strive for in future sets – otherwise they might get lazy if they find us too easy to please.

brickmatic

Yeah, actually not using the entire scale makes reviews less useful. You lose contrast. If every set is rated 4 to 5 stars, then 1-3 don't matter really. Unfortunately, I definitely think people rate sets higher just because we are so fond of LEGO. And worse still is the publication bias problem. I don't think people rate each set they buy, or even a random sample of the sets they buy. Instead they pick out which sets to rate based on the ones they want to rate.

Actually, come to think of it, it would be cool if there was a feature in which Brickset would randomly ask you to rate a set that you own after you click on a button or something. Or just make a friendly suggestion. Imagine a pop-up that said: "Dear user, we've noticed you've been a member of Brickset for x months and haven't written a review yet. It would be swell if you reviewed this random set! Want to give it a try?"

Town

Nevermind, it seems that pressing 'See all reviews' doesn't actually show hidden reviews like I thought it would.

Anyone know if the hidden reviews are still counted towards the average?

Huw

No, they shouldn't be

zuma

I recently looked through my collection and wrote some reviews for sets/gear that did not previously have a review, rather than add reviews to sets that already had reviews.
I find that reviews for sets that already have over 10 reviews often do not really add anything new.
Perhaps introduce a limit for the number of reviews one set can have (maximum 20?), although the current 'was this review helpful' system does help remove some of these unhelpful reviews. The reviews also need to assist all ages and collection interests, so there does need to be some diversity.

meyerc13

@zuma, I like the fact that there are many reviews of a set, even if they don't add much value. For example, if I see that 40 people have reviewed a set, that tells me something (either they loved it or hated it). If I see that a set's score is based on 40 reviews, that gives it more credibility than a set with 3 reviews.

Another reason why setting a number limit isn't good... take a licensed set. There might be 20 reviews from fans of the license talking about the great minifigs and how closely the set matches the source material... but as a non-fan of the theme I might be more interested in the parts in the set, so my review might take an entirely different angle than the 40 reviews posted before me over the past 5 years since the set was released.

I also agree with Town above, I'm sure at least one of my reviews was marked as unhelpful because I didn't really think a set was all that great and my review reflected that. It's the only explanation I can find... almost every other review I've done in the same style has been marked as helpful. Plus I've seen other neutral or critical reviews marked as unhelpful as well, so I think a lot of people use the function that way. I've even used my opening paragraph to beg the reader to give me a chance to explain my ratings - and interestingly that has seemed to work.

As for what I look for in a review... I have to agree with many of the above. I've seen people do a Eurobricks style review of a minifig... if someone can do that I expect every review to have either sections or at least pros/cons well documented.

A pet peeve of mine is "I got this set on sale at TRU." OK... why do I care when that sale is over or in another country? I want to know about the set - what was good, what was bad, were the parts useful, does it fall apart when you breath on it. I don't particularly care where you got it (unless it is something like a polybag that has limited distribution) or what price you paid.

Ypres

Well before I even read the review I look at the format... because lets face it, walls of text translate into "Blah blah blah". So make reviews simple and to the point.

Consider why someone may read your review. I for one like to see if a set is worth the retail price from the opinion of an owner. If it is retired, I'll want to know the price the reviewer paid (on Ebay) and then formulate a good price range. Also in some sets I like to check whether it includes stickers. I get bored when the reviewer talks too long about set details... so again, leave it short and sweet. Always think of why the reader needs your opinion.

vynsane

A pet peeve of mine is "I got this set on sale at TRU." OK... why do I care when that sale is over or in another country?

I rarely buy LEGO at retail price, so the 'value' rating would be higher for me than it might for someone who bought at MSRP. I like to point that out at the outset, usually by means of a sentence like "I bought this set during a BOGO50% sale at TRU, so you might not find the value as high as I've rated this set."

meyerc13

I wouldn't rate you down if you qualified your information in that fashion. In my reviews I usually take the opposite approach, rate the value on MSRP, but I might say in the review, "This set is a bit overpriced, but would be a great value if you can find it on sale."

I guess what really irks me is when someone talking about what they paid seems like more of a brag than a factual part of the review.

Matthew

I absolutely hate reviews like this one for Freeing Dobby:

this is a awesome set i really liked it.
now to pros and cons
pros dobby harry potter lucious malfoy
cons none

(BTW, this review doesn't seem to be on there any more, I saved it to my computer though as an example of a bad review!)

bluemoose

^Reviews like that should never get through the moderations process; if they are, then, frankly, someone isn't doing their job properly ... on the rare occasion I get the time, I go back through the last few days worth of reviews & selectively edit or delete them. Unfortunately, none of the staff are able to work on the site full-time.

legoDad

I recently looked through my collection and wrote some reviews for sets/gear that did not previously have a review, rather than add reviews to sets that already had reviews.
I find that reviews for sets that already have over 10 reviews often do not really add anything new.
Perhaps introduce a limit for the number of reviews one set can have (maximum 20?), although the current 'was this review helpful' system does help remove some of these unhelpful reviews. The reviews also need to assist all ages and collection interests, so there does need to be some diversity.

I know I try to do reviews on sets that previous reviewers might not have covered all bases...aspects/features of the kit. Like you, if I see a ton of reviews for a particular set, I guess just about all things have been covered and try not to review it...unless though...I'm so jazzed about the kit, I just gotta' share my rant and raves.

LuciusMalfoy7

I know that if and when I review, I am sure to be thorough and to think about the set from all aspects: collecting, value for money, minifig exclusivity, parts for MOCers, playability, etc.

crzypainter

I do find it annoying as well to spend a lot of time and energy putting together a review and then have it marked as "unhelpful". Unless an opportunity is given to comment on the review, the "helpful"/"unhelpful" buttons seem arbitrary. I would like to know what someone found to be unhelpful about the review and fix it if possible. But if it just is that they didn't appreciate my style of reviewing or didn't agree with my conclusions, then it really isn't fair to penalize the review because of personal opinion.

Si_UKNZ

@crzypainter - I wouldnt worry too much, it's probably like amazon in that many of the 'not useful' ratings are just people who dont agree with the review (ie they liked the set and you slated it). Note that amazon have brought in a 'like/ dislike' feature to filter out these kind of responses.
I also would really like the ability to comment on reviews/ chat about the set in general.

technicaltactics

What I look for in reviews is proper grammer and details that convince me that I need to go and buy the set that is being reviewed. Simply a pros and cons list saying " cool shooting function ", or, " cool stikerz ", does NOT make me want to buy the set. Someone's job is not being done correctly if a review gets through saying, " this set is kool and disurvz 5 starz, best set i ever got, buy is now, wurth money ". I question the moderaters when a short review, that isn't helpful, thats two lines gets through. To me, reviews should be helpful, and look as if someone took the time to write it. Thats my 2cents on this.

Town

@crzypainter - I wouldnt worry too much, it's probably like amazon in that many of the 'not useful' ratings are just people who dont agree with the review (ie they liked the set and you slated it). Note that amazon have brought in a 'like/ dislike' feature to filter out these kind of responses.
I also would really like the ability to comment on reviews/ chat about the set in general.

True, but do 'unhelpful' Amazon reviews get permanently hidden?

prof1515

Neither. I prefer a comprehensive essay-style review discussing the set in detail and demonstrating that the reviewer has knowledge not just of the set itself but also of past sets, parts, etc. (in other words, the ability to put the set and its contents in historical context).

legoDad

I like a combination of what you both said above...a pro's/con's bullet point list for quick reading and then combine that with what the Prof say's by having the historic knowledge and wax poetic by expanding on the pro/con list.
Also, if there's some clever or neat build technique would be nice to show a pic of that part of the build.

andhe

For new sets I want to know what's new (parts, moulds, colours techniques), what they main features are, does it 'work' (ie not poorly designed) and what it will add to my existing lego. And is it value for money.

For older sets, some of the stuff mentioned above but also what made it significant at the time (was it the first set with whatever) and how it sits in relation to improving on/being improved on in other similar sets.

Ernst

Reviews I don't read and mark as "unhelpfull" are reviews without content about the set other than a link to a Youtube review. Probably a lot of work but not my idea of a "Brickset-review".

Duq

Sorry for reviving an old thread but it just happened to me again: I write a critical Star Wars review and it gets around a 50% helpful vote. I think Town is right: most of my reviews get 80% or higher helpful votes but critical Star Wars ones get much lower votes.
What I look for in a helpful review is simple: an opinion. I'm not interested in lots of photos, I don't want to read a build report. 
I'd like to read something I can't work out from looking at the set picture; what interesting or rare pieces are in the set, how interesting or repetitive is the build, do the play features work?
Other than that I'm with Bluemoose on all the things that prevent me from making it to the end of a review...

Fauch

last time I checked, I think my only unhelpful review was a star wars one.

one thing I find hard to rate is the value for money, because the price you pay for a set can vary a lot, the rating means nothing. maybe something like "what is the max you would pay for this set" would be more helpful.

Dedgecko

Fauch said:

last time I checked, I think my only unhelpful review was a star wars one.

one thing I find hard to rate is the value for money, because the price you pay for a set can vary a lot, the rating means nothing. maybe something like "what is the max you would pay for this set" would be more helpful.

Value for the money should always be against the MSRP.  Do you believe everything the set provides is worth TLG's baseline price, before any discounts are applied?

My two cents

davee123

Duq said:

What I look for in a helpful review is simple: an opinion. I'm not interested in lots of photos, I don't want to read a build report. 

I'd like to read something I can't work out from looking at the set picture; what interesting or rare pieces are in the set, how interesting or repetitive is the build, do the play features work?

Personally, I don't really want opinions, just give me the facts.  The whole point of reviews (in my mind) is to let me form my OWN opinion.  I don't generally need a reviewer to tell me all about what they think of the set.

I like reviews that show me what the set is like.  Pictures are sometimes helpful, but often the ones in the BrickSet database will do just fine.  I want to know what new elements are there, how well the functionality works, what strange building techniques might be there, or other quirky things that I wouldn't know from looking at the pictures.

I also don't want fluff.  I'd rather not read a lively tale of building the set one night, complete with a description of how the set arrived that day, and how much the author was looking forward to it, and how they had to go pull out their existing X-wing fighter model in order to swoosh it around in a dogfight with the completed model.  For me, keep it brief.  As short as you can make it, please.

If you've got an opinion on the set, that's what the little "Rating" is for.  Give it however many stars are appropriate, and presto!  I know your opinion.  If you want to justify your rating in the review, that can sometimes be handy, but brevity is key. 

Typically, if I want to know opinions, I want to get opinions from wide groups of people, which means I don't want to wade through 50 reviews to get 50 people's opinions-- I'm happy to quickly scroll through the ratings, and off-handedly read a few people's reviews to get a feel for why it got the average score it did.

DaveE