Brickset.com site issues

Huw

Access to the site is intermittent today as you may have noticed.

It's due to problems with the network/firewall infrastructure which is preventing access to the database, apparently, and the fix is to move the database to a new server.

I'm hoping that it's going to be done today.

If the site hasn't stabilised by Monday I will delay roll-out of the new site.

Huw

Work continues on resolving the issues, but it's taking far too long and for that I apologise.

The DNS change has been made so hopefully that will propagate through the 'net over the next few hours so when it does come up you'll be able to find it.

Huw

The ISP is still having some sort of problem, I don't know what, which is causing the delay.

I can't do anything more than apologise and it's very frustrating!

Huw

I believe the site, the old one at least, is up and running although I can't connect to it, possibly because of the DNS change.

Could you reply here if you can access it, please?

rocao

Still down for me as of 12:45am PST (UTC -8)
And I don't think members can post here :P

Matthew

Old site currently working for me.

Yellowcastle

Unable to access either for me at 9:07am EST

drdavewatford

Not me.

Huw

I turned the new site off earlier. I'm out for the day so can't test anything but clearly somethings still not right. I'm at my wit's end...

Huw

Sorry all you should be able to comment here now.

leego76

@Huw, is there a way to 'choose' the old site? (using an iPhone)

mgemie

Managed to acess the site today and it is good to be back. Fingers crossed all is well now.

Paperballpark

Still down for me here. :(

@leego76 just go to www.brickset.com - huw hasn't changed that over to the 'new' site yet.

leego76

@Paperballpark - I was trying the main page but it just doesn't load up, I can wait. I was hoping to access the amazon page though, don't want to miss any bargains!

TheOneVeyronian

I haven't been able to access neither new nor old Brickset for 48 hours now. There's no "page not found" either, it just sits there and loads (perpetually) but nothing ever comes up, it just stays on a blank page. This is using IE11 by the way.

I'm just hoping all our data doesn't get wiped from our accounts because of these issues.

CapnRex101

I am in the same boat as @TheOneVeyronian, I have tried often to get in, but without success. I have had a couple of issues with the forum on occasion as well.

Paperballpark

Yeah the forum isn't loading the stylesheet at all this evening - I'm guessing it pulls that from brickset.com, although it was fine earlier, even though, as others have said, brickset.com itself hasn't been accessible for me for about 2 days now.

TheOneVeyronian

^^ Edit that, tried to load it again after posting that comment, a "page cannot be displayed" error came up after about a minute of loading.

rancorbait

@CapnRex101 Same here. Hasn't worked at all after Thursday. And the forum is very slow, it can take a few minutes just to load a page!

Paperballpark

ooh, the forum stylesheet has loaded now! Still not able to access the main site though :(

EDIT: aaand now it's not loading the stylesheet again. Oh well, I was getting hopeful for a moment.

Paperballpark

The website http://downforeveryoneorjustme.com/ says that www.brickset.com is up, and that it's just me. I'm not sure where it's based, but that gives me hope that the DNS change is propagating out through the net :)

DrLegOBrick

I've been unable to gain access for three days I believe, both new, old and the forum. I feel incomplete without my Brickset fix every day, I'll be crawling the walls soon...

khmellymel

The only thing that really works for me is accessing the forum on a mobile device. I've had no luck with the main site I'm afraid

Huw

I think they believe there's a DoS underway which isn't helping things...

Paperballpark

^ Oh dear! :(

@khmellymel the forum still works on a desktop computer - just use the dedicated website address, www.bricksetforum.com

canuck

Paperballpark said:

The website http://downforeveryoneorjustme.com/ says that www.brickset.com is up, and that it's just me. I'm not sure where it's based, but that gives me hope that the DNS change is propagating out through the net :)

Now says its down for everyone.

canuck

^Using chrome in Atlantic Canada if that means anything.

Legopants

Can just about access the Forum from my mobile.
Haven't been able to access the main site or Forum from my PC since Thursday :-S

akunthita

Huw said:

I think they believe there's a DoS underway which isn't helping things...

Huw, what do you mean? Are they telling you now that all along it was a DOS attack and not an issue with the database/firewall. If they made you transfer the entire site over this their tech people must be really incompetent. But if it is a DOS attack isn't changing the DNS would have taken care of that?

Also, most DOS attacks are against big news sites, banks, government websites, etc., not a hobby site. It takes some effort to pull off a DOS attack and I don't think anyone would go through all that trouble just to attack an innocent LEGO fansite. Unless we angered some MegaBlock fans...(c;

It may be just a regular spam attack that has gone a bit haywire. Or perhaps a DOS attack against the host itself. I hope you can sort this out soon. If you need, I can get you in touch with my tech person who is really good with this kind of stuff. He is the guy who saved Pat Flynn's website after a similar situation. He is from the UK.

red237

No problems reaching the forum on any venue as long as I google search "brickset forum". Cannot reach the site, it just loads and loads and loads. Occasionally I will get a peek at the homepage, but can't select anything....good luck. We miss it!

Aleydita

I think you need a new host :D

4 days and counting here in Belgium and no access to the main site.

rancorbait

Just tried again and it works! YEEEEESSSSSSSSS!! The precious is working! *happy dance* :o)

Paperballpark

Yay it's working again! :)

Stvoyager04

Homepage is loading but I can't seem to go to any other pages.

Huw

OK, so here's the story.

The site was hosted by Northstar, run by Eric Smith, now BrickLink admin. He sold the business to OnRamp Indiana a year or so ago. It remained on the same hardware and has been running perfectly well, until Monday last week.

Then I started seeing loads of network related errors from the front-end which couldn't connect to the back-end database, and database timeouts.

ORI said the Northstar network infrastructure was to blame and that the solution was to move it to their servers/network. That started on Thursday. It should have been a straightforward job but it took 3 days, until yesterday when they finally got it up and running. However it's now using a SQL Express back end instead of SQL2008, on the same server as the front-end, which is not ideal, because they claimed that it was overloading their SQL server. It was still incredibly slow as the CPU was constantly 100%. It *could* be because of a DoS attack overloading things, I'm personally not convinced and I will investigate further today.

As it stands now, I've taken the old site offline and run up the new one (at new.brickset.com) and the CPU is averaging 25% with c.250 users so it appears to be holding up.

*If* it is a DoS then the pages the attacker is requesting on the old site won't be served which will prevent the database and server being overloaded. I'l leave in that state today to see what happens. Do let me know how it is for you.

I am thinking of moving to a managed and monitored hosting solution with guaranteed uptime and SLAs, from a UK based company. (The US/UK time difference has proved frustrating). It will be far more expensive but obviously need to do something given the site is my livelihood now.

Anyway, thank you for your patience. I sincerely hope I can get everything back to normal soon.

TheOneVeyronian

I can only access the homepage on the new site - nothing else at all (it just perpetually loads) so I can't even check to see if all these problems has ended up wiping out my collection data. (those who do seem to have access beyond the main page - please tell me this hasn't happened!)

I know it's better than nothing, but just having access to the homepage is almost as bad as having no Brickset at all :-( so I hope everything is returned to normal soon.

But great job on keeping us updated Huw :-)

Huw

No data has or will be lost: I have backups on my local PC taken every day last week up to 31/1/14. Nothing entered yesterday or today should be lost either, but I would advise against making a lot of changes just in case.

It turns out there *is* a DoS underway, from a German IP. I've taken steps to minimise its effect but more needs to be done because I suspect the requests it's making on the server is hampering legitimate network traffic.

MrBerre

Huw said:

However it's now using a SQL Express back end instead of SQL2008, on the same server as the front-end, which is not ideal

Yikes.

Duq

DoS attacks don't come from one IP, that sounds more like someone has written a bad scraper.
Brickset on Sql Express on the Web server is bonkers. You're right Huw, time to move.

Are we ok to post new reviews now or best to wait?

richo

The forum still seems to work a lot better on my phone than ipad. Not sure why there should be a difference?

drdavewatford

The Forum has started to work much better for me over the past 30 minutes - the style sheet is back in place and page transitions are much faster; it almost feels back to normal.

New Brickset is also showing encouraging signs of life - some of the functionality has returned, although certain pages are still timing out and the UK Amazon Discount listing looks a couple of days out of date. Definitely headed in the right direction, though - what a relief !

*UPDATE 11.25 a.m.: the UK Amazon discount listing just successfully updated for the first time since 30th January so definite signs of progress*

hantot

While the site is having issues I've stopped anything of mine that uses the API just in case

KingDave

I can see the new site now, but I won't let me log in at all (meaning it won't load the log in window, I can't even try to log in).

Huw

I still can't get beyond the home page...

TheLoneTensor

So when exactly do you make the decision to switch and get hosted on a better service that can a) withstand the most basic internet attacks and b) recover from said attacks in hours, not days or weeks?

gratefulnat

Navigating the main menu seems to work pretty well for me.
I'm able to browse sets, year & theme, read reviews, read news, etc.

I was able to login earlier today, but couldn't view my_sets. (Yesterday on the old sight viewed my_sets with the result I own 0 sets.)

Can't login anymore though, site not responding right now.

Huw & team, you are amazing.
Take a break and indulge in a good single malt whiskey, sometimes it helps!
...and then get back to work ;-)

RikTheVeggie

The new site seems to be working here but a bit slow at times.
Sounds like PaaS might be the way to go, I guess Azure is the baseline reference service but EC2 might be an option with servers in Ireland.

If you can isolate the DoS to a single IP, the normal resolution is to just block it. And/or run a reverse DNS lookup and contact the ISP.

Goony

For someone with no knowledge of computers save the simple everyday use, this makes no sense :) Can anyone please explain it in simple terms for a simple guy? :)

JasonLEGO

Have you thought about using Amazon Web Services or something similar Huw? The site would be load balanced across the globe and a much more reliable and a better user experience. No idea what you are paying now, might be comparable in price.

Huw

I have just enabled a CDN/security gateway that apparently neutralises threats before they get very far: it required another DNS change which is still propagating. Certainly when I test it at http://network-tools.com/default.asp?prog=express&host=brickset.com it's made a heck of a difference in terms of hops and timings and should cache much of the static content and thus reduce bandwidth/load on the server.

Cam_n_Stu

@Goony, do you mean in general or RiKTheVeggie's post in particular?

Assuming the former, here is an attempt to turn Huw's post into 'English' ;) I am sure someone can improve the readability of this further...

The site was hosted by Northstar (a US based IT company that was run by a Lego fan. It provided the physical computer hardware that the Brickset.com website runs on). Northstar was sold to OnRamp Indiana (ORI is a slightly bigger IT company, which is also an Internet Service Provider). It remained on the same hardware and has been running perfectly well, until Monday last week.

Then I started seeing loads of network related errors from the front-end (the webserver) which couldn't connect to the back-end database. (The back-end database holds the records of all the sets and the data associated with them, i.e. what is in your collection, my collection, and every other user etc, along with a lot of other data that it provides to the webserver front-end to create webpages when you request them.)

ORI said the Northstar network infrastructure was to blame and that the solution was to move it to their servers/network. That started on Thursday. It should have been a straightforward job but it took 3 days, until yesterday when they finally got it up and running. However it's now using a SQL Express back end instead of SQL2008, on the same server as the front-end, which is not ideal, because they claimed that it was overloading their SQL server. It was still incredibly slow as the CPU was constantly 100%. It *could* be because of a DoS attack overloading things, I'm personally not convinced and I will investigate further today. (Translation: ORI have bodged a quick and dirty solution with everything (front-end webserver software and back-end database) all on one server - not the industry standard way to do it and not very scalable when you have lots of users - it is better for a number of reasons to have the front-end web server deal just with webpage serving and a separate back-end database server deal with the the data itself.)

As it stands now, I've taken the old site offline and run up the new one (at new.brickset.com) and the CPU is averaging 25% with c.250 users so it appears to be holding up. (It could be worse considering the above comment!)

*If* it is a DoS then the pages the attacker is requesting on the old site won't be served which will prevent the database and server being overloaded. I'l leave in that state today to see what happens. Do let me know how it is for you. (DoS is short for 'Denial of Service' attack, which in simple terms is when someone tries to crash or degrade a websites performance by repeatedly requesting the same page many times a second. When people say DoS they usually mean DDoS, which is a Distributed DoS attack where the attacker uses 10s, 100s or 1000s of PCs, rather than just the one, to make those malicious page requests. Large Internet providers and Web Hosting companies have specialised systems to mitigate these types of attacks.)

I am thinking of moving to a managed and monitored hosting solution with guaranteed uptime and SLAs, from a UK based company. (The US/UK time difference has proved frustrating). It will be far more expensive but obviously need to do something given the site is my livelihood now. (Given this is a 'business critical' site to Huw now, I'd agree it needs to be hosted by a serious player with the ability to deliver any uptime they promise and if something goes wrong have the spare capacity and expertise to fix the problem sharpish. Anyone with 'real' 24x7 support would be an improvement by the sound of it. Personally I'd look for someone that has a reputation for exceptional customer support be they in the US or the UK.