Brickset ForumPlease refrain from posting animated GIFs, memes, joke videos and so on in discussions other than those in the off topic area.
Dismiss this message to confirm your acceptance of this additional forum term of use.
Dismiss this message to confirm your acceptance of this additional forum term of use.
Brickset.com site issues
Huw
Brickset Towers, Hampshire, UKAdministrator Posts: 6,334
Access to the site is intermittent today as you may have noticed.
It's due to problems with the network/firewall infrastructure which is preventing access to the database, apparently, and the fix is to move the database to a new server.
I'm hoping that it's going to be done today.
If the site hasn't stabilised by Monday I will delay roll-out of the new site.
It's due to problems with the network/firewall infrastructure which is preventing access to the database, apparently, and the fix is to move the database to a new server.
I'm hoping that it's going to be done today.
If the site hasn't stabilised by Monday I will delay roll-out of the new site.
0
Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Quick Links
Categories
- 24K All Categories
- 2 Announcements
- 2 Current Announcements
- 9.6K General
- 2.9K Collecting
- 3.3K Everything else LEGO
- 1.5K Building and Techniques
- 12.5K Shopping
- 1.6K Shopping USA/Canada
- 321 Shopping Elsewhere
- 9.8K Marketplace
- 62 Charity Sales
- 2K Brickset
- 70 News
- 13 Ambassador
- 2 Announcements
- 959 Introduce yourself
- 296 The Database
- 377 Brickset.com
- 268 Forum Operation
- 15 FAQs
- 1.1K Other
- 532 Community and Events
- 13 Fairy Bricks
- 249 Photography/Video
Comments
The DNS change has been made so hopefully that will propagate through the 'net over the next few hours so when it does come up you'll be able to find it.
I can't do anything more than apologise and it's very frustrating!
Could you reply here if you can access it, please?
And I don't think members can post here :P
@leego76 just go to www.brickset.com - huw hasn't changed that over to the 'new' site yet.
I'm just hoping all our data doesn't get wiped from our accounts because of these issues.
EDIT: aaand now it's not loading the stylesheet again. Oh well, I was getting hopeful for a moment.
@khmellymel the forum still works on a desktop computer - just use the dedicated website address, www.bricksetforum.com
Haven't been able to access the main site or Forum from my PC since Thursday :-S
Also, most DOS attacks are against big news sites, banks, government websites, etc., not a hobby site. It takes some effort to pull off a DOS attack and I don't think anyone would go through all that trouble just to attack an innocent LEGO fansite. Unless we angered some MegaBlock fans...(c;
It may be just a regular spam attack that has gone a bit haywire. Or perhaps a DOS attack against the host itself. I hope you can sort this out soon. If you need, I can get you in touch with my tech person who is really good with this kind of stuff. He is the guy who saved Pat Flynn's website after a similar situation. He is from the UK.
4 days and counting here in Belgium and no access to the main site.
The site was hosted by Northstar, run by Eric Smith, now BrickLink admin. He sold the business to OnRamp Indiana a year or so ago. It remained on the same hardware and has been running perfectly well, until Monday last week.
Then I started seeing loads of network related errors from the front-end which couldn't connect to the back-end database, and database timeouts.
ORI said the Northstar network infrastructure was to blame and that the solution was to move it to their servers/network. That started on Thursday. It should have been a straightforward job but it took 3 days, until yesterday when they finally got it up and running. However it's now using a SQL Express back end instead of SQL2008, on the same server as the front-end, which is not ideal, because they claimed that it was overloading their SQL server. It was still incredibly slow as the CPU was constantly 100%. It *could* be because of a DoS attack overloading things, I'm personally not convinced and I will investigate further today.
As it stands now, I've taken the old site offline and run up the new one (at new.brickset.com) and the CPU is averaging 25% with c.250 users so it appears to be holding up.
*If* it is a DoS then the pages the attacker is requesting on the old site won't be served which will prevent the database and server being overloaded. I'l leave in that state today to see what happens. Do let me know how it is for you.
I am thinking of moving to a managed and monitored hosting solution with guaranteed uptime and SLAs, from a UK based company. (The US/UK time difference has proved frustrating). It will be far more expensive but obviously need to do something given the site is my livelihood now.
Anyway, thank you for your patience. I sincerely hope I can get everything back to normal soon.
I know it's better than nothing, but just having access to the homepage is almost as bad as having no Brickset at all :-( so I hope everything is returned to normal soon.
But great job on keeping us updated Huw :-)
It turns out there *is* a DoS underway, from a German IP. I've taken steps to minimise its effect but more needs to be done because I suspect the requests it's making on the server is hampering legitimate network traffic.
Brickset on Sql Express on the Web server is bonkers. You're right Huw, time to move.
Are we ok to post new reviews now or best to wait?
New Brickset is also showing encouraging signs of life - some of the functionality has returned, although certain pages are still timing out and the UK Amazon Discount listing looks a couple of days out of date. Definitely headed in the right direction, though - what a relief !
*UPDATE 11.25 a.m.: the UK Amazon discount listing just successfully updated for the first time since 30th January so definite signs of progress*
I'm able to browse sets, year & theme, read reviews, read news, etc.
I was able to login earlier today, but couldn't view my_sets. (Yesterday on the old sight viewed my_sets with the result I own 0 sets.)
Can't login anymore though, site not responding right now.
Huw & team, you are amazing.
Take a break and indulge in a good single malt whiskey, sometimes it helps!
...and then get back to work ;-)
Sounds like PaaS might be the way to go, I guess Azure is the baseline reference service but EC2 might be an option with servers in Ireland.
If you can isolate the DoS to a single IP, the normal resolution is to just block it. And/or run a reverse DNS lookup and contact the ISP.
Assuming the former, here is an attempt to turn Huw's post into 'English' ;) I am sure someone can improve the readability of this further...
The site was hosted by Northstar (a US based IT company that was run by a Lego fan. It provided the physical computer hardware that the Brickset.com website runs on). Northstar was sold to OnRamp Indiana (ORI is a slightly bigger IT company, which is also an Internet Service Provider). It remained on the same hardware and has been running perfectly well, until Monday last week.
Then I started seeing loads of network related errors from the front-end (the webserver) which couldn't connect to the back-end database. (The back-end database holds the records of all the sets and the data associated with them, i.e. what is in your collection, my collection, and every other user etc, along with a lot of other data that it provides to the webserver front-end to create webpages when you request them.)
ORI said the Northstar network infrastructure was to blame and that the solution was to move it to their servers/network. That started on Thursday. It should have been a straightforward job but it took 3 days, until yesterday when they finally got it up and running. However it's now using a SQL Express back end instead of SQL2008, on the same server as the front-end, which is not ideal, because they claimed that it was overloading their SQL server. It was still incredibly slow as the CPU was constantly 100%. It *could* be because of a DoS attack overloading things, I'm personally not convinced and I will investigate further today. (Translation: ORI have bodged a quick and dirty solution with everything (front-end webserver software and back-end database) all on one server - not the industry standard way to do it and not very scalable when you have lots of users - it is better for a number of reasons to have the front-end web server deal just with webpage serving and a separate back-end database server deal with the the data itself.)
As it stands now, I've taken the old site offline and run up the new one (at new.brickset.com) and the CPU is averaging 25% with c.250 users so it appears to be holding up. (It could be worse considering the above comment!)
*If* it is a DoS then the pages the attacker is requesting on the old site won't be served which will prevent the database and server being overloaded. I'l leave in that state today to see what happens. Do let me know how it is for you. (DoS is short for 'Denial of Service' attack, which in simple terms is when someone tries to crash or degrade a websites performance by repeatedly requesting the same page many times a second. When people say DoS they usually mean DDoS, which is a Distributed DoS attack where the attacker uses 10s, 100s or 1000s of PCs, rather than just the one, to make those malicious page requests. Large Internet providers and Web Hosting companies have specialised systems to mitigate these types of attacks.)
I am thinking of moving to a managed and monitored hosting solution with guaranteed uptime and SLAs, from a UK based company. (The US/UK time difference has proved frustrating). It will be far more expensive but obviously need to do something given the site is my livelihood now. (Given this is a 'business critical' site to Huw now, I'd agree it needs to be hosted by a serious player with the ability to deliver any uptime they promise and if something goes wrong have the spare capacity and expertise to fix the problem sharpish. Anyone with 'real' 24x7 support would be an improvement by the sound of it. Personally I'd look for someone that has a reputation for exceptional customer support be they in the US or the UK.