Expected server outage(s)

JeevesBond's picture

He has: 3,956 posts

Joined: Jun 2002

Megan's '21 ways to streamline your CSS' article is currently on Digg (no, we didn't submit it, so it's not our fault! Smiling ).

This has so far caused two server outages: the first because I had Apache's MaxClients set waaaay too low.

The second outage was due to a huge influx of visitors from Digg, causing the server to get load averages of ~100 and MySQL to get too many connections. Since the second outage, I've tweaked down Apache's MaxClients and increased MySQL's max_connections. The VPS is handling the current load nicely:

load average: 0.87, 3.16, 15.61

However, another large influx of visitors, will probably kill the VPS again. I'm monitoring closely, and will step-in and rescue the server if possible.

The forums may go down for a short period of time today. Blame Digg. Laughing out loud

a Padded Cell our articles site!

JeevesBond's picture

He has: 3,956 posts

Joined: Jun 2002

top - 21:13:12 up  1:43,  2 users,  load average: 2.50, 2.24, 5.35

Forgot to mention, I've also stopped courier, postfix, cyrus-sasl and any other services not essential to the OS or directly related to serving Web pages. Smiling

a Padded Cell our articles site!

decibel.places's picture

He has: 1,494 posts

Joined: Jun 2008

Your success is killing you! Shocked

(Time to up your advertising rates?)

JeevesBond's picture

He has: 3,956 posts

Joined: Jun 2002

(Time to up your advertising rates?)

I wish! Digg users all have adblock switched on. Still, we may get some decent backlinks to that article, and maybe a few more visitors. Smiling

Edit:
top - 22:19:05 up  2:49,  2 users,  load average: 0.81, 0.61, 0.60

Looks like we're through the worst of it.

a Padded Cell our articles site!

pr0gr4mm3r's picture

He has: 1,502 posts

Joined: Sep 2006

(Time to up your advertising rates?)

Easy for you to say. Sticking out tongue

Why would MySQL be crashing by a load spike such as this. Don't you have some sort of caching enabled?

decibel.places's picture

He has: 1,494 posts

Joined: Jun 2008

I met recently with developers from Northpoint Solutions in NYC who build Drupal sites capable of 1,000,000+ page views/minute.

They separate the dynamic and static content, and they use not only Drupal caching but also memcache and distributed servers.

The point is that Drupal is configurable for heavy traffic.

I thought Jeeves mentioned some kind of cache persistence problem a while back, maybe you left caching off?

JeevesBond's picture

He has: 3,956 posts

Joined: Jun 2002

Why would MySQL be crashing by a load spike such as this.

Well, it wasn't crashing, but it's max_connections were set quite low (80). The problem is that Drupal uses a few complicated queries, and lots of simple ones too. So we've got MySQL optimised to cache tables/queries, meaning it doesn't take many connections to gobble all the RAM on our server.

There was approximately a five-minute interval, as the story hit the front page of Digg, when everything went haywire. Otherwise the server seemed happy to handle 5-10 pages per second.

Don't you have some sort of caching enabled?

Yes, an article page only uses one SQL query. This probably saved the VPS from getting utterly destroyed. Smiling

a Padded Cell our articles site!

pr0gr4mm3r's picture

He has: 1,502 posts

Joined: Sep 2006

...but it's max_connections were set quite low (80). The problem is that Drupal uses a few complicated queries, and lots of simple ones too.

Does Drupal have anything similar to wp-super-cache? This plugin for Wordpress uses generatic htm pages and .htaccess rules, so the PHP engine isn't even loaded when serving a cached page. It helps several Wordpress sites survive the Digg effect.

JeevesBond's picture

He has: 3,956 posts

Joined: Jun 2002

Does Drupal have anything similar to wp-super-cache?

Yes, it has Boost. We were using it for a while, but there were a few problems (not that I can remember what they are now of course).

We ended up using the APC Opcode cache, some MySQL caching (this tuning primer script is handy). However, I'm glad you mentioned that, I should check again whether Boost could at least work for aPaddedCell. The Drupal caching is good, but still couldn't handle the worst of the Digg effect.

a Padded Cell our articles site!

Abhishek Reddy's picture

He has: 3,348 posts

Joined: Jul 2001

What's the status now? Are the caching and connection settings still up, and will you keep it that way or revert?

pr0gr4mm3r's picture

He has: 1,502 posts

Joined: Sep 2006

I think any caching that still loads the PHP engine is not good enough to handle a huge traffic spike on a VPS. Something that can serve static htm files in place of even calling a PHP script is best.

JeevesBond's picture

He has: 3,956 posts

Joined: Jun 2002

Are the caching and connection settings still up, and will you keep it that way or revert?

Am going to keep things as they are for now. Maybe tweak things back a bit and do some testing with siege.

Something that can serve static htm files in place of even calling a PHP script is best.

True. Will have to look into Boost (as mentioned above) again, see if we can use it on just aPaddedCell. Smiling

a Padded Cell our articles site!

pr0gr4mm3r's picture

He has: 1,502 posts

Joined: Sep 2006

Boost doesn't quite do that I'm talking about.

From one of their site pages:

...while reducing the database access to a single query that retrieves the cached page to display.

The fact that they are calling queries at all shows that they are still using the PHP engine & a MySQL connection.

There is a better way to cache your site, and it's at the Apache level. Let's say that, using the current permalink structure on this site, you create a filesystem version as it is being browsed, so you end up with a crawled version of the site on your local file system, stored in /cache/. So by viewing this page, some PHP script would save a copy of the output to /cache/twf-feedback-and-announcements/expected-server-outages/index.html.

With a couple .htaccess entries, we can direct visitors to this cached page before PHP is even loaded for this next request.

RewriteCond %{DOCUMENT_ROOT}/cache/$1/index.html -f
RewriteRule ^(.*) /cache/$1/index.html [L]

Now the static html page will be served. The PHP extension doesn't need to load, and no MySQL connection is needed to serve the cached page. So even if your database server crashed from visitors browsing the rest of the site, the initial Digg article landing page will remain intact.

To make sure we don't cache a page with someone's signin information, place this line above the two htaccess lines above:

RewriteCond %{HTTP_COOKIE} !^.*(signin).*$

Replace "signin" with the name of the cookie you give authenticated users.

Apache is more than able to saturate several T1 lines serving html pages, and using a cache method like this should solve any server crashes due to traffic spikes.

On a footnote, there would still have to be a method of clearing out expired pages.

Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.