Boost doesn't quite do that

pr0gr4mm3r — Mon, 17 Nov 2008 05:12:41 +0000

Boost doesn't quite do that I'm talking about.

From one of their site pages:

...while reducing the database access to a single query that retrieves the cached page to display.

The fact that they are calling queries at all shows that they are still using the PHP engine & a MySQL connection.

There is a better way to cache your site, and it's at the Apache level. Let's say that, using the current permalink structure on this site, you create a filesystem version as it is being browsed, so you end up with a crawled version of the site on your local file system, stored in /cache/. So by viewing this page, some PHP script would save a copy of the output to /cache/twf-feedback-and-announcements/expected-server-outages/index.html.

With a couple .htaccess entries, we can direct visitors to this cached page before PHP is even loaded for this next request.

RewriteCond %{DOCUMENT_ROOT}/cache/$1/index.html -f
RewriteRule ^(.*) /cache/$1/index.html [L]

Now the static html page will be served. The PHP extension doesn't need to load, and no MySQL connection is needed to serve the cached page. So even if your database server crashed from visitors browsing the rest of the site, the initial Digg article landing page will remain intact.

To make sure we don't cache a page with someone's signin information, place this line above the two htaccess lines above:

RewriteCond %{HTTP_COOKIE} !^.*(signin).*$

Replace "signin" with the name of the cookie you give authenticated users.

Apache is more than able to saturate several T1 lines serving html pages, and using a cache method like this should solve any server crashes due to traffic spikes.

On a footnote, there would still have to be a method of clearing out expired pages.

Are the caching and

JeevesBond — Mon, 17 Nov 2008 03:49:34 +0000

Are the caching and connection settings still up, and will you keep it that way or revert?

Am going to keep things as they are for now. Maybe tweak things back a bit and do some testing with siege.

Something that can serve static htm files in place of even calling a PHP script is best.

True. Will have to look into Boost (as mentioned above) again, see if we can use it on just aPaddedCell.

I think any caching that

pr0gr4mm3r — Fri, 14 Nov 2008 21:17:50 +0000

I think any caching that still loads the PHP engine is not good enough to handle a huge traffic spike on a VPS. Something that can serve static htm files in place of even calling a PHP script is best.

What's the status now? Are

Abhishek Reddy — Fri, 14 Nov 2008 18:19:25 +0000

What's the status now? Are the caching and connection settings still up, and will you keep it that way or revert?

Does Drupal have anything

JeevesBond — Fri, 14 Nov 2008 02:25:17 +0000

Does Drupal have anything similar to wp-super-cache?

Yes, it has Boost. We were using it for a while, but there were a few problems (not that I can remember what they are now of course).

We ended up using the APC Opcode cache, some MySQL caching (this tuning primer script is handy). However, I'm glad you mentioned that, I should check again whether Boost could at least work for aPaddedCell. The Drupal caching is good, but still couldn't handle the worst of the Digg effect.

...but it's max_connections

pr0gr4mm3r — Thu, 13 Nov 2008 23:58:30 +0000

...but it's max_connections were set quite low (80). The problem is that Drupal uses a few complicated queries, and lots of simple ones too.

Does Drupal have anything similar to wp-super-cache? This plugin for Wordpress uses generatic htm pages and .htaccess rules, so the PHP engine isn't even loaded when serving a cached page. It helps several Wordpress sites survive the Digg effect.

I met recently with

decibel.places — Thu, 13 Nov 2008 23:48:38 +0000

I met recently with developers from Northpoint Solutions in NYC who build Drupal sites capable of 1,000,000+ page views/minute.

They separate the dynamic and static content, and they use not only Drupal caching but also memcache and distributed servers.

The point is that Drupal is configurable for heavy traffic.

I thought Jeeves mentioned some kind of cache persistence problem a while back, maybe you left caching off?

Why would MySQL be crashing

JeevesBond — Thu, 13 Nov 2008 23:45:36 +0000

Why would MySQL be crashing by a load spike such as this.

Well, it wasn't crashing, but it's max_connections were set quite low (80). The problem is that Drupal uses a few complicated queries, and lots of simple ones too. So we've got MySQL optimised to cache tables/queries, meaning it doesn't take many connections to gobble all the RAM on our server.

There was approximately a five-minute interval, as the story hit the front page of Digg, when everything went haywire. Otherwise the server seemed happy to handle 5-10 pages per second.

Don't you have some sort of caching enabled?

Yes, an article page only uses one SQL query. This probably saved the VPS from getting utterly destroyed.

(Time to up your advertising

pr0gr4mm3r — Thu, 13 Nov 2008 22:56:50 +0000

(Time to up your advertising rates?)

Easy for you to say.

Why would MySQL be crashing by a load spike such as this. Don't you have some sort of caching enabled?

(Time to up your advertising

JeevesBond — Thu, 13 Nov 2008 22:18:00 +0000

(Time to up your advertising rates?)

I wish! Digg users all have adblock switched on. Still, we may get some decent backlinks to that article, and maybe a few more visitors.

Edit:
top - 22:19:05 up 2:49, 2 users, load average: 0.81, 0.61, 0.60

Looks like we're through the worst of it.