PHP 5 php-fpm memory leak?

(Fortunately, restarting php takes only a few seconds and is probably nothing more than a single slow page load for users)

You may have, and the chance just goes up as you add php modules. The easiest way to deal with the situation would be to have fpm restart the child processes after "x" number of requests, to keep their memory footprint artifically 'clean':

Sounds like:
pm.max_requests = 50 # or some other number you deem appropriate
from the documentation

Personally, I'd probably just do the above, and forget about the issue, but it may also be that the ten child processes really do end up consuming that much RAM if they are all required. You could try a 'dynamic' handling of child processes rather than the static one you have, say:

pm = dynamic
pm.max_children = 10
pm.start_servers = 2
pm.max_spare_servers = 2

This way, you'll see your heavy swapping when you need it, and it will go away when you don't. You'll see a slight penalty in performance when children aren't available to handle requests - they take a blink or two to start up.

Oh PHP, when will you and your associated libraries ever become thread safe?

Thanks for that. I'll be a while digesting it. I'm extremely green to sever admin.
But I think I get the basic concept.

I could sort of choke down how much memory php uses in threads, but then it may be that we really need that much memory so... result could be noticeable performance issues.

One thing I haven't checked yet is to see if the memory eating peaks at some point. That is, when it starts swapping, I haven't let it keep going to see how bad it gets. Maybe next week. I'm guessing if it peaks somewhere then levels off, we're probably fine. No tweaking needed, really. But if memory keeps climbing and we start seriously swap thrashing... then I either need more RAM or php config changes, or fix some of my overly hungry queries?

(One thing I'm not clear on is relationship between slow/nonindexed mysql queries and php memory consumption... maybe someone can give me the "for dummies" version?)

What may be happening is PHP is legitimately using that amount of RAM (or at least, what it would call legitimately), then theoretically it should max out at whatever magical amount of RAM it wants to use.

The problem being the amount of RAM in your VPS is less than the magical figure PHP wants to use. Shaggy's advice is (as usual) really good; you could follow that or see if decreasing the max_children means each child is able to hit that magical amount of RAM.

Update on this:
As an experiment, I upgraded to a larger Linode (from the 768 to a 1024). I really just wanted to see how high the RAM use would go and it was the easiest way to find out.
So here's the weird part: after 11 days of uptime and pretty average traffic, the used RAM never exceeds 600. Sitting at 594 right now.
Before, on the 768, it would exceed 600 in about 5 days and start swapping a bit.

So... I wonder why:
1. Having more RAM results in using less RAM
2. Or, if that's not what's happening, why it would eat up RAM so much more slowy. It may be that it's going to keep climbing but just do it really slow. At this rate, I'd guess it would have to be a few weeks yet before it starts get high enough for the swap to get busy.

I was wrong. Memory use keeps climbing... just a bit more slowly. It's up to 747 now. I'll probably try shaggy's idea soon but want to see how high it will go first.

Despite the fact that the process killer is being engaged when this type of problem happens, if I log in my server with regular SSH and check it with htop, it doesn't look like I'm out of memory at all.

Personally, I'd probably just do the above, and forget about the issue, but it may also be that the ten child processes really do end up consuming that much RAM if they are all required. You could try a 'dynamic' handling of child processes rather than the static one you have, say.