Strike That

Well, I've just been informed that I'm a liar, or misspoke if you prefer.

I didn't know when I made the previous post that we are actually still using 1,000 as our MaxRequestsPerChild setting. I guess I misunderstood what I was being told last week, and I certainly did not bother to check our configuration before making the previous blog post.

I apologize for any confusion this may have caused. However, the general observation that you may have to experiment with your Apache settings before you find the right combination for your server and its usage patterns still stands. You can't always expect the defaults to work right out of the box.

More Apache Fun

Another update to share with you what MVLC is doing with our Apache configuration for Evergreen.

We found last week with the MaxMemFree setting at 16 and MaxRequestsPerChild at 1,000 that we were getting more texts from our monitoring software about the load being high on our Evergreen server. We thought this might have to do with more frequent turnover among the Apache child processes, so we adjusted MaxRequestsPerChild back up to 10,000. However, during overnight monitoring, we discovered that this made the situation worse, or at least put us back where we were before trying all of these changes.

In the end, we have set MaxRequestsPerChild to 5,000 while leaving MaxMemFree at 16. We've been running this configuration for several days, including over the weekend, and things seem to have really settled down on our server. You may have to experiment with the settings to find something that works for you if you think you are having this issue.

Update on the Apache Situation

Thomas just told me that he changed another Apache configuration variable that seems to have helped things. He set the MaxMemFree directive to 16 in our mpm_prefork configuration section. This setting also limits the memory that an individual Apache process can consume before releasing the memory back to the operating system.

I thought we'd share this in case anyone else is bumping their heads against memory issues with Apache.

What has been going on.

TL;DR: We've had trouble with the memory consumption of Apache processes on our Evergreen server since we did our latest update on April 14, 2013. Along the way to figuring this out we've had a few minor detours and fixed another bug. Our breakthrough came when one of us realized that the longer Apache processes run the more memory they were using. We have made changes to our Apache configuration as a mitigation strategy. Basically, we have lowered our MaxRequestsPerChild from 10,000 down to 1,000. This appears to have helped, but only time will tell.

Read on for the gory details....