Multiple Application Pools impact on Virtual Memory and Pagefile Size

Hi

The last couple of weeks I have been assigned the task of finding out “How many Asp.Net websites can be hosted on a Windows 2003 server?”. The testing was a joint venture between Sitecore and Cohaesio (personified by Rune Mariboe). For reference the sites tested were Sitecore CMS websites.

You can’t really answer this question without making some assumtions about external factors. So we did too. The machine we tested was a 3GHz dual core machine with 4GB RAM (32 bit). We assumed that every site would generate somewhere around 15 pagehits a minute.

Each site used a SQL Express database and as our initial testing showed that each website on average resulted in an increase of about 100MB virtual memory (Public Bytes in perfmon). Knowing that we would risk OutOfMemory (OOM) exceptions if public bytes exceeded 1200MB, we chose to have a maximum of 10 sites per application pool.

Then we started testing, with increments of 10 sites per test. We managed to run 60 sites simultaneously at a total traffik of 14.79 pageviews per second. The size of Public Bytes were around 700 to 1400 MB per application pool. At this point we jumped to test 80 sites, but although response times were fine, we could see that we started to get HTTP 500 errors in the responses to MS Application Center Test. Some of the sites simply failed with OOM’s.

What actually happens when you create e.g.  8 Application pools is that you get 8 times 2 GB = 16GB virtual memory (potentially), because each application pool runs in its own process. However as the machine only has 4GB physical memory, obviously all this virtual memory cannot be held in physical memory all at once. Actually this only becomes a problem if the Asp.Net applications actually USE a lot of the 2GB virtual memory assigned. But in this case each pool did use 700-1400MB virtual memory as described above. If we just say that each pool on average used 1100MB, then 8 pools would use 8.8GB memory.

Now what Windows does when virtual memory exceeds physical memory is to swap some of the memory to the “PageFile”, which is recommended to be somewhere between 1.5 and 3 times the size of Physical memory, with a maximum of 4096 MB. In our case, the pagefile was the maximum 4096MB, so in total the system could handle 8GB virtual memory. Now, as we estimated that the application pools used 8.8GB virtual memory all in all, it is not strange that th ewebsite started to throw OOM exceptions. There has simply not been anywhere in the system that new memory could be allocated. But I guess very few get the chance to test a server to the memory limits as we did here. And quite impressive to see the practical results of such “abuse”.

Now it actually turns out that you can have more than one pagefile in Windows:

http://support.microsoft.com/default.aspx?scid=kb;en-us;237740&Product=win2000

So we tried to have 3 Pagefiles of 4GB each (on the same disk) for a total of 12GB pagefiles + 4GB physical memory. Additional testing showed that we could run 100 sites (10 app pools) without OOM’s at low traffic. However when we simulated traffic around 15 hits/minute/site, we could run 40 sites, but not 50. Page faults simply became too frequent, and the disk too slow to serve the incoming requests in a timely manner. Theory states that we should place each PageFile on its own Raid-0 disk to make sure that reading from the PageFiles was optimal, but this was a bit overkill for our test.

I believe the above tests pictures a hosting environment where a 64 bit Windows would show it’s strengths. That would allow more physical memory to be added as well as larger pagefiles.

I hope that the above has given you a little further insigt to Windows memory handling and what happens when you get too close to the memory borders. It certainly did to me.

Cheers, Jesper

Additional reading:

http://www.petri.co.il/pagefile_optimization.htm

Advertisements

2 Responses to “Multiple Application Pools impact on Virtual Memory and Pagefile Size”

  1. Christian Says:

    Interesting post 🙂

    You do not write what kind of asp.net website you are doing test against – is it Sitecore sites?

    Most websites use image, css, javascript etc. is this also including in your test?
    Most files will be cached on the client, but the will still make a hit on the server with the E-tag, so each of your *.aspx request should make 15 extra hits to static files

  2. jesperfj Says:

    Hi Christian
    Yes, these are Sitecore sites. However the focus of the article is more on what happens to the server environment at extreme memory loads rather than going into details of Sitecore. One should’t put too much into the actual numbers of sites that we cound run in this test as several conditions are artificial. E.g. the number of hits/site/minute was taken out of blue air, and as you also point to, we didn’t stress the server with requests to css, javsacript and image files during these tests.

    But if you are going to perform similar tests on your own, you should be able to find some gold nuggets that are not too obvious at first look.

    /Jesper

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: