Archive for September, 2007

Multiple Application Pools impact on Virtual Memory and Pagefile Size

September 21, 2007

Hi

The last couple of weeks I have been assigned the task of finding out “How many Asp.Net websites can be hosted on a Windows 2003 server?”. The testing was a joint venture between Sitecore and Cohaesio (personified by Rune Mariboe). For reference the sites tested were Sitecore CMS websites.

You can’t really answer this question without making some assumtions about external factors. So we did too. The machine we tested was a 3GHz dual core machine with 4GB RAM (32 bit). We assumed that every site would generate somewhere around 15 pagehits a minute.

Each site used a SQL Express database and as our initial testing showed that each website on average resulted in an increase of about 100MB virtual memory (Public Bytes in perfmon). Knowing that we would risk OutOfMemory (OOM) exceptions if public bytes exceeded 1200MB, we chose to have a maximum of 10 sites per application pool.

Then we started testing, with increments of 10 sites per test. We managed to run 60 sites simultaneously at a total traffik of 14.79 pageviews per second. The size of Public Bytes were around 700 to 1400 MB per application pool. At this point we jumped to test 80 sites, but although response times were fine, we could see that we started to get HTTP 500 errors in the responses to MS Application Center Test. Some of the sites simply failed with OOM’s.

What actually happens when you create e.g.  8 Application pools is that you get 8 times 2 GB = 16GB virtual memory (potentially), because each application pool runs in its own process. However as the machine only has 4GB physical memory, obviously all this virtual memory cannot be held in physical memory all at once. Actually this only becomes a problem if the Asp.Net applications actually USE a lot of the 2GB virtual memory assigned. But in this case each pool did use 700-1400MB virtual memory as described above. If we just say that each pool on average used 1100MB, then 8 pools would use 8.8GB memory.

Now what Windows does when virtual memory exceeds physical memory is to swap some of the memory to the “PageFile”, which is recommended to be somewhere between 1.5 and 3 times the size of Physical memory, with a maximum of 4096 MB. In our case, the pagefile was the maximum 4096MB, so in total the system could handle 8GB virtual memory. Now, as we estimated that the application pools used 8.8GB virtual memory all in all, it is not strange that th ewebsite started to throw OOM exceptions. There has simply not been anywhere in the system that new memory could be allocated. But I guess very few get the chance to test a server to the memory limits as we did here. And quite impressive to see the practical results of such “abuse”.

Now it actually turns out that you can have more than one pagefile in Windows:

http://support.microsoft.com/default.aspx?scid=kb;en-us;237740&Product=win2000

So we tried to have 3 Pagefiles of 4GB each (on the same disk) for a total of 12GB pagefiles + 4GB physical memory. Additional testing showed that we could run 100 sites (10 app pools) without OOM’s at low traffic. However when we simulated traffic around 15 hits/minute/site, we could run 40 sites, but not 50. Page faults simply became too frequent, and the disk too slow to serve the incoming requests in a timely manner. Theory states that we should place each PageFile on its own Raid-0 disk to make sure that reading from the PageFiles was optimal, but this was a bit overkill for our test.

I believe the above tests pictures a hosting environment where a 64 bit Windows would show it’s strengths. That would allow more physical memory to be added as well as larger pagefiles.

I hope that the above has given you a little further insigt to Windows memory handling and what happens when you get too close to the memory borders. It certainly did to me.

Cheers, Jesper

Additional reading:

http://www.petri.co.il/pagefile_optimization.htm