Archive for the ‘ASP.Net Performance’ Category

Asp.Net Memory Limit – part 2

December 10, 2007

Interesting article on the history of the memory limit. As can be seen, the memory limt for 64 bit Windows is recommended to be MIN(60%, 1TB).

http://blogs.msdn.com/tmarq/archive/2007/06/25/some-history-on-the-asp-net-cache-memory-limits.aspx 

Jesper Jørgensen, www.sitecore.net

Asp.Net Memory Limit

December 10, 2007

I saw this question today from Nick Simpson at osdir.com. I couldn’t see where to reply so I hope it is okay that I take it here:

Subject: ASP .NET Memory Limit.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

I have a couple of web sites on a single Windows 2003 server running ASP.NET 1.1.
The /3GB switch is set in boot.ini and the ASPNETENABLE3GB environment variable is also set. The machine.config has the default memoryLimit on the processModel node of 60.  So does that mean that my web apps have up to a maximum of 3GB RAM to play with each – 1.8GB max if i leave the machine.config setting as it is – expandable to a full 3GB per web app (assuming separate app pools) without worry of adversely affecting performance on that box? 

Is this limit per instance of the w3wp process, or for ASP .NET overall, or indeed for applications on that box overall?

I thought this was per process until i observed our box (with two sites – at 400MB and 800MB – separate app pools) throw loads of OutOfMemory exceptions. This was prior to the /3GB switch being in place – the limit would have been 1.2GB then – 60% of 2GB – but OutOfMemory exceptions were thrown when the combined amount of memory usage for all w3wp process instances were getting near to 1.2GB, and not when either of the two web sites individually were reaching 1.2GB. Am i misunderstanding something or is the memory limit for ASP .NET overall? 

I have read the following article but still find my questions unanswered: http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx  Our server currently has 4GB RAM. We are getting a new one. Would it just be a waste of money getting a new server with 8GB RAM?

Will ASP .NET 2.0 and 64 bit Windows add anything new in this respect?  Thanks for your help,  Nick 

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Hi Nick

In man article called “Understanding Asp.Net memory” here on my blog, I described how memoryLimit was changed from machine.config to a setting called “Maximum memory used” in the IIS manager of IIS 6.0 which is default in Windows 2003. So instead of setting a percentage you just set this setting to an amount of megabytes specific to each application pool. When using memoryLimit in IIS 5.0 you should be aware that Microsoft recommends that it is set to “the smaller of 60% and 800 MB” which is 800 MB on a machine with 2GB memory. Leaving it at 60% equal to 1200 MB increase the risk of OOM errors. Not that I believe this is the cause to your problems as I guess you are running IIS 6.0.

On the other hand, If “Maximum memory used” is not set at all, it is equal to not having a memoryLimit at all which can cause OOM’s anyway. The 3GB memory space you have with the 3GB switch is for each application pool, so when you experience an OOM it is for that single application pool not both pools together.

However, “OutOfMemory Exception” can happen for a lot of different reasons (none of them documented very well). As you can see in my other article: “Multiple Application Pools impact on Virtual Memory and Pagefile Size“, you will start to get OOM’s if you run out of physical memory and pagefile. What exacttly caused the OOM’s on your box is hard to say without deeper investigation, it could be lack og either virtual memory (on the app pool) or lack of physical memory (for the OS). You can’t just run an endless number of application pools on a box. And at some point adding an extra application pool will impact performance because memory is swapped to disk. 

Best regards

Jesper Jørgensen, www.sitecore.net

Multiple Application Pools impact on Virtual Memory and Pagefile Size

September 21, 2007

Hi

The last couple of weeks I have been assigned the task of finding out “How many Asp.Net websites can be hosted on a Windows 2003 server?”. The testing was a joint venture between Sitecore and Cohaesio (personified by Rune Mariboe). For reference the sites tested were Sitecore CMS websites.

You can’t really answer this question without making some assumtions about external factors. So we did too. The machine we tested was a 3GHz dual core machine with 4GB RAM (32 bit). We assumed that every site would generate somewhere around 15 pagehits a minute.

Each site used a SQL Express database and as our initial testing showed that each website on average resulted in an increase of about 100MB virtual memory (Public Bytes in perfmon). Knowing that we would risk OutOfMemory (OOM) exceptions if public bytes exceeded 1200MB, we chose to have a maximum of 10 sites per application pool.

Then we started testing, with increments of 10 sites per test. We managed to run 60 sites simultaneously at a total traffik of 14.79 pageviews per second. The size of Public Bytes were around 700 to 1400 MB per application pool. At this point we jumped to test 80 sites, but although response times were fine, we could see that we started to get HTTP 500 errors in the responses to MS Application Center Test. Some of the sites simply failed with OOM’s.

What actually happens when you create e.g.  8 Application pools is that you get 8 times 2 GB = 16GB virtual memory (potentially), because each application pool runs in its own process. However as the machine only has 4GB physical memory, obviously all this virtual memory cannot be held in physical memory all at once. Actually this only becomes a problem if the Asp.Net applications actually USE a lot of the 2GB virtual memory assigned. But in this case each pool did use 700-1400MB virtual memory as described above. If we just say that each pool on average used 1100MB, then 8 pools would use 8.8GB memory.

Now what Windows does when virtual memory exceeds physical memory is to swap some of the memory to the “PageFile”, which is recommended to be somewhere between 1.5 and 3 times the size of Physical memory, with a maximum of 4096 MB. In our case, the pagefile was the maximum 4096MB, so in total the system could handle 8GB virtual memory. Now, as we estimated that the application pools used 8.8GB virtual memory all in all, it is not strange that th ewebsite started to throw OOM exceptions. There has simply not been anywhere in the system that new memory could be allocated. But I guess very few get the chance to test a server to the memory limits as we did here. And quite impressive to see the practical results of such “abuse”.

Now it actually turns out that you can have more than one pagefile in Windows:

http://support.microsoft.com/default.aspx?scid=kb;en-us;237740&Product=win2000

So we tried to have 3 Pagefiles of 4GB each (on the same disk) for a total of 12GB pagefiles + 4GB physical memory. Additional testing showed that we could run 100 sites (10 app pools) without OOM’s at low traffic. However when we simulated traffic around 15 hits/minute/site, we could run 40 sites, but not 50. Page faults simply became too frequent, and the disk too slow to serve the incoming requests in a timely manner. Theory states that we should place each PageFile on its own Raid-0 disk to make sure that reading from the PageFiles was optimal, but this was a bit overkill for our test.

I believe the above tests pictures a hosting environment where a 64 bit Windows would show it’s strengths. That would allow more physical memory to be added as well as larger pagefiles.

I hope that the above has given you a little further insigt to Windows memory handling and what happens when you get too close to the memory borders. It certainly did to me.

Cheers, Jesper

Additional reading:

http://www.petri.co.il/pagefile_optimization.htm

Who’s hitting my website?

June 6, 2007

Yesterday I went to a company suffering from periodic performance on their .Net website. Initially we could see from the logfiles that in one hour during the night 18.000+ hits were received by the webserver. The IT responsible stated that this number was way beyond normal traffic patterns. The question was asked, if these requests could be sent from a single server.

No problem. In a few minutes I created the following script for LogParser:

c:\program files\log parser 2.2\logparser
    “SELECT TOP 10 COUNT(*), c-ip
 INTO TopTenIps.txt
 FROM ex070601.log
 WHERE EXTRACT_EXTENSION(cs-uri-stem)=’%aspx%’
 GROUP BY c-ip
 ORDER BY count(*) DESC “

The output was the following: 

COUNT(ALL *) c-ip
———— —————
100033           82.17.208.4
153            214.150.46.211
47            39.99.13.122
30            196.2.114.1
16            82.52.143.18
11            126.225.159.150
9            66.55.208.94
7            66.55.208.179
7            201.47.30.107
4            66.55.212.236

100K hits from the same IP is of course very extreme. They were able to relate the IP address to an inhouse search engine crawling the website at up to 17.000 hits per hour, or more than 3 hits per second. So be careful with webcrawlers ;-). If foreign IP’s are hammering your website, find a way to reject the requests before they spend to many resources on your server.

Understanding ASP.Net memory

May 23, 2007

Applies to 32 bit Windows

As many a developer has experienced memory is not an unlimited resource when using In-process memory like the ASP.Net cache. Ruuning out of it shows symptoms like the process recycling or OutOfmemory and MemoryLimit Exceded Exceptions here and there. Having a basic understanding of what is going on will help you a lot resolving problems like this.

Fact: In a standard setup your worker process always have 2GB Virtual memory available (no matter if you have 1, 2 or 4GB physical memory in the machine). If you use 1.5GB memory in your application but your machine only has 1GB physical memory, some of it is just swapped to disk. Thats why it is virtual memory. This also means that if you are running out of (virtual) memory it doesn’t help to add more physical memory.

So why am I limited to 2GB virtual memory?: Well, a 32bit operating system (OS) is only able to address 4GB memory with these 32 bits. In a standard setup 2 of these are reserved for the OS, and the remaining 2 reserved for applications like ASP.Net.

A common error in memory heavy applications is the OutOfMemory Exception (OOM). It happens when the framework is unable to allocate enough virtual memory to run a piece of code. This is a bad thing because they can be thrown almost anywhere in your code, making it hard to predict the outcome. What happens is that the framework lost a battle against custom code about access to the limited 2GB memory.

Microsoft solved this in IIS 5.0 by adding the setting memoryLimit to the machine.config file. This setting tells how many of the 2GB the workerprocess (custom code) should be allowed to use. The rest of the 2GB is then free for the Framework to use for executing code. In IIS 6.0 the Maximum used memory (in megabytes) property on the application pool is used for the same purpose. Where the setting is quite straightforward in IIS 6.0, this is not the case in IIS 5.0 where it is a percentage of physical memory. Say you have a machine with 2GB memory and the default setting is 60 (%) equalling 1.2GB. If you now add an extra 2GB for a total of 4GB, the 60% now equals 2.4GB out of 2 possible, which of course does not make sense. So make sure to review this setting in the beginning as well as at every change of physical memory.

So how many megabytes should be allowed for the workerprocess?: Microsoft recommends to “set the memory limit to the smaller of 60% of physical RAM or 800 MB“. Here is the explanation why. On almost anything but a 1GB machine 800MB will always be the smaller of the two numbers. So 800MB is the recommended setting for 2GB+ machines. Not really a lot of available cache memory if you have a machine with 4GB physical memory and 2GB virtual.

Microsoft argues: “There are a couple things to consider: First, the likelihood of experiencing an OutOfMemoryException begins to increase dramatically when “Process\Virtual Bytes” is within 600 MB of the virtual address space limit (generally 2 GB), and secondly, tests have shown that “Process\Virtual Bytes” is often larger than “Process\Private Bytes” by no more than 600 MB

If you search and read long on the net, you will eventually find that when you set the memoryLimit setting, the value it is compared against is the performance counter “Process\Private Bytes“, which you force to stay below 800MB. The logic is: 800MB plus 600 MB (that Virtual Bytes may be larger than Private Bytes) + 600MB (which is needed as buffer to avoid OOM’s) = 2000MB = 2GB which is the available amount of virtual memory. So if you set memoryLimit higher, you start risking OOM’s. Thats the logic behind.

So what will happen if you use more ASP.Net cache than is allowed through the memoryLimit? The system will throw a “memoryLimit Exceeded” exception and recycle the worker process to release memory. If you want to avoid this, you should make sure that Private Bytes never exceed the memoryLimit.

The last topic I will cover here is the 3GB switch which is available on some versions of Windows, but not all (none mentioned by purpose). This setting  makes only 1 GB available for the OS, and therefore leaves 3GB for applications like the worker process. Through the same maths as we used to determine the 800MB limit above, we can calculate that this setting allows a memory limut of 1800MB, as this still leaves 1200 MB for the rest. Below I have inserted a picture visualizing this:

ASP.Net Virtual Memory

Links: http://msdn2.microsoft.com/en-us/library/ms972959.aspx#monitor_perf_topic12

http://msdn2.microsoft.com/en-us/library/ms998583.aspx#scalenetchapt17_topic14