Understanding ASP.Net memory

Applies to 32 bit Windows

As many a developer has experienced memory is not an unlimited resource when using In-process memory like the ASP.Net cache. Ruuning out of it shows symptoms like the process recycling or OutOfmemory and MemoryLimit Exceded Exceptions here and there. Having a basic understanding of what is going on will help you a lot resolving problems like this.

Fact: In a standard setup your worker process always have 2GB Virtual memory available (no matter if you have 1, 2 or 4GB physical memory in the machine). If you use 1.5GB memory in your application but your machine only has 1GB physical memory, some of it is just swapped to disk. Thats why it is virtual memory. This also means that if you are running out of (virtual) memory it doesn’t help to add more physical memory.

So why am I limited to 2GB virtual memory?: Well, a 32bit operating system (OS) is only able to address 4GB memory with these 32 bits. In a standard setup 2 of these are reserved for the OS, and the remaining 2 reserved for applications like ASP.Net.

A common error in memory heavy applications is the OutOfMemory Exception (OOM). It happens when the framework is unable to allocate enough virtual memory to run a piece of code. This is a bad thing because they can be thrown almost anywhere in your code, making it hard to predict the outcome. What happens is that the framework lost a battle against custom code about access to the limited 2GB memory.

Microsoft solved this in IIS 5.0 by adding the setting memoryLimit to the machine.config file. This setting tells how many of the 2GB the workerprocess (custom code) should be allowed to use. The rest of the 2GB is then free for the Framework to use for executing code. In IIS 6.0 the Maximum used memory (in megabytes) property on the application pool is used for the same purpose. Where the setting is quite straightforward in IIS 6.0, this is not the case in IIS 5.0 where it is a percentage of physical memory. Say you have a machine with 2GB memory and the default setting is 60 (%) equalling 1.2GB. If you now add an extra 2GB for a total of 4GB, the 60% now equals 2.4GB out of 2 possible, which of course does not make sense. So make sure to review this setting in the beginning as well as at every change of physical memory.

So how many megabytes should be allowed for the workerprocess?: Microsoft recommends to “set the memory limit to the smaller of 60% of physical RAM or 800 MB“. Here is the explanation why. On almost anything but a 1GB machine 800MB will always be the smaller of the two numbers. So 800MB is the recommended setting for 2GB+ machines. Not really a lot of available cache memory if you have a machine with 4GB physical memory and 2GB virtual.

Microsoft argues: “There are a couple things to consider: First, the likelihood of experiencing an OutOfMemoryException begins to increase dramatically when “Process\Virtual Bytes” is within 600 MB of the virtual address space limit (generally 2 GB), and secondly, tests have shown that “Process\Virtual Bytes” is often larger than “Process\Private Bytes” by no more than 600 MB

If you search and read long on the net, you will eventually find that when you set the memoryLimit setting, the value it is compared against is the performance counter “Process\Private Bytes“, which you force to stay below 800MB. The logic is: 800MB plus 600 MB (that Virtual Bytes may be larger than Private Bytes) + 600MB (which is needed as buffer to avoid OOM’s) = 2000MB = 2GB which is the available amount of virtual memory. So if you set memoryLimit higher, you start risking OOM’s. Thats the logic behind.

So what will happen if you use more ASP.Net cache than is allowed through the memoryLimit? The system will throw a “memoryLimit Exceeded” exception and recycle the worker process to release memory. If you want to avoid this, you should make sure that Private Bytes never exceed the memoryLimit.

The last topic I will cover here is the 3GB switch which is available on some versions of Windows, but not all (none mentioned by purpose). This setting  makes only 1 GB available for the OS, and therefore leaves 3GB for applications like the worker process. Through the same maths as we used to determine the 800MB limit above, we can calculate that this setting allows a memory limut of 1800MB, as this still leaves 1200 MB for the rest. Below I have inserted a picture visualizing this:

ASP.Net Virtual Memory

Links: http://msdn2.microsoft.com/en-us/library/ms972959.aspx#monitor_perf_topic12

http://msdn2.microsoft.com/en-us/library/ms998583.aspx#scalenetchapt17_topic14

About these ads

9 Responses to “Understanding ASP.Net memory”

  1. Lars Nielsen Says:

    Wow. Nice one. More of this stuff, Jesper.

  2. Allan Thræn Says:

    Great article with in-depth research. I have run into asp.net memory issues in the past, and I’ll definetly remember this background article when I undoubtedly run into them again :-)

  3. Adam Says:

    Im running iis6 with win server 2003 with /3gb switch in boot.ini and 4gb ram.
    I have set the Max Used Memory to 800mb
    and im getting OOM exceptions fairly frequently.

    Ive tried changing it to 400mb, but now i get the process recycling every couple of minutes..

    The above looks and sounds great.. but for the life of me i cant get the process to recycle at a resonable frequency per day.

    any tips?

  4. jesperfj Says:

    Hi Adam
    If you start up perfmon to show “Process\Private Bytes” as well as Virtual Bytes you should see that private bytes stay below the 800Mb you stated in Max Used Memory. What are these values when you experience OOM’s? First of all you should double check that it really is OOM exceptions you experience.
    The “memoryLimit exceeded” (MLE) exception could also cause problems, and it is more or less so that whenever you decrease the MaxUsedMemory, you also decrease the risk of OOM’s but INCREASE the risk of MLE’s and vice versa. Therefore it is important to be certain what exception you actually experience.
    With your setup (MaxUsedMemory=800 and 3GB switch) you have 2.2 GB memory free for the tasks that usually cause OOM’s. So if your code uses up this 2.2GB it is a bit extreme. You can download a free trial of a memory profiler at memprofiler.com that can tell you about anything about the current memory status of your process. It should enable you to find the cause of the OOM’s.
    When you set Max Used Memory to 400Mb, I am almost certain that process recycles do to memoryLimit Exceeded. If you observer private bytes in perfmon you should be able to see it increase up to the 400MB limit, and then suddenly drop as the process is recycled. What is the highest “Private Bytes” you observe when you set Max Used Memory to 800MB?
    It sounds like you may have a “memory leak” somewhere in the code, that needs to be fixed in order to get rid of the recycles. If you have such a leak, it is only a question about time before you run into trouble.
    Feel free to write again.

  5. Raj Says:

    Hi jasperfj,
    Thank you very much for the information. Its been a very useful one. The myths about /3GB are now unravelled.
    I have a typical problem facing. I have a web server farm of about 36 servers (32 bit) and in each server the RAM size is 4GB. As you said earlier, since the 3GB switch is not enabled the kernel uses 2GB and other applications can use another 2GB.
    The problem is that my ASP application running on the IIS 6.0 is reaching its peak of 1.8 GB memory utilisation very frequently. i thought if i could change the boot.ini file by adding 3GB switch, the application will be able to use more virtual memory. Hoping that this may increase maximum limit for memory utilisation. But there wasn’t any success. I just rebooted the machine after adding 3GB switch. Should i be following any other steps to achieve what i am looking for ??
    Can you please suggest me how do i increase the virtual memory limit from 1.8GB to more. (P.S The application is an ASP application)

  6. jesperfj Says:

    Hi Raj
    You should check if it is “OutOfMemory” (OOM ) OR “memorylimit Exceeded” exceptions that is thrown when you get to 1.8GB. Assuming it is “memorylimit Exceeded”, you could increase the memoryLimit. That would be in IIS manager on IIS 6.0, where you should find the “Maximum Memory…” and adjust it to whatever you find right.

    Notice however that 1.8GB is the recommended memoryLimit when using the 3GB switch. Without the switch the recommendation is only 800MB. But you can of course increase the limit above 1,8GB and see what happens. But the risk of receiving OOM exceptions will increase.

    You could also see if you can decrease the memory usage by storing it outside the ASP.Net process. If you start new IIS website with its own application pool on the same server, this application pool will get its own 3GB virtual memory – giving you 3+3 = 6GB virtual memeory on one machine. However there is no such thing as a free meal, and boosting virtual memory like this will increase disk swapping by Windows memory management.

    What kind of exception is it you receive, and what is the memoryLimit set to?

  7. James Says:

    WE have 2 servers w/2GB physical RAM both are idetical.
    We were having OutOfMEM exceptions.

    We added 2GB to one to make 4GB and added the 3/GB switch.

    The servers host ASP.NET xml web services that use IIS / http for transport.

    The server with 2GB is slightly outperforming the server with 4GB in terms of response times.

    Also, the 4GB server is using much more CPU than the 2GB Server to do the same job.

    With using the 3GB switch, the kernel is now limited to 1GB.
    Could there be an issue with http.sys queuing up since it runs in kernel mode?

    We recycle the App pool on a schedule once a day. All other settings on the App pool are
    default.

    Should we try tuning with recycling maxed used mem: 800?

    Is it true that recyling using this method is standard practice?

    Recycling a system service to manage the Application doesnt sound as good as the Application
    performing Garbage collection on its own.

    Can you give some advice?

    Thanks,
    James

  8. jesperfj Says:

    Hi James
    Interesting observation. Do you run anything else than IIS on these servers? I am curious about what proces is actually using the CPU. I would set up three counters in perfmon on both servers:
    1) Total CPU usage
    2) CPU usage of w3wp.exe process
    3) .Net CLR memory > % time in GC

    By comparing 1) and 2) you should get an idea if is Asp.Net using the memory or “other stuff”. As the kernel got its memory reduced to 1 GB, it may be more CPU heavy.

    3) Shows how much CPU the .Net garbage collector uses. Now that you have increased the memory for .Net, the garbage collector should be more busy collecting garbage from this memory. You can also set up perfmon counters to see how many GC collects are performed.

    If you got rid of the OOM exceptions on the 4GB server, you may find it acceptable that it performs slightly worse than the other server because you never know when OOM will show its ugly face to the users and the result is usually a broken website. If recycling the App pool once a day avoids OOM exceptions, i very much prefer this to recycling at random times of day when memory is full. But I recommend a balanced strategy where you recycle once a day, but also have a memory limit where to recycle just in case you are getting close to a OOM. But it should be the exception that this occurs. I assume you add data to the Asp.Net cachesince you run out of memory. If you have set up any limits in your code where you stop to add more to the cache, you actually have a so called “self inflicted” memory leak. You can therefore avoid the OOM’s if your code stops adding to the cache at some limit.

    The Sitecore CMS is heavily relying on caching, but still in general our customer websites run for days, weeks and months without hitting memory limits or experiencing OOM’s. This has been obtained by monitoring the processes “Private bytes” in our code, and stop adding to the cache at configured thresholds.

    Notice: I agree that recycling the system service should not be the norm. But your code may keep pointers to the memory, which is e.g. the case if you store something in a (in process) session variable. The garbagfe collector will NEVER be able to collect this memory because it is in use. Therefore, at some point you have to prioritize what data to cache in memory.

    Brgds. Jesper

  9. Maximus Says:

    I would like to see a continuation of the topic

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: