Archive for May, 2007

Advanced stress testing with Application Center Test

May 31, 2007

At work I am facing a challenge of testing how many websites can be hosted on a single server. As you never know the real circumstances that real world websites and users create, you have to make some assumptions for such a test to make sense. In this case we know that all the sites will be variations from the same original site, so we will just use this site for testing. Regarding traffic patterns we are lucky enough to have real world logfiles from similar websites that we can use.

After spending most of my working hours trying to understand what goes on under the hood of websites (mainly ASP.Net, but 99% is HTTP requests anyway), I have learned not to care so much about simultaneous users and just cut to the bone by answering the question “… and how many requests is that per second”. From my experience the webserver do not care much if 10 users clicks once a minute or 5 users click every 30 seconds. To make a long story short, we are going to use LogParser from Microsoft to find the average number of requests per second during peak hours from these logfiles. This will give me a number of requests per second (minute or hour, whatever) PER website.

Say a website receives 2 requests per minute, and I want to stress test 17 identical websites. Overall this will be 1765 milliseconds between each request. But what is the best way to test if this performs fine? Well, I have been looking at Microsofts “Application Center Test” (ACT) made for the purpose.   ACT has a nice feature where I can record clicks in a browser and save it in vnscript files. However, having to click in 17 different (but identical sites) seems just too tiresome. And what do I do when I later on want to test with 34 identical sites, and how do I get it right with the 1765 milliseconds? Not to think of the sizes of these vbscript files.  

Then it hit me: vbscript = programming language. Work smarter, not harder. So I recorded a sample script, and manipulated it into a highly dynamical one. It has two arrays, one with the list of host names and another with a list of the relative urls we are going to test. It also has a delay that i can set to anything I want e.g. 1765 milliseconds (later on I might want to make the number random around an average number to make it more realistic. The script then just picks a random domain and a random url at each request to simulate real life traffic. I think it is kinda neat. The script goes here:

Option Explicit
Dim fEnableDelays
fEnableDelays = true

Sub SendRequest1(p_domain, p_url, p_delay)
    Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode
    If fEnableDelays = True then Test.Sleep (p_delay)
    Set oConnection = Test.CreateConnection(p_domain, 80, false)
    If (oConnection is Nothing) Then
        Test.Trace “Error: Unable to create connection to ” & p_domain
        Set oRequest = Test.CreateRequest
        oRequest.Path = p_url
        oRequest.Verb = “GET”
        oRequest.HTTPVersion = “HTTP/1.1”
        set oHeaders = oRequest.Headers
        oHeaders.Add “Accept”, “image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/, application/, application/msword, application/x-shockwave-flash, */*”
        oHeaders.Add “Accept-Language”, “en-us”
        oHeaders.Add “Accept-Encoding”, “gzip, deflate”
        oHeaders.Add “User-Agent”, “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”
        oHeaders.Add “Host”, “(automatic)”
        oHeaders.Add “Cookie”, “(automatic)”
        Set oResponse = oConnection.Send(oRequest)
        If (oResponse is Nothing) Then
            Test.Trace “Error: Failed to receive response for URL to ” + p_url
            strStatusCode = oResponse.ResultCode
        End If
    End If
End Sub
Sub Main()
 dim arDomain, arUrl
 dim i, delay
 dim domainIndex, urlIndex
 dim ubounddomain, uboundurl

 delay = 500
 arDomain = Array(““)
 arUrl = Array(“/product.aspx”, “/customers.aspx”, “/parners.aspx”)
 ubounddomain = ubound(arDomain)
 uboundurl = ubound(arUrl)
 for i = 0 to 30
  domainIndex=Int((ubounddomain) * rnd())
  urlIndex=Int((uboundurl) * rnd())
  call SendRequest1(arDomain(domainIndex), arUrl(urlIndex), delay)

End Sub

When I set the delay to 500 milliseconds and run it with 2 simultaneous users, I can see the graph flatlining around 4 requests per second, which is exactly as expected. So to add more sites I just need to add the extra domains and adjust the delay to simulate the extra stress.

Happy testing 🙂


Keep your newsletters out of spam filters

May 28, 2007

Spam emails are becoming more and more of a plaque in your and my inboxes. What is not yet common knowledge is that the fight against spam may also result in your legitimate newsletters getting caught up in spam filters. Did you ever get complaints that hotmail users never got yournewsletters?

Well, Microsoft who owns hotmail are playing hardball in the fight against spam, by enforsing senders to use the Sender Policy Framework (SPF) when sending emails to hotmail users. If not used, emails are bounced and never received by the hotmail user. It is not really a bad move because SPF is kinda cool, is easy to implement and it now gets a lot of deserved attention.

So what is it about? : Well, more and more people have tried to receive “underliverable emails” to users they never sent emails to. The reason is that spammers have used their domain to send emails from, and if they are rejected you get the email, not them. SPF is about making life hard for spammers who want to abuse YOUR domain. When implementing SPF, the sender sets up an SPF text record in their DNS, stating which servers are allowed to send emails from this domain. A receiving mailserver screening incoming mails can then check the domain against the IP address of the sender and see if they match. If not the mail is rejected and that is what happens when hotmails are bounced. Of course the receiving mail servers also have to support SPF to avoid spam, but that really isn’t your headache.

What is your headache is that you should make sure that the SPF record is set up for your domain. That way your newsletter will also arrive at mailservers checking the SPF records.

Read more about the Sender Policy Framework and how to set up SPF records here:

Understanding ASP.Net memory

May 23, 2007

Applies to 32 bit Windows

As many a developer has experienced memory is not an unlimited resource when using In-process memory like the ASP.Net cache. Ruuning out of it shows symptoms like the process recycling or OutOfmemory and MemoryLimit Exceded Exceptions here and there. Having a basic understanding of what is going on will help you a lot resolving problems like this.

Fact: In a standard setup your worker process always have 2GB Virtual memory available (no matter if you have 1, 2 or 4GB physical memory in the machine). If you use 1.5GB memory in your application but your machine only has 1GB physical memory, some of it is just swapped to disk. Thats why it is virtual memory. This also means that if you are running out of (virtual) memory it doesn’t help to add more physical memory.

So why am I limited to 2GB virtual memory?: Well, a 32bit operating system (OS) is only able to address 4GB memory with these 32 bits. In a standard setup 2 of these are reserved for the OS, and the remaining 2 reserved for applications like ASP.Net.

A common error in memory heavy applications is the OutOfMemory Exception (OOM). It happens when the framework is unable to allocate enough virtual memory to run a piece of code. This is a bad thing because they can be thrown almost anywhere in your code, making it hard to predict the outcome. What happens is that the framework lost a battle against custom code about access to the limited 2GB memory.

Microsoft solved this in IIS 5.0 by adding the setting memoryLimit to the machine.config file. This setting tells how many of the 2GB the workerprocess (custom code) should be allowed to use. The rest of the 2GB is then free for the Framework to use for executing code. In IIS 6.0 the Maximum used memory (in megabytes) property on the application pool is used for the same purpose. Where the setting is quite straightforward in IIS 6.0, this is not the case in IIS 5.0 where it is a percentage of physical memory. Say you have a machine with 2GB memory and the default setting is 60 (%) equalling 1.2GB. If you now add an extra 2GB for a total of 4GB, the 60% now equals 2.4GB out of 2 possible, which of course does not make sense. So make sure to review this setting in the beginning as well as at every change of physical memory.

So how many megabytes should be allowed for the workerprocess?: Microsoft recommends to “set the memory limit to the smaller of 60% of physical RAM or 800 MB“. Here is the explanation why. On almost anything but a 1GB machine 800MB will always be the smaller of the two numbers. So 800MB is the recommended setting for 2GB+ machines. Not really a lot of available cache memory if you have a machine with 4GB physical memory and 2GB virtual.

Microsoft argues: “There are a couple things to consider: First, the likelihood of experiencing an OutOfMemoryException begins to increase dramatically when “Process\Virtual Bytes” is within 600 MB of the virtual address space limit (generally 2 GB), and secondly, tests have shown that “Process\Virtual Bytes” is often larger than “Process\Private Bytes” by no more than 600 MB

If you search and read long on the net, you will eventually find that when you set the memoryLimit setting, the value it is compared against is the performance counter “Process\Private Bytes“, which you force to stay below 800MB. The logic is: 800MB plus 600 MB (that Virtual Bytes may be larger than Private Bytes) + 600MB (which is needed as buffer to avoid OOM’s) = 2000MB = 2GB which is the available amount of virtual memory. So if you set memoryLimit higher, you start risking OOM’s. Thats the logic behind.

So what will happen if you use more ASP.Net cache than is allowed through the memoryLimit? The system will throw a “memoryLimit Exceeded” exception and recycle the worker process to release memory. If you want to avoid this, you should make sure that Private Bytes never exceed the memoryLimit.

The last topic I will cover here is the 3GB switch which is available on some versions of Windows, but not all (none mentioned by purpose). This setting  makes only 1 GB available for the OS, and therefore leaves 3GB for applications like the worker process. Through the same maths as we used to determine the 800MB limit above, we can calculate that this setting allows a memory limut of 1800MB, as this still leaves 1200 MB for the rest. Below I have inserted a picture visualizing this:

ASP.Net Virtual Memory


Welcome to my blog

May 23, 2007

My name is Jesper Jørgensen, I was born in 1971 in Denmark and as my interests span wide, I will write about “stuff”. For insights to my main interests, please have a look in the About menu.