What is a #1 Google Ranking Worth

Being a scientist at heart I simply love when somebody provides information that makes be better at quantifying and predicting th evalue of my efforts. Thats why I am a big fan of:

Gilb’s Law: Anything you need to quantify can be measured in some way that is superior to not measuring it at all

Therefore I will use todays post to credit Aaron Wall from Seobook for his great article:

How Much Money is a Top Google Ranking Worth to Your Business?

As a short apetizer, here is an extract on how you could estimate the value of going from #5 to #1:

Quote:Patrick, a moderator on our community forums, runs a Bingo Card Creator website which recently ranked #5 in Google for bingo cards.

• A #5 ranking sent him 6,000 unique visitors per month.
• A #1 spot, using the leaked AOL search data (referenced later in this document), is worth 8.5 times what #5 is. 6,000 * 8.5 = ~50,000 uniques per month
• His site currently makes $40 for every 1,000 pageviews.
• His estimated income from ranking #1 for [bingo cards]: $2,000 a month.

Exit Jesper

A Clever Internet Marketer Is Trying to Trademark “SEO”

As you might know SEO is an acronym for Search Engine Optimization, and had been used for ages to describe whatever it is SEO experts do. The acronym is comparable to the acronym CMS for the content management industry.

No, figure: Some guy has applied to get SEO registered as a trademark, and has gotten so close that it is scary. You can read fthe full story by Sarah Bird here: http://www.seomoz.org/blog/pulling-a-fast-one-a-clever-internet-marketer-is-trying-to-trademark-seo

It is scary but intereseting reading on some of the absuridities in our high tech world…

/Jesper

Google App Engine

Having spent time with extremely high trafficed sites on the Sitecore CMS platform, I have always been intriqued by the challenges of sites that experience high traffic during short periods of the yeat. Think: Official World Cup homepage etc.

How many servers do you need running idle 51 weeks a year in order to handle the load 1 week a year? Woudl it somehow be possible to rent server resources when you expect traffic peaks? This morning came a post from Matt Cutts blog about a new Google initiative called Google App Engine.

Quote: At tonight’s Campfire One we launched a preview release of Google App Engine — a developer tool that enables you to run your web applications on Google’s infrastructure. The goal is to make it easy to get started with a new web app, and then make it easy to scale when that app reaches the point where it’s receiving significant traffic and has millions of users.

Google App Engine gives you access to the same building blocks that Google uses for its own applications, making it easier to build an application that runs reliably, even under heavy load and with large amounts of data. The development environment includes the following features:

And continuing:

During this preview period, applications are limited to 500MB of storage, 200M megacycles of CPU per day, and 10GB bandwidth per day. We expect most applications will be able to serve around 5 million pageviews per month. In the future, these limited quotas will remain free, and developers will be able to purchase additional resources as needed.

So you get something like 5 million pageviews per month for free. That is a whole lot. On average close to 2 pageviews per second. Imagine the possibilities if your CMS can outsource html reendering to a service like this during peak hours. It is of course not limited to CMS related tasks, but can be used for anything you can make fit into the framework.

Exciting initiative.

/Jesper

Asp.Net Memory Limit – part 2

Interesting article on the history of the memory limit. As can be seen, the memory limt for 64 bit Windows is recommended to be MIN(60%, 1TB).

http://blogs.msdn.com/tmarq/archive/2007/06/25/some-history-on-the-asp-net-cache-memory-limits.aspx 

Jesper Jørgensen, www.sitecore.net

Asp.Net Memory Limit

I saw this question today from Nick Simpson at osdir.com. I couldn’t see where to reply so I hope it is okay that I take it here:

Subject:	ASP .NET Memory Limit.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

I have a couple of web sites on a single Windows 2003 server running ASP.NET 1.1.
The /3GB switch is set in boot.ini and the ASPNETENABLE3GB environment variable is also set. The machine.config has the default memoryLimit on the processModel node of 60.  So does that mean that my web apps have up to a maximum of 3GB RAM to play with each – 1.8GB max if i leave the machine.config setting as it is – expandable to a full 3GB per web app (assuming separate app pools) without worry of adversely affecting performance on that box? 

Is this limit per instance of the w3wp process, or for ASP .NET overall, or indeed for applications on that box overall?

I thought this was per process until i observed our box (with two sites – at 400MB and 800MB – separate app pools) throw loads of OutOfMemory exceptions. This was prior to the /3GB switch being in place – the limit would have been 1.2GB then – 60% of 2GB – but OutOfMemory exceptions were thrown when the combined amount of memory usage for all w3wp process instances were getting near to 1.2GB, and not when either of the two web sites individually were reaching 1.2GB. Am i misunderstanding something or is the memory limit for ASP .NET overall? 

I have read the following article but still find my questions unanswered: http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx  Our server currently has 4GB RAM. We are getting a new one. Would it just be a waste of money getting a new server with 8GB RAM?

Will ASP .NET 2.0 and 64 bit Windows add anything new in this respect?  Thanks for your help,  Nick 

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Hi Nick

In man article called “Understanding Asp.Net memory” here on my blog, I described how memoryLimit was changed from machine.config to a setting called “Maximum memory used” in the IIS manager of IIS 6.0 which is default in Windows 2003. So instead of setting a percentage you just set this setting to an amount of megabytes specific to each application pool. When using memoryLimit in IIS 5.0 you should be aware that Microsoft recommends that it is set to “the smaller of 60% and 800 MB” which is 800 MB on a machine with 2GB memory. Leaving it at 60% equal to 1200 MB increase the risk of OOM errors. Not that I believe this is the cause to your problems as I guess you are running IIS 6.0.

On the other hand, If “Maximum memory used” is not set at all, it is equal to not having a memoryLimit at all which can cause OOM’s anyway. The 3GB memory space you have with the 3GB switch is for each application pool, so when you experience an OOM it is for that single application pool not both pools together.

However, “OutOfMemory Exception” can happen for a lot of different reasons (none of them documented very well). As you can see in my other article: “Multiple Application Pools impact on Virtual Memory and Pagefile Size“, you will start to get OOM’s if you run out of physical memory and pagefile. What exacttly caused the OOM’s on your box is hard to say without deeper investigation, it could be lack og either virtual memory (on the app pool) or lack of physical memory (for the OS). You can’t just run an endless number of application pools on a box. And at some point adding an extra application pool will impact performance because memory is swapped to disk. 

Best regards

Jesper Jørgensen, www.sitecore.net

Multiple Application Pools impact on Virtual Memory and Pagefile Size

Hi

The last couple of weeks I have been assigned the task of finding out “How many Asp.Net websites can be hosted on a Windows 2003 server?”. The testing was a joint venture between Sitecore and Cohaesio (personified by Rune Mariboe). For reference the sites tested were Sitecore CMS websites.

You can’t really answer this question without making some assumtions about external factors. So we did too. The machine we tested was a 3GHz dual core machine with 4GB RAM (32 bit). We assumed that every site would generate somewhere around 15 pagehits a minute.

Each site used a SQL Express database and as our initial testing showed that each website on average resulted in an increase of about 100MB virtual memory (Public Bytes in perfmon). Knowing that we would risk OutOfMemory (OOM) exceptions if public bytes exceeded 1200MB, we chose to have a maximum of 10 sites per application pool.

Then we started testing, with increments of 10 sites per test. We managed to run 60 sites simultaneously at a total traffik of 14.79 pageviews per second. The size of Public Bytes were around 700 to 1400 MB per application pool. At this point we jumped to test 80 sites, but although response times were fine, we could see that we started to get HTTP 500 errors in the responses to MS Application Center Test. Some of the sites simply failed with OOM’s.

What actually happens when you create e.g.  8 Application pools is that you get 8 times 2 GB = 16GB virtual memory (potentially), because each application pool runs in its own process. However as the machine only has 4GB physical memory, obviously all this virtual memory cannot be held in physical memory all at once. Actually this only becomes a problem if the Asp.Net applications actually USE a lot of the 2GB virtual memory assigned. But in this case each pool did use 700-1400MB virtual memory as described above. If we just say that each pool on average used 1100MB, then 8 pools would use 8.8GB memory.

Now what Windows does when virtual memory exceeds physical memory is to swap some of the memory to the “PageFile”, which is recommended to be somewhere between 1.5 and 3 times the size of Physical memory, with a maximum of 4096 MB. In our case, the pagefile was the maximum 4096MB, so in total the system could handle 8GB virtual memory. Now, as we estimated that the application pools used 8.8GB virtual memory all in all, it is not strange that th ewebsite started to throw OOM exceptions. There has simply not been anywhere in the system that new memory could be allocated. But I guess very few get the chance to test a server to the memory limits as we did here. And quite impressive to see the practical results of such “abuse”.

Now it actually turns out that you can have more than one pagefile in Windows:

http://support.microsoft.com/default.aspx?scid=kb;en-us;237740&Product=win2000

So we tried to have 3 Pagefiles of 4GB each (on the same disk) for a total of 12GB pagefiles + 4GB physical memory. Additional testing showed that we could run 100 sites (10 app pools) without OOM’s at low traffic. However when we simulated traffic around 15 hits/minute/site, we could run 40 sites, but not 50. Page faults simply became too frequent, and the disk too slow to serve the incoming requests in a timely manner. Theory states that we should place each PageFile on its own Raid-0 disk to make sure that reading from the PageFiles was optimal, but this was a bit overkill for our test.

I believe the above tests pictures a hosting environment where a 64 bit Windows would show it’s strengths. That would allow more physical memory to be added as well as larger pagefiles.

I hope that the above has given you a little further insigt to Windows memory handling and what happens when you get too close to the memory borders. It certainly did to me.

Cheers, Jesper

Additional reading:

http://www.petri.co.il/pagefile_optimization.htm

Who’s hitting my website?

Yesterday I went to a company suffering from periodic performance on their .Net website. Initially we could see from the logfiles that in one hour during the night 18.000+ hits were received by the webserver. The IT responsible stated that this number was way beyond normal traffic patterns. The question was asked, if these requests could be sent from a single server.

No problem. In a few minutes I created the following script for LogParser:

c:\program files\log parser 2.2\logparser
    ”SELECT TOP 10 COUNT(*), c-ip
 INTO TopTenIps.txt
 FROM ex070601.log
 WHERE EXTRACT_EXTENSION(cs-uri-stem)=’%aspx%’
 GROUP BY c-ip
 ORDER BY count(*) DESC “

The output was the following: 

COUNT(ALL *) c-ip
———— —————
100033           82.17.208.4
153            214.150.46.211
47            39.99.13.122
30            196.2.114.1
16            82.52.143.18
11            126.225.159.150
9            66.55.208.94
7            66.55.208.179
7            201.47.30.107
4            66.55.212.236

100K hits from the same IP is of course very extreme. They were able to relate the IP address to an inhouse search engine crawling the website at up to 17.000 hits per hour, or more than 3 hits per second. So be careful with webcrawlers . If foreign IP’s are hammering your website, find a way to reject the requests before they spend to many resources on your server.

Advanced stress testing with Application Center Test

At work I am facing a challenge of testing how many websites can be hosted on a single server. As you never know the real circumstances that real world websites and users create, you have to make some assumptions for such a test to make sense. In this case we know that all the sites will be variations from the same original site, so we will just use this site for testing. Regarding traffic patterns we are lucky enough to have real world logfiles from similar websites that we can use.

After spending most of my working hours trying to understand what goes on under the hood of websites (mainly ASP.Net, but 99% is HTTP requests anyway), I have learned not to care so much about simultaneous users and just cut to the bone by answering the question “… and how many requests is that per second”. From my experience the webserver do not care much if 10 users clicks once a minute or 5 users click every 30 seconds. To make a long story short, we are going to use LogParser from Microsoft to find the average number of requests per second during peak hours from these logfiles. This will give me a number of requests per second (minute or hour, whatever) PER website.

Say a website receives 2 requests per minute, and I want to stress test 17 identical websites. Overall this will be 1765 milliseconds between each request. But what is the best way to test if this performs fine? Well, I have been looking at Microsofts “Application Center Test” (ACT) made for the purpose.   ACT has a nice feature where I can record clicks in a browser and save it in vnscript files. However, having to click in 17 different (but identical sites) seems just too tiresome. And what do I do when I later on want to test with 34 identical sites, and how do I get it right with the 1765 milliseconds? Not to think of the sizes of these vbscript files.  

Then it hit me: vbscript = programming language. Work smarter, not harder. So I recorded a sample script, and manipulated it into a highly dynamical one. It has two arrays, one with the list of host names and another with a list of the relative urls we are going to test. It also has a delay that i can set to anything I want e.g. 1765 milliseconds (later on I might want to make the number random around an average number to make it more realistic. The script then just picks a random domain and a random url at each request to simulate real life traffic. I think it is kinda neat. The script goes here:

Option Explicit
Dim fEnableDelays
fEnableDelays = true

Sub SendRequest1(p_domain, p_url, p_delay)
    Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode
    If fEnableDelays = True then Test.Sleep (p_delay)
    Set oConnection = Test.CreateConnection(p_domain, 80, false)
    If (oConnection is Nothing) Then
        Test.Trace “Error: Unable to create connection to ” & p_domain
    Else
        Set oRequest = Test.CreateRequest
        oRequest.Path = p_url
        oRequest.Verb = “GET”
        oRequest.HTTPVersion = “HTTP/1.1″
        set oHeaders = oRequest.Headers
        oHeaders.RemoveAll
        oHeaders.Add “Accept”, “image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*”
        oHeaders.Add “Accept-Language”, “en-us”
        oHeaders.Add “Accept-Encoding”, “gzip, deflate”
        oHeaders.Add “User-Agent”, “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”
        oHeaders.Add “Host”, “(automatic)”
        oHeaders.Add “Cookie”, “(automatic)”
        Set oResponse = oConnection.Send(oRequest)
        If (oResponse is Nothing) Then
            Test.Trace “Error: Failed to receive response for URL to ” + p_url
        Else
            strStatusCode = oResponse.ResultCode
        End If
        oConnection.Close
    End If
End Sub
Sub Main()
 dim arDomain, arUrl
 dim i, delay
 dim domainIndex, urlIndex
 dim ubounddomain, uboundurl

 delay = 500
 arDomain = Array(“www.mysite.net“)
 arUrl = Array(“/product.aspx”, “/customers.aspx”, “/parners.aspx”)
 ubounddomain = ubound(arDomain)
 uboundurl = ubound(arUrl)
 
 randomize()
 for i = 0 to 30
  domainIndex=Int((ubounddomain) * rnd())
  urlIndex=Int((uboundurl) * rnd())
  call SendRequest1(arDomain(domainIndex), arUrl(urlIndex), delay)
 next
   

End Sub
Main

When I set the delay to 500 milliseconds and run it with 2 simultaneous users, I can see the graph flatlining around 4 requests per second, which is exactly as expected. So to add more sites I just need to add the extra domains and adjust the delay to simulate the extra stress.

Happy testing

Keep your newsletters out of spam filters

Spam emails are becoming more and more of a plaque in your and my inboxes. What is not yet common knowledge is that the fight against spam may also result in your legitimate newsletters getting caught up in spam filters. Did you ever get complaints that hotmail users never got yournewsletters?

Well, Microsoft who owns hotmail are playing hardball in the fight against spam, by enforsing senders to use the Sender Policy Framework (SPF) when sending emails to hotmail users. If not used, emails are bounced and never received by the hotmail user. It is not really a bad move because SPF is kinda cool, is easy to implement and it now gets a lot of deserved attention.

So what is it about? : Well, more and more people have tried to receive “underliverable emails” to users they never sent emails to. The reason is that spammers have used their domain to send emails from, and if they are rejected you get the email, not them. SPF is about making life hard for spammers who want to abuse YOUR domain. When implementing SPF, the sender sets up an SPF text record in their DNS, stating which servers are allowed to send emails from this domain. A receiving mailserver screening incoming mails can then check the domain against the IP address of the sender and see if they match. If not the mail is rejected and that is what happens when hotmails are bounced. Of course the receiving mail servers also have to support SPF to avoid spam, but that really isn’t your headache.

What is your headache is that you should make sure that the SPF record is set up for your domain. That way your newsletter will also arrive at mailservers checking the SPF records.

Read more about the Sender Policy Framework and how to set up SPF records here:

http://www.openspf.org/Introduction

Understanding ASP.Net memory

Applies to 32 bit Windows

As many a developer has experienced memory is not an unlimited resource when using In-process memory like the ASP.Net cache. Ruuning out of it shows symptoms like the process recycling or OutOfmemory and MemoryLimit Exceded Exceptions here and there. Having a basic understanding of what is going on will help you a lot resolving problems like this.

Fact: In a standard setup your worker process always have 2GB Virtual memory available (no matter if you have 1, 2 or 4GB physical memory in the machine). If you use 1.5GB memory in your application but your machine only has 1GB physical memory, some of it is just swapped to disk. Thats why it is virtual memory. This also means that if you are running out of (virtual) memory it doesn’t help to add more physical memory.

So why am I limited to 2GB virtual memory?: Well, a 32bit operating system (OS) is only able to address 4GB memory with these 32 bits. In a standard setup 2 of these are reserved for the OS, and the remaining 2 reserved for applications like ASP.Net.

A common error in memory heavy applications is the OutOfMemory Exception (OOM). It happens when the framework is unable to allocate enough virtual memory to run a piece of code. This is a bad thing because they can be thrown almost anywhere in your code, making it hard to predict the outcome. What happens is that the framework lost a battle against custom code about access to the limited 2GB memory.

Microsoft solved this in IIS 5.0 by adding the setting memoryLimit to the machine.config file. This setting tells how many of the 2GB the workerprocess (custom code) should be allowed to use. The rest of the 2GB is then free for the Framework to use for executing code. In IIS 6.0 the Maximum used memory (in megabytes) property on the application pool is used for the same purpose. Where the setting is quite straightforward in IIS 6.0, this is not the case in IIS 5.0 where it is a percentage of physical memory. Say you have a machine with 2GB memory and the default setting is 60 (%) equalling 1.2GB. If you now add an extra 2GB for a total of 4GB, the 60% now equals 2.4GB out of 2 possible, which of course does not make sense. So make sure to review this setting in the beginning as well as at every change of physical memory.

So how many megabytes should be allowed for the workerprocess?: Microsoft recommends to “set the memory limit to the smaller of 60% of physical RAM or 800 MB“. Here is the explanation why. On almost anything but a 1GB machine 800MB will always be the smaller of the two numbers. So 800MB is the recommended setting for 2GB+ machines. Not really a lot of available cache memory if you have a machine with 4GB physical memory and 2GB virtual.

Microsoft argues: “There are a couple things to consider: First, the likelihood of experiencing an OutOfMemoryException begins to increase dramatically when “Process\Virtual Bytes” is within 600 MB of the virtual address space limit (generally 2 GB), and secondly, tests have shown that “Process\Virtual Bytes” is often larger than “Process\Private Bytes” by no more than 600 MB“

If you search and read long on the net, you will eventually find that when you set the memoryLimit setting, the value it is compared against is the performance counter “Process\Private Bytes“, which you force to stay below 800MB. The logic is: 800MB plus 600 MB (that Virtual Bytes may be larger than Private Bytes) + 600MB (which is needed as buffer to avoid OOM’s) = 2000MB = 2GB which is the available amount of virtual memory. So if you set memoryLimit higher, you start risking OOM’s. Thats the logic behind.

So what will happen if you use more ASP.Net cache than is allowed through the memoryLimit? The system will throw a “memoryLimit Exceeded” exception and recycle the worker process to release memory. If you want to avoid this, you should make sure that Private Bytes never exceed the memoryLimit.

The last topic I will cover here is the 3GB switch which is available on some versions of Windows, but not all (none mentioned by purpose). This setting  makes only 1 GB available for the OS, and therefore leaves 3GB for applications like the worker process. Through the same maths as we used to determine the 800MB limit above, we can calculate that this setting allows a memory limut of 1800MB, as this still leaves 1200 MB for the rest. Below I have inserted a picture visualizing this:

Links: http://msdn2.microsoft.com/en-us/library/ms972959.aspx#monitor_perf_topic12

http://msdn2.microsoft.com/en-us/library/ms998583.aspx#scalenetchapt17_topic14