You are here

DNN Module Caching and Crawlers

DNN LogoJust researched a curious issue with DotNetNuke. It all started with a public page rarely visited by users but quite often visited by crawlers - Pingdom bot and Googlebot.

The page is pretty DB intensive and with no caching it takes around 3 seconds to load. With all modules cached to disk for an hour, page load time goes down to 0.3s. And it all worked well during my test from the browser - I noticed that every second and subsequent visits to the page within one hour were blazingly fast, so I thought I am done and can move on.

However I was pretty surprised later when I enabled page monitoring with pingdom and it showed that an average page load time is 3 seconds! Even when pingdom is configured to retrieve page every 1 minute! Why? Google webmaster tools also showed pretty nasty numbers. What was wrong? Why does module cache work when I use browser and does not work when page is retrieved by crawler?

Later I found an explanation. In 2010 DNN team added a piece of code that would disable saving of module rendering results (writing to the cache) when the page is visited by any robot. However it would still use the cache (read from the cache) if it was generated previously by browser request.

Honestly, I did not understand the explanation given by the DNN team here. It did not make sense to me. But here we go, current version of DNN has the following line:

 

if (!string.IsNullOrEmpty(cachedOutput) && (!HttpContext.Current.Request.Browser.Crawler))
{
    //Save content to cache
    ...
 
Well, at least it is easy to adjust in my case - the only way DNN would know it is the crawler is by the User Agent string, which I can change on Pingdom. I set it to IE7, and now it all works fine.
Social Buttons: 

Add new comment

Get Updates

RSS Twitter

Subscribe by Email:

Recent Comments

Who's online

There are currently 0 users online.