How To Make Your AJAX Site Crawlable: 3 Simple Steps

One of the most important aspects of any search engine optimized website is that its content is accessible to search engine bots.  If your site is using javascript there is a good chance that some of your content is not being indexed.  This will result in lower rankings on google and other search engines.

A Google Solution
Luckily Google has a specification for Making AJAX Applications Crawlable.  My implementation of the specification is outlined in three steps below:

1) Hash Fragments That Begin With !

The first requirement is that all javascript-based links (i.e., links that perform some ajax call and return content as a result) should contain '!' as the first character of the hash fragment:

//example Link from my site
<a href="#!about">about</a>

For history management I'm using jquery address with crawlable set to true.  You will need to make sure that your javascript code is able to appropriately handle links of this nature.

2) Create Static HTML Pages

In order for this solution to work you need some html to map to when  requests are made by Google-bot.  For each AJAX link from step 1, generate a separate html page.  For simplicity, name the html page whatever name you have given the hash fragment (i.e., #!about => about.html).  The content of each .html page should contain whatever results from clicking the AJAX link.  NOTE: These static pages will be used only for bots.

3) Handle "_escaped_fragment_" Server Side

There is a slight bit of ambiguity in Google's specification about mapping between #! and _escaped_fragment_.  All this really boils down to is that when Google-bot is scanning your site, urls with #! will get replaced by ?_escaped_fragment_=.  In other words, if you have a url containing #!<some_value>, you can be assured that when Google-bot attempts to index this url, a request containing parameter '_escaped_fragment_' with value '<some_value>' will be made to your server.  My primary jsp that handles all requests before doing anything else will look for this parameter and redirect if necessary as follows:


<%
if(request.getParameter("_escaped_fragment_")!=null) {
String escapedFragment = request.getParameter("_escaped_fragment_");
String decodedEscapedFrag = URLDecoder.decode(escapedFragment,"UTF-8");
response.sendRedirect(App.context()+decodedEscapedFrag+".html");
return;
}
%>


In the above code snippet notice that I am appending '.html' to the end of the decodedEscapedFragment.  This code takes the hash fragment and redirects to the appropriate static html page.  For instance, if the link clicked has href="#!about", then request.getParameter("_escaped_fragment_") will be equal to "about".  The resulting redirect will have a value of "/about.html"

Test It Out!
Now that you are finished, there are a couple ways to see if you're content is now accessible.  For each URL containing your new hash fragment, replace "#!" with "?_escaped_fragment_=", and make sure your static html page is returned by your server.  You can try this out at my website.  You'll notice that http://dsswebdesign.com/#!/about can be changed to http://dsswebdesign.com/?_escaped_fragment_=about, and the page looks the same.

A second way to test this is to add your site to Webmaster Tools.  Under diagnostics there is an option to 'Fetch as Googlebot'.  From there you can make sure that each of your #! links are reached successfully by Google-bot.

Making Sure Your Site Works In Internet Explorer

With Chrome usage on the rise and Firefox holding steady, more and more sites are utilizing CSS3 while neglecting to consider behavior in Internet Explorer. IE usage has been in decline now for some time, however, at almost 1/4 market share according to W3, this ugly monster cannot be ignored. When coding your site to here are 3 tips to consider:

1) :hover,:focus,:active pseudo-classes lack support for most elements in IE

Thanks to poor browser standards support in IE, many pseudo-classes like hover and focus work for anchor tags only. To address this IE pseudo-class flaw an easy all-in-one solution is to use jquery to add/remove hover/focus/active css classes for elements that should change on these events. For example, if you want the background color of a div to change on hover, your non-IE solution may look like the following:

//Warning: This won't work for Internet Explorer 8 and earlier

div{background:red;}
div:hover{background:blue;}

Instead use jquery to update your css class as follows:

$('div').hover(function(){$(this).addClass('hover');},function(){$(this).removeClass('hover');});

Because we are using a css class rather than the pseudo-class, your css will change only slightly:

div{background:red;}
div.hover{background:blue;}

What I find convenient about this solution is you can handle additional elements hover effects by simply updating your jquery selector. Say in addition to the div background change we want to add border to all our 'li' elements on hover. The selector from above becomes:

$('div,li')

And you can add your border in css:

li.hover{border:1px solid green;}

2) Graceful Degradation for IE

In some cases you many find that a CSS3 feature does not degrade in an acceptable way for IE. If you decide to use text-shadows, for example, it's important to consider what the text will look like without them since they are not going to show up in IE. But maybe you just really love the white text on light background with a dark text-shadow. In this case you will have to come up with a workaround for IE so that the text is visible. If you are not using strict doctype then an easy solution is to define in CSS the rule you would like to use for IE, then below define your rule for 'all other browsers' using a child selector:

//Rule used by all browsers

div .innerdiv{color:black;}

//Override color for all browsers except IE

div>.innerdiv{color:white;text-shadow:1px 1px 5px black;}

In the above example the resulting text of an element with class name 'innerdiv' will be white with black text-shadow in non-IE browsers. Internet Explorer will render the text black without text-shadow, but keep in mind this trick only works when using a non-strict doctype. If, on the other hand, you are using strict doctype, then conditional statements can be used as follows:

<!--[if IE]>
<link href="/ie.css" rel="stylesheet" type="text/css">
<![endif]-->

Inside of the ie.css file you can include all your ie specific css

3) Using Borders To Address Lack Of Box-Shadow

Box-shadow is a great feature of CSS3 that can really help to achieve an updated look and feel you may be hoping for, however, for versions of IE prior to 9 and Firefox prior to 3.5 using box-shadow or -moz-box-shadow in your CSS file would do nothing. As a result, it is necessary to consider what your site looks like without the shadows. If you need an elements boundaries to be identifiable it's best to use a border (in most cases the same color as the box-shadow color you have chosen).