How To Make Your AJAX Site Crawlable: 3 Simple Steps

One of the most important aspects of any search engine optimized website is that its content is accessible to search engine bots.  If your site is using javascript there is a good chance that some of your content is not being indexed.  This will result in lower rankings on google and other search engines.

A Google Solution
Luckily Google has a specification for Making AJAX Applications Crawlable.  My implementation of the specification is outlined in three steps below:

1) Hash Fragments That Begin With !

The first requirement is that all javascript-based links (i.e., links that perform some ajax call and return content as a result) should contain '!' as the first character of the hash fragment:

//example Link from my site
<a href="#!about">about</a>

For history management I'm using jquery address with crawlable set to true.  You will need to make sure that your javascript code is able to appropriately handle links of this nature.

2) Create Static HTML Pages

In order for this solution to work you need some html to map to when  requests are made by Google-bot.  For each AJAX link from step 1, generate a separate html page.  For simplicity, name the html page whatever name you have given the hash fragment (i.e., #!about => about.html).  The content of each .html page should contain whatever results from clicking the AJAX link.  NOTE: These static pages will be used only for bots.

3) Handle "_escaped_fragment_" Server Side

There is a slight bit of ambiguity in Google's specification about mapping between #! and _escaped_fragment_.  All this really boils down to is that when Google-bot is scanning your site, urls with #! will get replaced by ?_escaped_fragment_=.  In other words, if you have a url containing #!<some_value>, you can be assured that when Google-bot attempts to index this url, a request containing parameter '_escaped_fragment_' with value '<some_value>' will be made to your server.  My primary jsp that handles all requests before doing anything else will look for this parameter and redirect if necessary as follows:


<%
if(request.getParameter("_escaped_fragment_")!=null) {
String escapedFragment = request.getParameter("_escaped_fragment_");
String decodedEscapedFrag = URLDecoder.decode(escapedFragment,"UTF-8");
response.sendRedirect(App.context()+decodedEscapedFrag+".html");
return;
}
%>


In the above code snippet notice that I am appending '.html' to the end of the decodedEscapedFragment.  This code takes the hash fragment and redirects to the appropriate static html page.  For instance, if the link clicked has href="#!about", then request.getParameter("_escaped_fragment_") will be equal to "about".  The resulting redirect will have a value of "/about.html"

Test It Out!
Now that you are finished, there are a couple ways to see if you're content is now accessible.  For each URL containing your new hash fragment, replace "#!" with "?_escaped_fragment_=", and make sure your static html page is returned by your server.  You can try this out at my website.  You'll notice that http://dsswebdesign.com/#!/about can be changed to http://dsswebdesign.com/?_escaped_fragment_=about, and the page looks the same.

A second way to test this is to add your site to Webmaster Tools.  Under diagnostics there is an option to 'Fetch as Googlebot'.  From there you can make sure that each of your #! links are reached successfully by Google-bot.