How to setup a robots.txt in Salesforce site
To those who have a little bit experience in site development and SEO optimisation, robots.txt is not an extraordinary term. They know what exactly it is, and what is the use of it. In Layman’s language, we can describe it as a guide for the robots that search engines send to search your sites. These robots are not supposed to search your site at real time, rather they index your site for further use. The work of robots.txt is to tell these pages that which bot can access which page, or if a bot can access anything at all. In a Salesforce site also robots are required. This is today’s topic of discussion how to write a robots.txt for salesforce page?
The aftermath
As this heading suggests, we are directly jumping to the aftermath, skipping the entire development of the site, community and VF pages. Assuming that it is already done, we shall move on now to the topic at hand, how can we create a robots.txt for this site. Now let us consider that you have created a few pages out of which you don’t want the robot to touch the login page and index every other page. For this to work, we will first create a VF page with any name that you can think of and add the below code.
<apex:page contentType="text/plain"> User-agent: * Allow: / Disallow: /login </apex:page>
With this, half of our work is done. Now we have to just add it in the place of the default robots.txt. Did I forget to mention, Salesforce also generates a robots.txt? Yes, the default robots.txt allows the bots to crawl every page. We have made some modifications to the original robots.txt to disallow that very same bot from few pages.
To replace the original robots.txt go to Setup|Develop|Sites and click on your site name. Now click on the edit button on the top of the page. In the robots.txt section click the lookup icon and select the page that we created just now. Then save the site, and this will do the trick. Now go to your site and append ‘/robots.txt’ at the very end and you’ll see the updated robots.txt here.
Support
That’s all about setting up a robots.txt in Salesforce site, for any further queries feel free to contact us at:
https://wedgecommerce.com/contact-us/
Or let us know your views about this blog in comments section below.
