Googlebot Spider Cannot Access CSS and JS of Joomla Site

If you are an user of brilliant Google Webmaster Tools, you probably received the message from our title from Google recently. This article explains how to fix it quickly.

There were several major upgrades in search algorithms during 2015. One of the most significant requests from Google for website owners was to check if Joomla site is mobile-friendly. Googlebot is a "spider" that crawls all websites and helps Google to position them properly on the search results page. Before the era of mobile devices, it was simpler software. It was reading textual data from the websites and also collecting information about links between the websites. The spiders are now improved. Nowadays, they are "looking" at the structure of the websites to check if they are mobile-friendly. They are also reading the website's JavaScript code.

No matter what platform you are using for your website, in the root folder of your site you will find "robots.txt" file. This is a plain text file with some instructions that search engine spiders should obey. Default configuration of Joomla, actually, default values in "robots.txt" file might disallow spiders to read CSS and JS of Joomla site. If you are running a Joomla website, that's the most common reason why you received the message from our title in your Google Webmaster Tools account.

To fix this, open the "robots.txt" file in your favorite plain text editor (such as Notepad) and look if there's a line "Disallow: /templates/". If you found this line, delete it. This will allow spiders to access the template's files. As you probably know, Joomla templates contain CSS and JS files besides the other files.

Joomla templates are important parts of each Joomla site, but we are also using components, modules and plugins to build our sites. These Joomla extensions also might contain some CSS and JS and files. You can guess that the "robots.txt" file also blocks the folders where components, modules, and plugins are placed. Good practice in Joomla development is to place all extension assets inside the media folder. Some developers are doing that, while others prefer to place all the assets inside the extension's folder.

Let's summarize the above text. In order to be 100% sure that Googlebot can access CSS and JS files of your Joomla Site, you must allow access to components, modules, plugins, and media folders of your site. If you decide to make this change in your website, you should delete these lines in your "robots.txt" file:

Disallow: /components/
Disallow: /images/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/

Note in the above code that we also "unlocked" the images folder. This is strongly recommended because it allows the listing of your images in search results.

Please be aware this operation might lead to unwanted results. Inside the unlocked folders, there could be some content that you don't want to be listed as a part of your website. For example, extensions might contain TXT or PDF files with instructions. They could also contain some images that are irrelevant to your website. Unlocking these folders will allow Googlebot to index this content as a part of your website. So, it's recommended to be careful and check the content of all folders that you are opening for the public.