How to Create robots.txt File
Вставка
- Опубліковано 6 вер 2024
- Video tutorial showing how to create robots.txt file on a web server. Learn how to use Robots Exclusion Protocol directives to control user-agent web crawlers like Googlebot crawling.
IMPORTANT: most website owners believe that using robots.txt file directives will let Google not index certain URLs. This is NOT true. The only real use cases for using robots.txt file rules are:
1. if you want to let certain parts of a website not be crawled by Googlebot
2. if Googlebot is requesting to many URLs thus causing server performance issues.
Basically, to be able to control Google to not index certain URLs on a website, you will need to use other methods such as noindex.
Introduction to robots.txt
developers.goo...
More robots.txt file examples and how to's can be found here:
www.rankya.com...
RankYa is a passionate digital marketer, website optimizer, content creator, and a fully qualified web developer helping businesses of all sizes (big or small) to achieve better results online.
We love sharing our passion through freely available how to videos and courses related to Google (Search Console, Ads, Analytics, UA-cam), SEO, HTML5, Structured Data and WordPress on RankYa UA-cam channel.
Rest assured that these proven insights will serve you well as well.
Securely purchase your new private SEO courses online (instant access). Insights that also come with Guaranteed Results for your business website.
www.rankya.com...
A superb tutorial - thank you
You're very welcome!
A lovely demontration.
Thank you for the comment, since you're an expert in this field, may I ask, what do you think of Google's Bard or ChatGPT?
I am rarely using it @RankYa . Content at @Lexsense is user case tailored. This is not something you can produced with #ChatGPT software. 👍
Thanks for this!
You're welcomed Kyle, new videos are coming soon :-)
this is an amazing tutorial. thank you sir
You are most welcome Ali
So helpful you got a new sub
Great video, thanks for sharing.
I'm glad it has been of help Lee
Do I create this in text editor such as atom or visual studio? Isn't the root index.html?
Yes, the root folder which is usually named as public_html
Each time the URL is requested, server looks for a file named *index* (this could be index.html, index.php, index.htm etc.)
That means, when you create a file named robots.txt within the root folder, only search engine bots like Googlebot will request this file robots.txt
You should just use a text editor (e.g. Microsoft Notepad) to create your robots.txt file
@@RankYa Great, thanks for this. I have a website that was found as a wordpress website template as it was hosted too early. I have over the months changed this to a html and CSS website but the website only has one page ranked in search, when the website has 7 pages. I go to consol to ask for indexing again, but pages are never indexed on Google. Can you advise? Is it worth me changing domain name and relaunching the website on new server?
In your case you do not need to change domain name to get indexed by Google. Do reliase that Google does take bit of time for indexing. So your 7 URLs may still get indexed by Google. However, do realize that the URLs *must be original* unique and useful for Google to index it (I am presuming yours are) therefore, the only other thing you can do is send some quality backlinks pointing to your website for Google to find and index quicker. Please do let us know the outcome
thanks
You're welcome!