{"id":53592,"date":"2016-01-19T16:28:30","date_gmt":"2016-01-19T15:28:30","guid":{"rendered":"https:\/\/www.humanlevel.com\/diccionario-marketing-digital\/robots-txt"},"modified":"2020-06-02T09:25:00","modified_gmt":"2020-06-02T07:25:00","slug":"robots-txt","status":"publish","type":"dicmarketing","link":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt","title":{"rendered":"Robots.txt"},"content":{"rendered":"<p><iframe loading=\"lazy\" title=\"\u00bfQu\u00e9 es ROBOTS.TXT? | Diccionario de Marketing Online de Human Level\" width=\"1200\" height=\"675\" src=\"https:\/\/www.youtube.com\/embed\/uTo5vws9aGE?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<h2>What is the robots.txt file?<\/h2>\n<p><strong><img loading=\"lazy\" class=\" size-full wp-image-22085 alignleft\" src=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/robot4.png\" alt=\"robot icon\" width=\"128\" height=\"128\" \/>Robots.txt is a text file<\/strong> with a .txt extension, which we create and upload to our website and <strong>use to prevent the robots of certain search engines from crawling content that we do not want them to index or display in their results<\/strong>.<\/p>\n<p>That is to say, it is a public file that we use to <strong>indicate to those crawlers or spiders which part or parts of our web page should not be crawled and indexed<\/strong>. In it, we can specify in a simple way, the directories, subdirectories, URLs or files of our website that should not be crawled or indexed by search engines.<\/p>\n\n<p>Because <strong>is closely related to website indexing<\/strong>, it is essential to program this file properly, especially if our website is made with a <a href=\"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/cms-content-manager\" target=\"_blank\" rel=\"noopener noreferrer\">content management system (CMS)<\/a> The system will automatically generate it, since it may happen that parts that should be crawled are accidentally included as non-indexable.<\/p>\n<p>Also called <em>robots exclusion protocol<\/em> or <em>robots.txt protocol<\/em>, it<strong> is advisory and does not guarantee full secrecy, <\/strong>but we sometimes find it used to keep parts of a website private. Since this isolation is not complete<strong>, its use is not recommended to keep certain areas private<\/strong>, as it serves as a recommendation and not as an obligation, being a treat for hackers who, with a browser and the necessary knowledge, can easily access them.<\/p>\n<p>Normally the most common uses are to prevent accessibility to certain parts of the website, to prevent the indexing of duplicate content (e.g. printable versions of the website), or to tell Google what our sitemap is, including its URL in the file.<\/p>\n<p><em>http:\/\/www.example.com\/sitemap.xml<\/em><\/p>\n<p class=\"p1\">[banner id=&#8221;39438&#8243;][\/banner]<\/p>\n<h2>How do we create the robots.txt file?<\/h2>\n<p>In order to create it, we need access to the root of the domain and upload the text file named &#8220;robots.txt&#8221; to the top-level root directory of our web server.<\/p>\n<p><em>http:\/\/www.example.com\/robots.txt<\/em><\/p>\n<figure id=\"attachment_22087\" aria-describedby=\"caption-attachment-22087\" style=\"width: 377px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" class=\"size-full wp-image-22087\" src=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/robotx.txt-de-Matt-Cutts.png\" alt=\"example robots.txt\" width=\"387\" height=\"102\" \/><figcaption id=\"caption-attachment-22087\" class=\"wp-caption-text\">sample robots.txt file by Matt Cutts<\/figcaption><\/figure>\n<h2>Robots.txt file elements<\/h2>\n<h3>Commands<\/h3>\n<p>The main commands that we will use in a robots.txt will be<\/p>\n<ul>\n<li><strong><em>User-agent<\/em>\u00a0<\/strong>are the robots or spiders of the search engines. You can see most of them in this <a title=\"Go to robots.txt listing\" href=\"http:\/\/www.robotstxt.org\/db.html\" target=\"_blank\" rel=\"noopener noreferrer\">database of web robots<\/a>. Its syntax would be:<br \/>\n<code>User-agent: [name of the robot to which the rule will be applied]<\/code><\/li>\n<li><strong><em>Disallow <\/em><\/strong>tells the user agent not to access, crawl or index a particular URL, subdirectory or directory.<br \/>\n<code>Disallow: [directory you want to block]<\/code><\/li>\n<li><em><strong>Allow\u00a0<\/strong><\/em>arises as against the previous one, with it you indicate to the crawler a URL, subdirectory or directory to which it should enter, crawl or index.<br \/>\n<code>Allow: [URL of a blocked directory or subdirectory you want to unblock]<\/code><\/li>\n<\/ul>\n<h3>Specific rules<\/h3>\n<p>The <strong>rules specified in the <em>Disallow<\/em> and<em> Allow<\/em><\/strong> only apply to the user agents that we have specified in the line before them. Several <em>Disallow<\/em> lines can be included to different user agents.<\/p>\n\n<h3>Other elements<\/h3>\n<ul>\n<li><strong>Slash &#8220;\/&#8221;, <\/strong>must be attached before the element you want to lock.<\/li>\n<li><strong>Matching rules<\/strong> are patterns that can be used to simplify the code of the robots.txt file.<br \/>\nExample: *, ?, $<br \/>\nAsterisk (*): locks a sequence of characters<br \/>\nDollar sign ($): to block URLs that end in a specific form<\/li>\n<\/ul>\n<h2>Most used command syntax in robots.txt<\/h2>\n<ul>\n<li><strong>Indications to a specific bot:<\/strong><\/li>\n<\/ul>\n<p><code>User-agent: [bot name]<br \/>\nUser-agent: Googlebot<\/code><\/p>\n<ul>\n<li><strong>Indications to all bots:<\/strong><\/li>\n<\/ul>\n<p><code>User-agent: *<\/code><\/p>\n<ul>\n<li><strong>Blocking of the entire website,<\/strong> using a slash &#8220;\/&#8221;:<\/li>\n<\/ul>\n<p><code>Disallow: \/<\/code><\/p>\n<ul>\n<li><strong>To lock a directory and its contents<\/strong>, include the directory name after the slash:<\/li>\n<\/ul>\n<p><code>Disallow: \/directory\/<\/code><\/p>\n<ul>\n<li><strong>To block a specific web page,<\/strong> indicate the specific page after the bar:<\/li>\n<\/ul>\n<p><code>Disallow: \/private-page.html<\/code><\/p>\n<ul>\n<li><strong>Block all images on the website:<\/strong><\/li>\n<\/ul>\n<p><code>User Agent: Googlebot-Image<br \/>\nDisallow: \/<\/code><\/p>\n<ul>\n<li><strong>Lock a single image,<\/strong> specifies the image behind the slash:<\/li>\n<\/ul>\n<p><code>User-agent: Googlebot-Image<br \/>\nDisallow: \/image\/private.jpeg<\/code><\/p>\n<ul>\n<li><strong>Block a specific file type,<\/strong> mentioning, after the slash, the extension:<\/li>\n<\/ul>\n<p><code>User-agent: Googlebot<br \/>\nDisallow: \/*.png$<\/code><\/p>\n<ul>\n<li><strong>To block a sequence of characters<\/strong>, use the asterisk:<\/li>\n<\/ul>\n<p><code>User-agent: <\/code><strong><br \/>\n<\/strong><code>Disallow: \/private-directory*\/<\/code><\/p>\n<ul>\n<li><strong>To block URLs that end in a specific form<\/strong>, add the $ symbol at the end:<\/li>\n<\/ul>\n<p><code>User-agent: *<br \/>\nDisallow: \/*.pdf$<\/code><\/p>\n<ul>\n<li><strong>Allow full access to all robots:<\/strong><\/li>\n<\/ul>\n<p><code>User-agent: *<br \/>\nDisallow:<\/code><\/p>\n<p>Another way would be to not use the robots.txt file or leave it empty.<\/p>\n<ul>\n<li><strong>Block a specific robot or bot:<\/strong><\/li>\n<\/ul>\n<p><code>User-agent: [bot name]<br \/>\nDisallow: \/<\/code><\/p>\n<ul>\n<li><strong>Allow tracking to a specific bot:<\/strong><\/li>\n<\/ul>\n<p><code>User-agent: [bot name]<br \/>\nDisallow:<\/code><\/p>\n<p><code>User-agent: *<br \/>\nDisallow: \/<\/code><\/p>\n<p>When writing them you must take into account that there is a distinction between uppercase, lowercase and spaces.<\/p>\n<h2>Testing the robots.txt file in Google<\/h2>\n<p>To test the operation of the robots.txt file, we have the<strong> test tool for robot.txt in Google Search Console<\/strong>, where you can test and see how the <a title=\"Definition of Googlebot in our online marketing dictionary\" href=\"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/googlebot\" target=\"_blank\" rel=\"noopener noreferrer\">Googlebot<\/a> will read it, so that it will show you possible errors or defects that the file has or may cause.<\/p>\n<p>To perform the test, go to <a title=\"Go to Google Search Console\" href=\"https:\/\/search.google.com\/search-console\/about\" target=\"_blank\" rel=\"noopener noreferrer\">Google Search Console<\/a> and in its control panel, in the Crawl section, choose the option <strong>&#8220;<em>Robots.txt tester<\/em>&#8220;<\/strong>.<\/p>\n<figure id=\"attachment_22088\" aria-describedby=\"caption-attachment-22088\" style=\"width: 179px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" class=\" wp-image-22088\" src=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/Probador-de-robots-de-Google-Search-Console.png\" alt=\"robots.txt tester\" width=\"189\" height=\"443\" \/><figcaption id=\"caption-attachment-22088\" class=\"wp-caption-text\">How to use the robots.txt tester in Google Search Console<\/figcaption><\/figure>\n<p>Inside the tester, your current robots.txt file will appear, you can edit it, or copy and paste the one you want to test. Once you have written the robots file to test, select the URL you want to check if it is going to be blocked and the crawler robot you want to test it with.<\/p>\n<figure id=\"attachment_22089\" aria-describedby=\"caption-attachment-22089\" style=\"width: 590px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/Probador-de-robots.txt-de-Google-Search-Console.jpg\"><img loading=\"lazy\" class=\" wp-image-22089\" src=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/Probador-de-robots.txt-de-Google-Search-Console.jpg\" alt=\"Google Search Console robots.txt tester\" width=\"600\" height=\"426\" srcset=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/Probador-de-robots.txt-de-Google-Search-Console.jpg 997w, https:\/\/www.humanlevel.com\/wp-content\/uploads\/Probador-de-robots.txt-de-Google-Search-Console-400x284.jpg 400w, https:\/\/www.humanlevel.com\/wp-content\/uploads\/Probador-de-robots.txt-de-Google-Search-Console-754x535.jpg 754w, https:\/\/www.humanlevel.com\/wp-content\/uploads\/Probador-de-robots.txt-de-Google-Search-Console-900x639.jpg 900w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><figcaption id=\"caption-attachment-22089\" class=\"wp-caption-text\">Edit your robots.txt file, choose the URL you want to check if it is affected, the robot that will crawl it, and test<\/figcaption><\/figure>\n<p>The tool will give you two options: <em><strong>&#8220;allowed&#8221; <\/strong><\/em>i.e. the URL is not blocked, or <em><strong>&#8220;blocked&#8221;<\/strong>, <\/em>indicating the line of code that is blocking that URL.<\/p>\n<figure id=\"attachment_22090\" aria-describedby=\"caption-attachment-22090\" style=\"width: 590px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/URL-bloqueada-en-probador-de-robots.txt-en-Google-Search-Console.png\"><img loading=\"lazy\" class=\" wp-image-22090\" src=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/URL-bloqueada-en-probador-de-robots.txt-en-Google-Search-Console.png\" alt=\"URL blocked in the robots.txt tester of Google Search Console\" width=\"600\" height=\"508\" srcset=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/URL-bloqueada-en-probador-de-robots.txt-en-Google-Search-Console.png 668w, https:\/\/www.humanlevel.com\/wp-content\/uploads\/URL-bloqueada-en-probador-de-robots.txt-en-Google-Search-Console-400x339.png 400w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><figcaption id=\"caption-attachment-22090\" class=\"wp-caption-text\">Example of how the robots.txt tester tells us when a URL is blocked and the line of code where the block is generated<\/figcaption><\/figure>\n<h2>References<\/h2>\n<p><a title=\"Go to robotstxt.org\" href=\"http:\/\/www.robotstxt.org\/robotstxt.html\" target=\"_blank\" rel=\"noopener noreferrer\">Information about the robots.txt file on the robots website<\/a><\/p>\n<p><a title=\"Go to Google Help on Robots.txt\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro?hl=es&amp;visit_id=638244996066636623-3334235526&amp;rd=1\" target=\"_blank\" rel=\"noopener noreferrer\">Google help on robots.txt file<\/a><\/p>\n\n<section class=\"rounded-xlg cta-image\">\n<div>\n<p class=\"gb-text cta-abovetitle\">Tu marca, visible y encontrable<\/p>\n\n\n\n<p class=\"gb-text cta-title\">Est\u00e1s a un paso de dominar tu categor\u00eda online<\/p>\n\n\n\n<a class=\"gb-text btn\" href=\"#anchor-formulario\">Start now<\/a>\n<\/div>\n\n\n\n<img loading=\"lazy\" class=\"gb-media-f45eabef\" alt=\"\" title=\"HL20-SEO\" src=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/HL20-SEO-1-e1770805082974.png\" width=\"500\" height=\"400\"\/>\n<\/section>\n\n","protected":false},"excerpt":{"rendered":"<p>What is the robots.txt file? Robots.txt is a text file with a .txt extension, which we create and upload to&#8230;<\/p>\n","protected":false},"author":13,"featured_media":48015,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"area":[412,410],"letra":[399],"class_list":["post-53592","dicmarketing","type-dicmarketing","status-publish","format-standard","has-post-thumbnail","hentry","area-seo","area-web-development","letra-r"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>What is robots.txt file? | Human Level<\/title>\n<meta name=\"description\" content=\"This file is used to prevent the robots of certain search engines from crawling content that we do not want them to index or display in their results.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is robots.txt file? | Human Level\" \/>\n<meta property=\"og:description\" content=\"This file is used to prevent the robots of certain search engines from crawling content that we do not want them to index or display in their results.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt\" \/>\n<meta property=\"og:site_name\" content=\"Human Level\" \/>\n<meta property=\"article:modified_time\" content=\"2020-06-02T07:25:00+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/robotstext.png\" \/>\n\t<meta property=\"og:image:width\" content=\"400\" \/>\n\t<meta property=\"og:image:height\" content=\"400\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"What is robots.txt file\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.humanlevel.com\/wp-content\/uploads\/Robots-e1443164497926.jpg\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":[\"Article\",\"BlogPosting\"],\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt\"},\"author\":{\"name\":\"Fani S\u00e1nchez\",\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en#\\\/schema\\\/person\\\/ab0429d9436e7e29a8d3d9835f6fd8a1\"},\"headline\":\"Robots.txt\",\"datePublished\":\"2016-01-19T15:28:30+00:00\",\"dateModified\":\"2020-06-02T07:25:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt\"},\"wordCount\":1090,\"publisher\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.humanlevel.com\\\/wp-content\\\/uploads\\\/robotstext.png\",\"inLanguage\":\"en-US\"},{\"@type\":[\"WebPage\",\"ItemPage\"],\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt\",\"url\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt\",\"name\":\"What is robots.txt file? | Human Level\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.humanlevel.com\\\/wp-content\\\/uploads\\\/robotstext.png\",\"datePublished\":\"2016-01-19T15:28:30+00:00\",\"dateModified\":\"2020-06-02T07:25:00+00:00\",\"description\":\"This file is used to prevent the robots of certain search engines from crawling content that we do not want them to index or display in their results.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt#primaryimage\",\"url\":\"https:\\\/\\\/www.humanlevel.com\\\/wp-content\\\/uploads\\\/robotstext.png\",\"contentUrl\":\"https:\\\/\\\/www.humanlevel.com\\\/wp-content\\\/uploads\\\/robotstext.png\",\"width\":400,\"height\":400},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-dictionary\\\/robots-txt#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Inicio\",\"item\":\"https:\\\/\\\/www.humanlevel.com\\\/en\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SEO\",\"item\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/digital-marketing-glossary\\\/seo\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Robots.txt\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en#website\",\"url\":\"https:\\\/\\\/www.humanlevel.com\\\/en\",\"name\":\"Human Level\",\"description\":\"Web positioning and online marketing consultant Human Level\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.humanlevel.com\\\/en?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en#organization\",\"name\":\"Human Level\",\"url\":\"https:\\\/\\\/www.humanlevel.com\\\/en\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.humanlevel.com\\\/wp-content\\\/uploads\\\/logohl25x3.png\",\"contentUrl\":\"https:\\\/\\\/www.humanlevel.com\\\/wp-content\\\/uploads\\\/logohl25x3.png\",\"width\":600,\"height\":93,\"caption\":\"Human Level\"},\"image\":{\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.linkedin.com\\\/company\\\/human-level-communications\",\"https:\\\/\\\/www.youtube.com\\\/user\\\/humanlevelcommunica\",\"https:\\\/\\\/bsky.app\\\/profile\\\/humanlevel.bsky.social\",\"https:\\\/\\\/instagram.com\\\/humanlevel\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.humanlevel.com\\\/en#\\\/schema\\\/person\\\/ab0429d9436e7e29a8d3d9835f6fd8a1\",\"name\":\"Fani S\u00e1nchez\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e3df2162f547c63a9e6304eaa2736129a4c817f1111b6bd9cca01fba18d6974c?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e3df2162f547c63a9e6304eaa2736129a4c817f1111b6bd9cca01fba18d6974c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e3df2162f547c63a9e6304eaa2736129a4c817f1111b6bd9cca01fba18d6974c?s=96&d=mm&r=g\",\"caption\":\"Fani S\u00e1nchez\"},\"description\":\"Former senior SEO consultant at Human Level. Graduated in Advertising and Public Relations.\",\"sameAs\":[\"https:\\\/\\\/www.linkedin.com\\\/in\\\/fanisanchez\",\"https:\\\/\\\/x.com\\\/faniperdia\"],\"url\":\"https:\\\/\\\/www.humanlevel.com\\\/en\\\/author\\\/fani\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is robots.txt file? | Human Level","description":"This file is used to prevent the robots of certain search engines from crawling content that we do not want them to index or display in their results.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt","og_locale":"en_US","og_type":"article","og_title":"What is robots.txt file? | Human Level","og_description":"This file is used to prevent the robots of certain search engines from crawling content that we do not want them to index or display in their results.","og_url":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt","og_site_name":"Human Level","article_modified_time":"2020-06-02T07:25:00+00:00","og_image":[{"width":400,"height":400,"url":"https:\/\/www.humanlevel.com\/wp-content\/uploads\/robotstext.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_title":"What is robots.txt file","twitter_image":"https:\/\/www.humanlevel.com\/wp-content\/uploads\/Robots-e1443164497926.jpg","twitter_misc":{"Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["Article","BlogPosting"],"@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt#article","isPartOf":{"@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt"},"author":{"name":"Fani S\u00e1nchez","@id":"https:\/\/www.humanlevel.com\/en#\/schema\/person\/ab0429d9436e7e29a8d3d9835f6fd8a1"},"headline":"Robots.txt","datePublished":"2016-01-19T15:28:30+00:00","dateModified":"2020-06-02T07:25:00+00:00","mainEntityOfPage":{"@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt"},"wordCount":1090,"publisher":{"@id":"https:\/\/www.humanlevel.com\/en#organization"},"image":{"@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt#primaryimage"},"thumbnailUrl":"https:\/\/www.humanlevel.com\/wp-content\/uploads\/robotstext.png","inLanguage":"en-US"},{"@type":["WebPage","ItemPage"],"@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt","url":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt","name":"What is robots.txt file? | Human Level","isPartOf":{"@id":"https:\/\/www.humanlevel.com\/en#website"},"primaryImageOfPage":{"@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt#primaryimage"},"image":{"@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt#primaryimage"},"thumbnailUrl":"https:\/\/www.humanlevel.com\/wp-content\/uploads\/robotstext.png","datePublished":"2016-01-19T15:28:30+00:00","dateModified":"2020-06-02T07:25:00+00:00","description":"This file is used to prevent the robots of certain search engines from crawling content that we do not want them to index or display in their results.","breadcrumb":{"@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt#primaryimage","url":"https:\/\/www.humanlevel.com\/wp-content\/uploads\/robotstext.png","contentUrl":"https:\/\/www.humanlevel.com\/wp-content\/uploads\/robotstext.png","width":400,"height":400},{"@type":"BreadcrumbList","@id":"https:\/\/www.humanlevel.com\/en\/digital-marketing-dictionary\/robots-txt#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Inicio","item":"https:\/\/www.humanlevel.com\/en"},{"@type":"ListItem","position":2,"name":"SEO","item":"https:\/\/www.humanlevel.com\/en\/digital-marketing-glossary\/seo"},{"@type":"ListItem","position":3,"name":"Robots.txt"}]},{"@type":"WebSite","@id":"https:\/\/www.humanlevel.com\/en#website","url":"https:\/\/www.humanlevel.com\/en","name":"Human Level","description":"Web positioning and online marketing consultant Human Level","publisher":{"@id":"https:\/\/www.humanlevel.com\/en#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.humanlevel.com\/en?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.humanlevel.com\/en#organization","name":"Human Level","url":"https:\/\/www.humanlevel.com\/en","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.humanlevel.com\/en#\/schema\/logo\/image\/","url":"https:\/\/www.humanlevel.com\/wp-content\/uploads\/logohl25x3.png","contentUrl":"https:\/\/www.humanlevel.com\/wp-content\/uploads\/logohl25x3.png","width":600,"height":93,"caption":"Human Level"},"image":{"@id":"https:\/\/www.humanlevel.com\/en#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.linkedin.com\/company\/human-level-communications","https:\/\/www.youtube.com\/user\/humanlevelcommunica","https:\/\/bsky.app\/profile\/humanlevel.bsky.social","https:\/\/instagram.com\/humanlevel"]},{"@type":"Person","@id":"https:\/\/www.humanlevel.com\/en#\/schema\/person\/ab0429d9436e7e29a8d3d9835f6fd8a1","name":"Fani S\u00e1nchez","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/e3df2162f547c63a9e6304eaa2736129a4c817f1111b6bd9cca01fba18d6974c?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/e3df2162f547c63a9e6304eaa2736129a4c817f1111b6bd9cca01fba18d6974c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e3df2162f547c63a9e6304eaa2736129a4c817f1111b6bd9cca01fba18d6974c?s=96&d=mm&r=g","caption":"Fani S\u00e1nchez"},"description":"Former senior SEO consultant at Human Level. Graduated in Advertising and Public Relations.","sameAs":["https:\/\/www.linkedin.com\/in\/fanisanchez","https:\/\/x.com\/faniperdia"],"url":"https:\/\/www.humanlevel.com\/en\/author\/fani"}]}},"_links":{"self":[{"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/dicmarketing\/53592","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/dicmarketing"}],"about":[{"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/types\/dicmarketing"}],"author":[{"embeddable":true,"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/comments?post=53592"}],"version-history":[{"count":3,"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/dicmarketing\/53592\/revisions"}],"predecessor-version":[{"id":53615,"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/dicmarketing\/53592\/revisions\/53615"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/media\/48015"}],"wp:attachment":[{"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/media?parent=53592"}],"wp:term":[{"taxonomy":"area","embeddable":true,"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/area?post=53592"},{"taxonomy":"letra","embeddable":true,"href":"https:\/\/www.humanlevel.com\/en\/wp-json\/wp\/v2\/letra?post=53592"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}