How is Google finding pages which don’t have any links to them?

How can Googlebot crawl and index pages that don’t have any links to them on my website? I find each day two or three pages in the index that don’t have any links to them on my site. The pages are generated by the search field of my website. Danny, Bucharest, Romania
Video Rating: 4 / 5

www.SERoundtable.com – This week at the Search Engine Roundtable, we noticed that Yahoo and Bing should begin testing the Bing search results directly in the Yahoo Search interface any day now and launch by the end of September. Google may have tweaked the May Day ranking update on June 2nd. GoogleBot slowed it’s crawling process on June 15th, was it a bug? Google promises to fix the soft-404 misclassifications. AdWords is tested related AdWords ads on the search results. Is the World Cup resulting in less traffic and revenue for your site? Don’t hide links from Google, don’t. Google calls themselves the “biggest kingmaker on this earth.” US government wants to classify search marketers. Google didn’t do a logo for Flag Day but Bing, DogPile and others did. That was this past week at the Search Engine Roundtable. Yahoo Begins Testing Bing Search Results Now, Switch Over Complete August/September : www.seroundtable.com Was There a Google May Day Tweak On June 2nd? : www.seroundtable.com GoogleBot Stop Crawling Your Site On June 15th? : www.seroundtable.com Google To Fix Soft 404 Reporting in Google Webmaster Tools : www.seroundtable.com Google AdWords Testing Related Broad Matches AdWords Ads : www.seroundtable.com World Cup: Webmasters Complaining About Less Searches, Traffic & Ad Clicks : www.seroundtable.com Hide Links From Google? Google Says No To SEO : www.seroundtable.com Google Says: We Are “The Biggest Kingmaker On This Earth” : www.seroundtable.com US Government
Video Rating: 5 / 5

25 Replies to “How is Google finding pages which don’t have any links to them?”

  1. If the pages are generated by the search field of the website, then they are linked to somewhat.

  2. @SEOLEX Your right, however I believe that when Matt mentioned robots.txt he was more or less referring to URL’s googlebot will discover by self-fulfilled forms and not from other sites.

  3. Another case… The user can search for something in your “search box”. Find an interesting result, copy the URL and post on Twitter.

  4. Another case… The user can search for something in your “search box”. Find an interesting result, copy the URL and post on Twitter.

  5. All major browsers like FF, safari, chrome, etc, sent all urls surfed to google. So all this video is wrong as he didn’t say main thing.

  6. And what about The Google Toolbar acting as spyware submitting reports of websites visited for indexing purposes?

  7. Matt which are those text you submit, ha vae a ecommerce site. and spider has crawl the XAMPP etc folder..

  8. How come when you do a search and get like 300,000 results, only the first 20-30 links are relevant and the rest have no connection to the search query? Hmmm?

  9. @SEOLEX Read the question: “How is Google finding pages which don’t have any links to them?” in THAT situation, robots.txt can work; robots.txt cannot work *if there are other links* – but it CAN work where there are none. Which is what he said.

  10. @heenan73 It makes no sense to me what you say? I’m not the one asking and not the one wishing to FTP content that should not be in the index… I’m only asking why Matt states that robots.txt can prevent it when he in other videos state the opposite. I think you’ve got me all wrong?

  11. @SEOLEX robots.txt does work as stated; it cannot prevent a site being picked up if it has other links; but the point remains, if you don’t want it listed, don’t FTP it – and don’t get incoming links. Why is everything always Google’s fault? YOU manage your content; Google obeys the rules …. you have to as well. Remember the original question was “pages that don’t have any links” – in that context, Matt was 100% right – you seem to be rewriting the question.

  12. @heenan73 I don’t know. Please understand that I am not the one asking the question in Matts video. I just wondered why Matt said that robots.txt will prevent the content from being visible in Google since that is not how robots.txt works 😉

  13. i’ve also made the experience, that crawlers started visiting my non-linked pages after i had sent an *email with the direct url*

  14. @NMITYou If you add robots.txt after the listing has occurred, it may be a while before the pages drop out of the index. And many people have errors in their robots.txt

  15. @SEOLEX If you really don’t want content seen, why put it on the web? Robots.txt is not a perfect solution, but Google does respect it – many other spiders do not. Or you could use password protection?

Leave a Reply

Your email address will not be published. Required fields are marked *