russell@unturf.com
137d65cc65
- Fix case duplicates (guide→Guide, project→Project, etc.) - Consolidate single-post tags into broader categories - Add technology tags (Python, AWS, Salt, Docker, Kubernetes) to 93 posts - Add lib/tag_stats.py tool for checking tag usage - Update CLAUDE.md with tag guidelines
50 lines
1.8 KiB
ReStructuredText
50 lines
1.8 KiB
ReStructuredText
Google Bot Attempts to Crawl Shortest Urls First
|
|
################################################
|
|
:date: 2011-06-25 09:37
|
|
:author: Russell Ballestrini
|
|
:tags: Code, Greatest Hits, Opinion, Project, Python
|
|
:slug: google-bot-attempts-to-crawl-shortest-urls-first
|
|
:status: published
|
|
:summary:
|
|
|
|
**Recently I built https://school.yohdah.com a Python, Pyramid, and
|
|
mongoDB project during the last couple weekends.**
|
|
|
|
|image0|
|
|
|
|
The site features a directory style navigation of nearly every public
|
|
school in the US. We have 61 state pages, approximately 19,000 city
|
|
pages, and over 103,000 school pages.
|
|
|
|
It seems the Google Bots have noticed
|
|
`school.yohdah.com <https://school.yohdah.com>`__ and started crawling
|
|
the site. Since the initial crawl I started reviewing a sample of the
|
|
sites apache logs in an attempt to track the bot's activity. After a few
|
|
minutes of viewing the logs, I locked onto a pattern; Google Bot's
|
|
algorithm appears to crawl the short URLs first!
|
|
|
|
PersonalCompute (a user) attached a graph of the fetched URL lengths
|
|
here:
|
|
|
|
|school.yohdah.com.graph|\
|
|
|
|
I have attached a zip containing the apache google bot crawl logs
|
|
here:
|
|
`access-school.yohdah.log.zip </uploads/2011/06/access-school.yohdah.log_.zip>`_
|
|
|
|
I found the pattern by opening the file in vim and scrolling very
|
|
quickly down. You will notice the log lines will grow slowly to the
|
|
right, as the urls being fetched increase by one character.
|
|
|
|
**Why does Google do this? Does anyone have speculation as to what this
|
|
means?**
|
|
|
|
.. |image0| image:: /uploads/2011/06/us-public-schools1.png
|
|
:target: https://school.yohdah.com
|
|
:alt: school.yohdah.com
|
|
:class: wordwrap-left
|
|
|
|
.. |school.yohdah.com.graph| image:: /uploads/2011/06/school.yohdah.com_.graph_.png
|
|
:target: /uploads/2011/06/school.yohdah.com_.graph_.png
|
|
|
|
.. contents::
|