O'Reilly Hacks
oreilly.comO'Reilly NetworkSafari BookshelfConferences Sign In/My Account | View Cart   
Book List Learning Lab PDFs O'Reilly Gear Newsletters Press Room Jobs  



HACK
#44
Scraping Google Results
Scraping the results of a Google search into a comma-delimited file.
The Code

Contributed by:

[03/13/03 | Discuss (17) | Link to this hack]

Because you can use the Google API to get results and put them in any format you like, why in the world would you bother to do a manual search result on Google, save the results, and then scrape them with a Perl program? You might not want, or indeed be able, to do anything as fancy as the Google API allows; you might just want to grab some results, drop them into a spreadsheet, and go.

Just like we did in Peeling Phone Numbers [Hack #49], you can save Google web search results to a file, and then process them into a comma-delimited text file with a short Perl script.

TIP

Be sure to set your preferences [Hack #1] to 100 results per page to get the most out of this hack.

The Code

The Results

Here's a sample run on the results of a search for Mac OS X:

$ perl google2csv.pl < results.html
"title","url","size","domain suffix"
"Apple - Mac OS X","http://www.apple.com/macosx/","","com"
"Apple - Software - Mac OS X Server","http://www.apple.com/server/",
"29k","com"
"Mac OS X Development","http://developer.apple.com/macosx/","28k","com"
"Mac OS X Hints - Get the most from X!","http://www.macosxhints.com/",
"","com"
"Mac OS X Apps - The Source For Mac OS X Software",
"http://www.macosxapps.com/","39k","com"
"VersionTracker.com - free Macintosh software downloads for Mac 
OS ... ","http://www.versiontracker.com/macosx/","101k","com"
"O'Reilly Mac OS X Conference",
"http://conferences.oreillynet.com/macosx2002/","25k","com"
"MacNN | OS X","http://osx.macnn.com/","94k","com"
"???? - Mac OS X","http://www.apple.co.jp/macosx/","43k","jp"
"Apple - Support - Mac OS X",
"http://www.info.apple.com/usen/macosx/","36k","com"

You'll see that the program records four attributes to the CSV file: title, URL, size (when available), and top-level domain. The "snippet" of web page usually included with a Google result was omitted, because it's difficult to read in a spreadsheet format.

So why include the page size and domain? Research. If you're generating a set of results to be referred to later, it's handy to be able to sort them by suffix. "edu" results tend to be different from "org" results, which tend to be different from "com" results, and so on. Not to mention differing result sets by country, .uk versus .jp, for instance. And if you're generating links to contact later (to ask for a reciprocal link, for example), it's handy to be able to set apart the less-commercial suffixes such as .edu and .org.


O'Reilly Home | Privacy Policy

© 2007 O'Reilly Media, Inc.
Website: | Customer Service: | Book issues:

All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners.