Comments on this hack

Showing messages 1 through 17 of 17.

• Scraping Google for Fun and Profit
  2009-11-21 06:15:21  automated [View]
  
  
  Thanks for the nice content.
  
  I found a much more in depth article I'd like to share here.
  It is called "Scraping Google for Fun and Profit" and contains a much more detailed source code project in PHP.
  It can get much more results and scrape multiple pages of a keyword instead of only the first.
  
  Here the link: http://google-scraper.squabbel.com

• Google Alert
  2003-07-16 14:05:00  anonymous2 [View]
  
  
  A popular free tool that uses the Google APIs is Google Alert (http://www.googlealert.com). It runs daily Google searches for you and emails you whenever new results appear.
  
  It has the ability to send out the results as HTML or RSS feeds.

• Is there going to be a coldfusion version?
  2003-05-28 11:26:10  anonymous2 [View]
  
  
  That would be highly valuable.
  
  It is really amazing, all the things you can do with google.
  
  I really respect their developmental mindset, in terms of encouraging, growth and new ideas.
  
  While also sticking to standards that make sense.
  
  BTW, webmasterworld, is the place to discuss google. http://www.webmasterworld.com

• regex problems
  2003-04-19 21:12:33  abeusher [View]
  
  
  I am miffed. For whatever reason, I cannot get it to work either.
  
  I think the regex in line 12 might be causing the problem.

• cgi version
  2003-04-16 21:30:46  netaustin [View]
  
  
  I posted a CGI version of this code for the convenience of people who don't understand or don't want to mess with Perl on their own system. Check out http://gcsv.netaustin.net. The script is a little different from this one, but they do the same thing. I also posted a different version of the command line code.
  
  The neatest thing that this set of Google tools can do is search directly via Soap and return you the results in a sortable table or in .csv without you ever bothering to download anything or write your own script.

• Of Errata and Fragility
  2003-04-14 15:54:01  Rael Dornfest | [View]
  
  
  Howdy,
  
  Those of you who have given this hack a whirl, whether from this online version or the book, have a couple of forces working against you...
  
  First, a typo in the script prevents it from returning anything useful. The print on the third-to-last line should be on the second-to-last line, reading:
  
  

  print qq{"$title","$url","$size","$suffix"\n};

  
  
  Second, as we noted in the introduction to this chapter (see "Things to Keep in Mind While Scraping: Scrapers break" in the book), scraping is a fragile art subject to the whims or the Webmaster tending the format of the site/page you're scraping. Google altered a simple paragraph tag in its HTML:
  
  

  <p>

  
  
  and added an HTML comment:
  
  

  <p class=g><!--m-->

  
  
  breaking our scraper. If you simply remove the <p> from the regular expression match, leaving:
  
  

  m!<a href="?(.+?)"?>(.+?)</a>.+?\s+-\s+(\d+k)?!mgis

  
  
  all will again be well.
  
  We've made the appropriate alterations to the source code of this hack online and will be doing the same in the next printing of the book.
  
  Of course that doesn't mean the second bug won't re-appear, should Google decide again to alter their HTML. But then that's the nature of the scraping business.
  
  Rael
• Of Errata and Fragility
  2003-04-14 22:15:05  anonymous2 [View]
  
  
  Tried your suggestion and it worked like a charm!
  
  Great hack, now I'll have to buy the book.
  
  mikeg

• i am missing something
  2003-04-10 10:42:02  anonymous2 [View]
  
  
  Those of us Perl know-nothings need slightly more.
  
  I’m using Windows 98.
  I make a folder.
  I copy the code from the webpage to notepad and save the script in the folder.
  I change the name of the file from google2csv.txt to google2csv.pl
  I save the google search to the folder as a restlts.htm
  I go to a prompt. Go to the folder.
  I type $ perl script.pl < results.htm > results.csv [Enter]
  And I get Bad command or file name
  
  What am I missing?
  
• i am missing something
  2003-04-10 10:53:28  anonymous2 [View]
  
  
  oops
  i mis-spoke
  
  I type $ perl google2csv.pl < results.htm > results.csv [Enter]
  And I get Bad command or file name
  
  
• i am missing something
  2003-04-10 22:13:26  anonymous2 [View]
  
  
  Try leaving out the $? And make sure you have Perl installed.

• hmm
  2003-04-09 15:45:48  anonymous2 [View]
  
  
  umm hi

• doesn't work
  2003-04-01 22:36:10  anonymous2 [View]
  
  
  I'm not sure if it is the result of google's HTML changing, but I have not been successful in getting this, or any of the perl scraping scripts, to work. Unfortunately, I don't know perl well enough to modify these scripts so that they will work again. Mildly disappointing.
• doesn't work
  2003-04-09 22:14:08  anonymous2 [View]
  
  
  I have tried it several times also, and I found it not to work.
• doesn't work (it's fine)
  2003-04-10 05:12:27  anonymous2 [View]
  
  
  The Perl code is fine, it's probably how you're saving it. If you're on OS X, use pico from the terminal or BBEdit from Bare Bones. Windows, use Pico, Notepad, Wordpad, etc.
  The scripts will need to be saved with a line feed (LF) at the end of each line, UNIX style.
• doesn't work (it's fine) Not for me
  2003-04-10 14:09:42  rodger [View]
  
  
  It's not fine for me.
  I run the script and it only prints the first line.
  "title","url","size","domain suffix"
  When I run it using the debugger, this is what I found.
  Command 1 - Displays to the screen. OK
  Command 2 - Loads file into $results var. OK
  Command 3 - The while fails and falls through the loop without doing anything. BAD
  
  Am I missing something?
  
• doesn't work (it's fine) Not for me
  2005-07-07 16:57:51  mlozier [View]
  
  
  Hi Rodger-
  I was getting this same problem as well, but I just removed the '

  ' between the 'm!' and the '

• doesn't work (it's fine) Not for me
  2003-08-02 10:10:51  anonymous2 [View]
  
  
  I'm having the same problem trying this hack and also #47 Scraping Google News. It makes a CSV file with the column titles, but no data from the web pages. Any suggestions, anyone?