Chapter 9. Image-Capturing Webbots
In this chapter, I’ll describe a webbot that identifies and downloads all of the images on a web page. This webbot also stores images in a directory structure similar to the directory structure on the target website. This project will show how a seemingly simple webbot can be made more complex by addressing these common problems:
Finding the page base, or the address that defines the address from which all relative addresses are referenced
Dealing with changes to the page base, caused by page redirection
Converting relative addresses into fully resolved URLs
Replicating complex directory structures
Properly downloading image files with binary formats
In Chapter 17, you’ll expand on these concepts to develop a spider ...
Get Webbots, Spiders, and Screen Scrapers, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
