Creating Multiple Instances of a Webbot
There are three basic ways to create additional instances of a webbot:
Fork additional harvesting processes from the same process.
Use the operating system to create multiple instances of the same script.
Execute the same webbot on multiple pieces of hardware.
Forking Processes
Some webbot developers prefer to create new instances of the same webbot by forking processes from a single script. Forking is the method of creating somewhat independent scripts from a parent script. It allows a script to execute tasks in parallel. In the case of webbot development, forking could allow a single script to download web pages from multiple target websites at the same time.
Forking is mentioned only because it is something ...
Get Webbots, Spiders, and Screen Scrapers, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
