I’ve been working on a data crawling process that requires open up the browser to crawl data and then close it, then repeats.
Here it goes:
I have created a queue of items, could be 2/300 items or more, basically links that I need to go through. Each link will lead me to a webpage which are structured in a similar way, for instance the Amazon item page.
So, the process would open the browser and go to this webpage and perform the data scraping activity, then close the browser, then repeat itself until it hits the bottom of the queue.
Somehow, after open/close the browser for a number of times (it varies, as each time I run the process to test it, the number changes), for instance 10 or 20 times, then I would loose the Internet connection, so that after that all the webpages won’t load up. All come back blank.
The bizarre part is that it seems only happening when I’m testing the process at home, but not anywhere else (all under Wi-Fi environment)
My guess is that the frequent network requests from this process overloads my router.
Is there anyone experiencing the same thing?
Any help would be appreciated!