Using the data scrapping wizard to scrape a table from multiple pages. On the last page it sits there until the timeout time is reached (30 seconds by default) then continues to the next activity. Is that how this is supposed to work?
The data all gets scrapped properly, but the delay on the last page is annoying. I can reduce the delay by setting the Timeout property, but then I risk the page not fully loading before it times out. How is the data scrapping wizard supposed to recognize that it is at the last page? I would think that once it finds that that there is no “next” link it would move to the next activity, but it seems that that instead of relying on a “complete” status from “WaitForReady” as the signal to move on if no “next” link is found, it is waiting the whole time out period.
Here are the details on data I’m scraping:
http://webapps.rrc.state.tx.us/PR/publicQueriesMainAction.do
Test query inputs
Lease Type: “Gas Well”
District: “06”
Prod Month Range: “Jan 2017 - Feb 2020”
Next button Selector: