Data Scrape where last page still has a "next page" arrow

If you use the Data Scraping wizard to make a data table from a webpage table, and that webpage table has multiple pages, if the last page of the data has a next page button (the button is just greyed out so you can’t actually click it) will the activity fail? I’m trying to hone in on an issue I’m having and I suspect this may be the culprit.

@skarbrevik

Please open uiexplore and try to modify selector for next page

If possible pleaSe provide link of web portal.

Regards,
Aditya

@rahatadi

Thanks for the reply. I have validated the selector and it doesn’t seem to have anything strange that would make it too specific or vague. I can’t provide the particular webpage as it’s private. The data scraping works perfectly if I specify a MaxResults that is lower than the actual results of the table. So turning to new pages in the data is not normally an issue.

The issue only appears if I choose 0 or a number above the actual max results of the table I’m trying to scrape. It seems to be specifically when the scraping tool tries to look for the next page to get more results that something goes wrong.

The weirdest thing to me is that even if the data scraping tool is getting caught up on the idea of trying to get to another page that doesn’t exist, why does that corrupt all the data that was scraped to that point? When I set ContinueOnError to True and then write the Data Table to CSV, I get an error from the “write to csv” activity and I see that a CSV was output with three or four random characters in it and nothing else…

Here is the error that data scraping with 0 for MaxResults gives me if I have ContinuesOnError set to False:

“Message: Input string was not in a correct format.
Exception Type: System.FormatException”

I can provide an example that may be available to you as well.
Kibana dashboards use a pagination where upon last page the HTML object indicating the next page is still present, it’s just disabled.
It is difficult to tell in the selectors what is different between the two.

The HTML difference is:
Active Next link case:

  • <button class="kuiButton kuiButton--basic kuiButton--icon" ng-click="toolBarPagerButtons.nextPage()" ng-disabled="!toolBarPagerButtons.hasNextPage" data-test-subj="btnNextPage" uipath_custom_id="2"><span class="kuiButton__icon kuiIcon fa-chevron-right" uipath_custom_id="1"></span></button>
    

The last page situation is:

  • <button class="kuiButton kuiButton--basic kuiButton--icon" ng-click="toolBarPagerButtons.nextPage()" ng-disabled="!toolBarPagerButtons.hasNextPage" data-test-subj="btnNextPage" uipath_custom_id="2" disabled="disabled"><span class="kuiButton__icon kuiIcon fa-chevron-right" uipath_custom_id="1"></span></button>
    

So basically the selector for the next page should somehow to support a ‘does not contain’ kind of construct, so that you can state that you have a next page as long as the button tag does not have a field called ‘disabled’.

I fully agree with skarbrevik: the scraping should return at least whatever it scraped correctly until the last page. Currently in 2019.4.2 version the scraping returns with an empty object, which then promptly fails in the next step.

This is like a year later but incase someone struggles with this issue, try opening in UI Explore and check the css selector, that should be inactive when the next page button is greyed out.

1 Like