I want to extract all my past Tweets into Excel (click on Tweets from your home page). A two-minute job, or so I thought. I set up Data Scraping to select Date and Tweet content and set zero in the number field so all Tweets would be returned. Twitter (on the page that presents all your past Tweets) doesn’t have a “next page” button, so I left that blank. This activity I followed with a “Write CSV”.
The robot works as required, extracting Date and Tweet body into Excel columns, EXCEPT that it captures only the first 120 Tweets or so (out of >1700).
The problem is, Twitter doesn’t display all the Tweets but selects a few more to display when the slider is moved (close) to the end of its “travel”. The robot evidently does not trigger this behaviour. Is there something that can be entered into the “NextLinkSelector” to keep displaying the Tweets or is an altogether more complex, procedural approach needed?