Data scrapping from twitter

Hello,
I want to extract all post from this page https://twitter.com/CoinbasePro . I tried in several ways but nothing’s working. I was trying also with twitter android app and twitter PC app. I think the problem with that is that twitter fanpage as far as I see can’t be loaded at once.

Thanks in advance!

You should use the twitter API and an HTTP Request activity

You can get the first post and second post, you’ll get a element that will be increamenal

Take that element in a variable, pass it to selector of the Post

use a Do While loop

Have a element Exists and put that selector, if that is found then extract the element using same selector (Imp: same element Exists variable should be put in the do while Loop condition)

Have a condition that will send a Send Hotkey Of phdown after every 4 post extraction.

This will keep extracting the Post till the end , till the time that elements exists fails

Hope this helps!

Twitter is not replying for API request : /

I think that it solves the problem with scrolling down, but there is still a problem with selector. I wasn’t able to find selector which is suitable to scrap all the page. Maybe I’m not that advanced. I tried also using profile logo as an anchor

Hi

Did we try with Twitter connectors

Cheers @Michal_Podlasin

This fanpage has 1500~ posts. This function has “Get Tweets - Retrieves tweet details for last 100 tweet using the keyword provided.” option. I will try with this one but I don’t know if it works.

Thanks a lot !

1 Like

It’s limited to 100 : < any ideas how to solve my problem?

I hope we can change the count value in that property
@Michal_Podlasin

In tutorial for this package is said that 100 is maximum.

Is it editable @Michal_Podlasin

Do we need to have twitter API? I don’t know what to put in first 4 windows

image

In tutorial there is no information I think what to put in these fields. I tried to type as in tutorial without variables and it’s not working

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.