How to run same Data Scraping script on multiple pages with similar formats

Help
1

Really struggling on this one. For example, I created a data scraping script that takes a web page with tabular data and outputs to Excel. That’s fine.

But when I try to loop it to work on a similar page, it doesn’t output anything. The pages are pretty darn similar (two categories from the same main site). Do I need to create a scraping script for every single unique URL? I doubt it, as that doesn’t sound very automated at all. So I must be doing something wrong.

Am I missing something obvious?

The two sites in question are:
https://www.newyorker.com/goings-on-about-town/art
https://www.newyorker.com/goings-on-about-town/classical-music

Thank you

2

Hello there,

Due to dynamic selector and extractMetaData.

<html title='Classical Music, Concerts &amp; Opera Performances in New York City *' />
<html title='Art Events, Gallery Shows, Museum Exhibits in New York City - Th*' />

<extract><column attr='text' name='Column1' exact='1'>
<webctrl class='page art listing-grid' tag='div' idx='1' /><webctrl class='grid component ' tag='div' idx='1' />
<webctrl class='listing grid-item component  is-closing-soon ' tag='div' /></column></extract>

<extract><column exact='1' name='Column1' attr='text'>
<webctrl idx='1' tag='div' class='page classical-music listing-grid' /><webctrl idx='1' tag='div' class='grid component ' />
<webctrl tag='div' class='listing grid-item component is-happening-today  ' /></column></extract>

However you could do using single data scarping by making selector and ExtractMetaData as dynamic variable and pass it in data scraping property .

Works for me :slight_smile:
Here we go.
DatascrapingOnNewYorker.xaml (15.4 KB)