Parallel for each with REframework

Help Studio
,
1

Hello,
I am using REFramework for web scraping. It might be faster if using in parallel so I was trying out changing the GetTransactionData with using Get queue Items, which then get read into IEnumerable. Then I use this array so that everything within Process is run inside ParallelForEach.
This seems to work up until inside SetTransactionStatus where it is trying to change the queue item status in SetTransactionStatus there it always times out. Parallel is not waiting on setTransactionStatus and just goes straight to the next item, is that the problem? I have tried increasing the time out to 20 second and it still times out. Is this something that should work or are there limitations of doing it like this?

2

if you have 200 queueitems, and your set transactionstatus is within the parallel for each, there is a chance you send 200 orchestrator requests within a short time. This might cause some traffic jams server side maybe. (or at least from a specific volume.)

But if you see that the status updates are done correctly in the orchestrator you could take the risk.

Personally I’m more for quality over quantity and go for a classic serial approach.

3

I tried out moving the ParallelForEach just around the Process part. It is the web scraping I would like making faster, it is using Chrome webdriver.
Process scrapes data from a very simple site. It is kind of slow. Took about 15 hours to go over 5000 items. I changed it so now it uses Chrome webdriver but I thought parallel might help as well.

I am getting some other errors, this sounds like some kind of race condition:

image
image1529×541 33.2 KB

Is REframework not good when you want to use parallel?