Restarting a process

I have a process that downloads files from thousands of urls. Every few hundred urls it runs into an issue while trying to download a file. When this happens I manually close all open browsers and update the initialization of my counter variables (that are tracking what url I should be at) and kick off the process again.

What is the best way to automate this “try again where you left off” idea?

I was thinking a Try/Catch activity over the whole process and on Exception it closes all browsers and jumps back to the beginning of the process (and counters shouldn’t even need to be updated since the process has never actually stopped). But I’m not sure what this “jump back to previous step” activity is called. Thanks!

Hello,

you can use the Orchestrator queue for keeping track of URL list to be downloaded, if it fails also it have the option to retry, you can use the RE Framework so that it can be integrated easily.

Thanks,
Meg

Thanks. Orchestrator seems like a great thing for me to look into down the road. Is there a reasonable way for me to do it without Orchestrator?

Okay,
You can achieve this without Orchestrator by keeping a local tracker list,

step 1: put the list of urls into list/data-table
step 2: keep it in two parts, one to open applications(browser or others as per your process)/folders etc another to process/download from URL. now iterate one URL and at the end of the process check either it successfully downloaded or not, if downloaded remove URL from data-table/list
step 3: keep a flag at the end of step 2 process and check either any issue happens so that you can re-start the process, else continue to next URL

use try catch appropriately so that you can handle in single run.

1 Like

Sounds like you want a “Retry” approach.

You basically are right in saying that you can use a Try/Catch surrounding your process. If an error occurs, then increment a Retry count, and loop back to your process again. In order to retry from where you left off in processes where you loop through many items, you need a way to “remember” which item you were on before it threw an error, and there are a few ways to do this depending on how your loop it. (for example, ForEach loops, you need to store the item index and use with a Skip, so it reads like For each item in list.Skip(itemIndex)

You should also explore State Machines. There is a template in Studio that you can load to reference a retry approach, however, it is built to use the Queues which may or may not be ideal.

Essentially, the logic would be like this:

Kill Processes for clean environment
Load global settings
Start Process loop
    Try
        Invoke Process item
     Catch
        Log exception
        increment retryNumber
        Kill Processes
If retryNumber < MaxRetryNumber, Attempt Retry

Closure
   Kill Processes
   Log last exception if error occured

That’s just a basic example.

1 Like

@ClaytonM would state actually be an issue? If I’m just doing a try catch then isn’t the state of my variables at the time of error still preserved until the Catch finishes?

The problem I’m imagining is when it hits an error and goes to the Catch activities, how do I tell it to go back to the start of a flowchart?

And if that then runs into an error… do I need a try/catch inside a try/catch?

Yeah, your variables if they are in the outer scope (and in the same workflow) of the Try/Catch will still be set when it goes through the Catch. However, the problem arises when you process the items using a Loop that is inside the Try/Catch, because it exits the loop.

You can either store the index of the item in the loop, then continue where you left off by using a .Skip() in the loop - I can explain this better if you need. Or, you can store the process item in a sequence that is outside the Try/Catch. - Also, review the ReFramework Template that is in Studio when you go to the Start pane.

So it basically depends on how you are processing the items.
Essentially this logic would look like this:

 -- Get Process Item

 -- Try
    <perform actions on item>
  Catch
    Log exception
    Decrement item number
    Close Applications

 -- Loop back to Get next Process Item

I don’t particularly enjoy this method though, cause it relies on a counter.

Using a ForEach, would look like this:

 --  For each item in data
      -- Try
          <perform actions on item>
       Catch
          Log exception
          Decrement item number
          Close Applications

If you use a State Machine, though, this logic might need some streamlining. Also, keep in mind that you can place a Flowchart inside the For Each activity by deleting the inner sequence first then moving in the Flowchart.

I hope I answered your question.

Regards.

Thanks for the reply. You might be answering my question and I just don’t see it. I looked into the REFramework and state machines and agree that may be the way to go for me. I was essentially hoping the idea of a “transition” activity existed without “state” activities so I could just link back to an arbitrary activity on Error. I have counters (so many counters…) in place to track where things are at and where they should be picked up at so I don’t really have a problem there, it’s mainly getting things to kick off again where I’d want them to.

So yeah if you were answering that and it just went over my head haha some clarity would be appreciated but otherwise I think I’ll give state machines a shot.

The transitions in State Machines are essentially like the Switch decisions in a flowchart. It checks an expression, then go in the direction of the expression that was true. The “states” are basically just organizing the framework into parts, such as an Init, Process, and Closure.

So, first it enters the state, then at the end it checks from a list of expressions and enters the transition from top to bottom of which one is True first. Also, “transitions” are termed transitions because you can add activities that are to be performed during that pathway.

I can’t think of any examples to help explain this further, but hope it makes more sense. Anyway, “state” and “transition” are called that because they basically do exactly what those words mean. - Like you are in a “state” of confusion but are going to “transition” into a “state” of understanding :laughing:

Regards.

OK thanks Clayton. I played around with state machines a bit and think I’ve transitioned to that state of some understanding for their use :slight_smile:

It dawned on me that I could use the very simple solution of having an ERROR_CHECK boolean variable that gets set to True in the Catch part of my Try/Catch and then just have a simple if/else logic right after to see if ERROR_CHECK is True and if it is then point to a “Restart sequence” where I kill everything open and point back to the top of the Flow chart.

So probably not the best solution, or maybe what you were trying to tell me to do, but I think this should work!

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.