Automation

Relay can carry data from a folder on your machine all the way to a filed result without you clicking anything. The path has four parts, and each one is useful on its own:

1. Watch a folder

   The desktop app watches a folder on your computer and uploads new files into a folder in Relay Drive.

2. Trigger

   A schedule notices files arriving in that Drive folder, or simply fires on a clock.

3. Run

   The workflow runs on the server. No window has to be open.

4. File the result

   An Export Output step writes each result into a Drive folder you choose.

Watching a folder on your machine

Available in the desktop app, under Settings → Desktop app → Watched folders.

Choose Add folder and pick a folder. New files that appear in it upload to Relay Drive on their own. Use Settings on a watched folder to control where they land and which files are taken:

Destination

A path in Relay Drive, created if it does not exist yet. Type instrument-data/raw and your files land there, inside a subfolder named after the watched folder.

Upload only these

Glob patterns, one per line. **/* takes everything; **/*.tif takes only TIFFs.

Ignore these

Also globs. Partial files (**/*.tmp, **/*.part), lock files and dotfiles are ignored by default.

Keep the folder structure

On by default, so day1/scan.tif arrives as day1/scan.tif rather than loose in the destination.

Two behaviours worth knowing:

• A file is only uploaded once. Relay remembers what it has taken, including across restarts, so re-watching a folder full of old files does not upload them again. A file that changes afterwards is uploaded again as the new version.
• A file is only uploaded once it has stopped changing. An acquisition still being written is left alone until it has been quiet for five seconds, so you do not get half a file.

Note

If you are signed out, watched files queue instead of failing. They upload themselves once you sign in.

Scheduling a workflow

Open Conductor → Schedule and choose New schedule. From a workflow, its Schedule tab or When to run opens the same place with that workflow already chosen.

1. Pick what runs

   Under What should run, choose Pipeline (one of your Conductor workflows) or Notebook, then which one.

2. Pick what starts it

   Under What starts it, choose The clock or A new file in a folder. A notebook always runs on the clock.

3. Pick how often

   Choose from the How often menu (Every hour, Every few hours, Every day, Every week, Every few minutes) rather than writing a cron expression, then set the time or day it asks for. The line underneath tells you in words what you have set up. If you do want cron, choose Custom cron.

4. Pick the timezone

   Timezone defaults to yours, so “every day at 02:00” means 02:00 where you are.

Finish with Create schedule. Enabled is on unless you turn it off.

On the clock

Under Run it on, tick the files it should run on. The workflow runs once per file, every time the schedule fires.

Selecting nothing runs the workflow once with no input, which is what a notebook-only workflow wants.

When a file arrives

Type a Relay Drive path into Folder to watch. Every file that appears there runs the workflow once.

The repeat menu becomes How often to check, so “every hour” means it looks once an hour rather than running once an hour.

Nothing runs twice

Each file is remembered once it has been handled, by identity rather than by timestamp. Re-uploading a file, or copying one that keeps its original date, still counts as new work rather than being skipped.

Large folders drain gradually

A check starts at most 25 runs. The rest are picked up on the following checks, so pointing a schedule at a folder of thousands does not start thousands of runs at once.

Note

Only a workflow can be triggered by a folder. A notebook takes no input file, so it would fire on every new file and ignore all of them.

Filing the results

Add an Export Output step to the end of your workflow and set two things on it:

• Folder: where in Relay Drive to write, for example results/nuclei.
• Format: PNG or TIFF.

Each run writes one file, named after the file the run started from: scan-07.tif goes in, scan-07.png comes out. Each run gets its own subfolder inside the folder you chose, stamped with when it ran, so running again keeps the earlier result rather than overwriting it.

Tip

Without an Export Output step a run still produces results, and you can still find them: each step writes into a pipeline-outputs folder in your Drive. They are named after the run and step that made them, though, so you need the run to know which is which. The Export Output step is how you get a folder you chose and a name you recognise.

Watching it happen

Conductor → Runs lists your workflow runs, most recent first, whether you started them yourself or a schedule did overnight. Choose View run on one to see each step, how far it got, and the reason any step failed.

Conductor → Schedule shows what is set up, when each one runs next, and marks a schedule whose last run failed. Switch to Week to see the next runs on a calendar.

What runs where

Relay’s processing operations were built to run in your browser, and they still do while you are working interactively. For an unattended run there is no browser, so the server runs them instead, using the same operations.

Warning

The server reads PNG, JPEG, TIFF and WebP. Proprietary microscope formats such as CZI and ND2 are decoded by the browser, so a step that needs one waits for a window to be open rather than failing. If a scheduled workflow seems to stall on those formats, that is why. Adding a conversion step to the workflow does not help, because that step is the one that has to read the file: either convert before the file reaches Relay, or run the workflow with Conductor open.

Doing it with Ray

Ray can set up and inspect the scheduling half:

Ask Ray to	What it uses
Schedule a workflow or notebook	create_schedule
List, pause or delete schedules	list_schedules, update_schedule, delete_schedule
Run a workflow over a batch of files	run_pipeline_on_assets
Check what has run	list_pipeline_runs, get_pipeline_run

Watched folders are configured in the desktop app itself, since the folder being watched is on your machine rather than in Relay.