Distributed Processing Basics

What is Distributed Processing?

A distributed processing system consists of a cluster of several processing 'Nodes' that can communicate with each other and coordinate computing tasks. In SURE, Distributed Processing (DP) is based on a hierarchical structure where one ‘Master’ node communicates with ‘Processing’ nodes in different machines within the cluster and assigns them tasks. In this way, SURE Distributed Processing divides large projects into smaller sub-projects and distributes them to the processing machines available.

It is worth noting that one can also benefit from DP using a single machine that houses the Master node and a processing node (referred to as Local-DP).

Benefits of Distributed Processing

  • Process large projects in Local DP (single-machine)

  • Easily scale-up production by adding Processing nodes on demand

  • Inspect initial results and perform quality assessment within 1 or 2 days

  • Minimize progress loss in case of machine restart, power outage, or network disruption

  • Reduce required disk space

SURE Distributed Processing Workflow

First

The Master:

  • Performs the Analysis step

  • Divides the project into smaller sub-projects

  • Assigns and sends the sub-projects to Processing Nodes

Then

The Processing Nodes:

  • Receive sub-project from Master

  • Process sub-project and return results to Master

  • Wait to receive the next sub-project

Lastly

Once all sub-projects are completed:

  • the Master merges sub-projects and performs final steps (depending on results requested)