Distributed Processing

To increase throughput when processing massive projects, SURE software provides the capability to automatically split projects into a certain number of subprojects. These subprojects can be automatically distributed to worker nodes on the local network and are designed in a way that guarantees for consistent results across the various subproject's borders.

Follow these easy steps to utilize distributed processing capabilities for your projects:

Setup worker nodes

The SURE installation package contains an executable "SURE-node.exe" which is used to start a worker node. It can be used from command line or by double clicking on the SURE-node.exe. Once successfully running, a worker node will be on standby and waiting for a master node to assign a job (subproject) to it.

CLI
usage: SURE-node.exe [-h] [-i IP_ADDR] [--workspace WORKSPACE]
optional arguments:
-h, --help show this help message and exit
-i IP_ADDR, --ip_addr IP_ADDR IP address of network interface to use (may be needed in case of multiple interfaces)
--workspace WORKSPACE workspace for the sure node

Network interfaces

  • If the "–ip_addr" parameter is omitted, SURE will try and detect the correct interface/IP on it's own. If the correct interface cannot be detected, please use this parameter to specify the IP address manually.
  • All nodes (including master) must be within the same subnet (IPv4).

Workspace

When working with the SURE GUI, a default user configuration "sure_ui.init" will be created and stored within the directory ".SURE", located in the current users home folder, e.g. "C:\Users\Dan\.SURE". This configuration file also stores the default workspace where your project directories will be stored in. In case you've used the SURE GUI before on the node, by default the distributed processing tasks will be stored within a directory called "SURE_Node_Workspace" within the default project directory. Otherwise, since the "SURE-node.exe" doesn't create the "sure_ui.init" file by itself you need to manually specify the workspace using the "–workspace" parameter.

Master node

The master node refers to the machine you're starting/controlling the distributed processing from. In case you'd like to also use this machine for the actual processing, you need to simply also setup a worker node (as explained above) here. You cannot actively start a master node using the "master.exe" shipped with the software. The software will take care of this for you once you start a distributed project.

License requirements

On each worker node a SURE Aerial license needs to be activated.

Firewall

If a firewall permission request pops up, we recommend to Allow Access and restart the  SURE-node.exe executable to make sure the permissions are set correctly.

Setup project

For both GUI or for CLI, set up your project just like in a conventional way as if distributed processing would not be used. Additionally tell SURE how many subprojects you'd like to split your project into. This mainly depends on the number of available worker nodes and the size of the project area. The smallest subproject will consist of 5 Tiles.


CLI
usage: SURE.exe
-sub_projects_count <approx_number_of_sub_areas> -> Approximated number of required partitions.
-distributed -> Use distributed processing
  • use the "-sub_area_count" parameter to specify the number of subprojects you want to split your project into
  • use the "-distributed" flag to start distributed processing once the project generation and splitting is done
GUI
  • During project set up or after project generation is done within the advanced configuration, set the number of subprojects you'd like to split your project into.
  • In the lower part of the processing control panel on the right-hand side, enable the "Activate Distributed Processing" checkbox. Note how the "Play" button changes its icon.


   


Start process

SURE will now run the analysis stage on the master node in order to gather some information required to split up the project. Once done and after splitting the project, you'll find all created subprojects within the project folders directory "SubProjects", each of which contains a complete independent SURE project which then will be distributed to an assigned node for processing.

Monitor progress

  • After starting the distributed processing project, you can switch to the Project Status tab to monitor the overall progress of your project. By selecting one of the tasks referring to one specific subproject, you can gain more information on this specific task, whether it's pending, running, or already finished.
  • On the machine where you started the master process, you can connect to the cluster manager where you can obtain information on the cluster (e.g. nodes, IP's,  running tasks). Thereto, open your favorite browser and navigate to "localhost:5006".
  • All information regarding cluster management and scheduling will be written to "dp.log" within the project directory of the master project.

Results

Once all subprojects have been completed, the results are moved to their corresponding location in the master project's basepath. The results of each subproject are self-sufficient and can already be inspected or pre-delivered while other subprojects are still processing. This allows to start with quality checks while other parts of a project are still being generated. Once all subprojects have been completed, the master instance will take care to generate a complete and consistent result by merging the various subprojects.

Raw Dense Clouds

Due to the massive storage requirements, the raw point clouds generated during the dense cloud stage of the software will not be copied back to the master node's subproject's folders.


Troubleshooting & known issues

  • If a node or job fails (e.g. power down or other interference, the respective job will be redistributed to another SURE-node.
  • In case the master fails, SURE will continue only with the unfinished subprojects.
  • If a node doesn't terminate when pressing CTRL-C you may close the terminal, however, please check whether unwanted files remain in the nodes workspace.



Area-based splitting into subprojects



Subproject tiling




Distributed processing workflow