To increase throughput when processing massive projects, SURE software provides the capability to automatically split projects into a certain number of subprojects. These subprojects can be automatically distributed to worker nodes on the local network and are designed in a way that guarantees for consistent results across the various subproject's borders.
Follow these easy steps to utilize distributed processing capabilities for your projects:
Setup worker nodes
The SURE installation package contains an executable "SURE-node.exe" which is used to start a worker node. It can be used from command line or by double clicking on the SURE-node.exe. Once successfully running, a worker node will be on standby and waiting for a master node to assign a job (subproject) to it.
- If the "–ip_addr" parameter is omitted, SURE will try and detect the correct interface/IP on it's own. If the correct interface cannot be detected, please use this parameter to specify the IP address manually.
- All nodes (including master) must be within the same subnet (IPv4).
When working with the SURE GUI, a default user configuration "sure_ui.init" will be created and stored within the directory ".SURE", located in the current users home folder, e.g. "C:\Users\Dan\.SURE". This configuration file also stores the default workspace where your project directories will be stored in. In case you've used the SURE GUI before on the node, by default the distributed processing tasks will be stored within a directory called "SURE_Node_Workspace" within the default project directory. Otherwise, since the "SURE-node.exe" doesn't create the "sure_ui.init" file by itself you need to manually specify the workspace using the "–workspace" parameter.
On each worker node a SURE Aerial license needs to be activated.
If a firewall permission request pops up, we recommend to Allow Access and restart the SURE-node.exe executable to make sure the permissions are set correctly.
For both GUI or for CLI, set up your project just like in a conventional way as if distributed processing would not be used. Additionally tell SURE how many subprojects you'd like to split your project into. This mainly depends on the number of available worker nodes and the size of the project area. The smallest subproject will consist of 5 Tiles.
SURE will now run the analysis stage on the master node in order to gather some information required to split up the project. Once done and after splitting the project, you'll find all created subprojects within the project folders directory "SubProjects", each of which contains a complete independent SURE project which then will be distributed to an assigned node for processing.
- After starting the distributed processing project, you can switch to the Project Status tab to monitor the overall progress of your project. By selecting one of the tasks referring to one specific subproject, you can gain more information on this specific task, whether it's pending, running, or already finished.
- On the machine where you started the master process, you can connect to the cluster manager where you can obtain information on the cluster (e.g. nodes, IP's, running tasks). Thereto, open your favorite browser and navigate to "localhost:5006".
- All information regarding cluster management and scheduling will be written to "dp.log" within the project directory of the master project.
Once all subprojects have been completed, the results are moved to their corresponding location in the master project's basepath. The results of each subproject are self-sufficient and can already be inspected or pre-delivered while other subprojects are still processing. This allows to start with quality checks while other parts of a project are still being generated. Once all subprojects have been completed, the master instance will take care to generate a complete and consistent result by merging the various subprojects.
Raw Dense Clouds
Due to the massive storage requirements, the raw point clouds generated during the dense cloud stage of the software will not be copied back to the master node's subproject's folders.
Troubleshooting & known issues
- If a node or job fails (e.g. power down or other interference, the respective job will be redistributed to another SURE-node.
- In case the master fails, SURE will continue only with the unfinished subprojects.
- If a node doesn't terminate when pressing CTRL-C you may close the terminal, however, please check whether unwanted files remain in the nodes workspace.
Area-based splitting into subprojects
Distributed processing workflow