CUDA Driver Timeout
CUDA is a computing platform supported by NVIDIA graphics cards. SURE’s most time-consuming processes, especially the Dense Cloud stage, have been implemented for CUDA support for increased performance.
To utilize CUDA's full capabilities, please make sure to keep your NVIDIA graphics card drives up-to-date.
Watch Dog Timer / Driver Timeout
The "Watch Dog Timer" is a tool that ensures the OS doesn’t lose contact with the GPU. Whenever the GPU doesn't report back to the OS within a specified time, the OS will restart the GPU driver automatically.
Thus, if CUDA is processing for the airborne images, it is very likely that one function’s run time exceeds the default Watch Dog Time (2 seconds). Then the OS will terminate CUDA and throw the "Time Out Exception", you may experience CUDA crash and black screen. In this case, CUDA cannot recover and all the following processing will switch to CPU.
To prevent this from happening, we recommend you to change the default Watch Dog Time to a larger value (e.g. 90 seconds). To do so, open the Run-Dialogue by pressing <Windows-Key>-R. Type in regedit to open the Registry Editor. Navigate through the tree structure to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers.
Right-click on GraphicsDrivers and add a new DWORD (32-bit) value and name it TdrDelay. Now a new key of type REG_DWORD is listed. Double click it, make sure the Decimal Option is checked and enter the new timeout delay (e.g. 90) in seconds. In order to make the changes take effect, please restart your computer.