CUDA

Background

• cryptographic hash functions
• machine learning
• molecular dynamics simulations
• physics engines

Ontology

Programming abilities

• cuBLAS – CUDA Basic Linear Algebra Subroutines library
• CUDART – CUDA Runtime library
• cuFFT – CUDA Fast Fourier Transform library
• cuRAND – CUDA Random Number Generation library
• cuSOLVER – CUDA based collection of dense and sparse direct solvers
• cuSPARSE – CUDA Sparse Matrix library
• NPP – NVIDIA Performance Primitives library
• nvGRAPH – NVIDIA Graph Analytics library
• NVML – NVIDIA Management Library
• NVRTC – NVIDIA Runtime Compilation library for CUDA C++

• nView – NVIDIA nView Desktop Management Software
• NVWMI – NVIDIA Enterprise Management Toolkit
• GameWorks PhysX – is a multi-platform game physics engine

• CUTLASS 1.0 – custom linear algebra algorithms,
• NVIDIA Video Decoder was deprecated in CUDA 9.2; it is now available in NVIDIA Video Codec SDK

• nvJPEG – Hybrid JPEG processing

• CUB is new one of more supported C++ libraries
• MIG multi instance GPU support
• nvJPEG2000 – JPEG 2000 encoder and decoder

Advantages

• Scattered reads – code can read from arbitrary addresses in memory
• Unified virtual memory
• Unified memory
• Shared memory – CUDA exposes a fast shared memory region that can be shared among threads. This can be used as a user-managed cache, enabling higher bandwidth than is possible using texture lookups.
• Faster downloads and readbacks to and from the GPU
• Full support for integer and bitwise operations, including integer texture lookups

Limitations

• Whether for the host computer or the GPU device, all CUDA source code is now processed according to C++ syntax rules. This was not always the case. Earlier versions of CUDA were based on C syntax rules. As with the more general case of compiling C code with a C++ compiler, it is therefore possible that old C-style CUDA source code will either fail to compile or will not behave as originally intended.
• Interoperability with rendering languages such as OpenGL is one-way, with OpenGL having access to registered CUDA memory but CUDA not having access to OpenGL memory.
• Copying between host and device memory may incur a performance hit due to system bus bandwidth and latency.
• Threads should be running in groups of at least 32 for best performance, with total number of threads numbering in the thousands. Branches in the program code do not affect performance significantly, provided that each of 32 threads takes the same execution path; the SIMD execution model becomes a significant limitation for any inherently divergent task.
• No emulation or fallback functionality is available for modern revisions.
• Valid C++ may sometimes be flagged and prevent compilation due to the way the compiler approaches optimization for target GPU device limitations.
• C++ run-time type information and C++-style exception handling are only supported in host code, not in device code.
• In single-precision on first generation CUDA compute capability 1.x devices, denormal numbers are unsupported and are instead flushed to zero, and the precision of both the division and square root operations are slightly lower than IEEE 754-compliant single precision math. Devices that support compute capability 2.0 and above support denormal numbers, and the division and square root operations are IEEE 754 compliant by default. However, users can obtain the prior faster gaming-grade math of compute capability 1.x devices if desired by setting compiler flags to disable accurate divisions and accurate square roots, and enable flushing denormal numbers to zero.
• Unlike OpenCL, CUDA-enabled GPUs are only available from Nvidia as it is proprietary. Attempts to implement CUDA on other GPUs include:
• * Project Coriander: Converts CUDA C++11 source to OpenCL 1.2 C. A fork of CUDA-on-CL intended to run TensorFlow.
• * CU2CL: Convert CUDA 3.2 C++ to OpenCL C.
• * GPUOpen HIP: A thin abstraction layer on top of CUDA and ROCm intended for AMD and Nvidia GPUs. Has a conversion tool for importing CUDA C++ source. Supports CUDA 4.0 plus C++11 and float16.
• * ZLUDA is a drop-in replacement for CUDA on AMD GPUs and formerly Intel GPUs with near-native performance. The developer, Andrzej Janik, was separately contracted by both Intel and AMD to develop the software in 2021 and 2022, respectively. However, neither company decided to release it officially due to the lack of a business use case. AMD's contract included a clause that allowed Janik to release his code for AMD independently, allowing him to release the new version that only supports AMD GPUs.
• * ChipStar can compile and run CUDA/HIP programs on advanced OpenCL 3.0 or Level Zero platforms.
• * is a CUDA-compatible programming toolkit for ahead of time compilation of CUDA source code on AMD GPUs, aiming to expand support for other GPUs in the future.

Example

GPUs supported