GPUs on the Edge: Big GPUs Are Down, NVIDIA Jetson Is Up

Originally designed for the rendering of 3D graphics, graphics processing units (GPUs) have become quite popular in compute-intensive industrial applications. Their ability to speed up computational processes has fueled demand in artificial intelligence and industrial robotics. GPU technology has advanced significantly in recent years and modern GPUs are faster and more powerful than ever before.

At the same time, the physical size of GPUs has become an interesting factor to consider. NVIDIA’s RTX 4090 GPU, for example, measures 12 inches (length) x 5.4 inches (width) compared to the 6.6 x 2.7 inch NVIDIA RTX A2000 or the 6.7 x 2.2 inch NVIDIA L4. Higher-end GPUs are getting larger over time, and it’s not just silicon. These big GPUs require more power, which leads to larger form factors. NVIDIA’s RTX 4080 offers power consumption of 320 W, while the L4 uses just 72 W. The power wattage increase means that these GPUs need larger heatsinks and fans to prevent overheating. 

While these GPUs pack more power than their smaller counterparts, smaller GPUs like the RTX A2000 and L4 still offer advanced graphics capabilities in smaller form factors. So CoastIPC customers looking to deploy GPUs into industrial edge applications often opt for smaller GPUs instead of the big ones. And some applications don’t have enough room or enough power for a larger card. As GPU technology and industrial application needs evolve, what does this mean for GPU deployment at the edge going forward? 

New Edge AI Options Emerge 

With larger, faster, and more power-hungry GPUs emerging, PC case size matters more than ever before. But if the cards continue getting larger, will cases need to continue growing? Will PCIe slots remain strong enough to handle the weight of these new GPUs? Another obvious consideration: Does the application at hand require a larger GPU or can a smaller card handle the job? While those in the gaming community may opt to adapt and improvise by deploying these GPUs into larger cases with clever cooling implementations, if trends persist, will gamers eventually have to design the PCs inside the GPUs? 

On the industrial side, in areas including AI (and its machine learning and deep learning subsets), robotics, video analytics, and in-vehicle applications, a paradigm shift is underway. Systems integrators, original equipment manufacturers (OEMs), and end users have shifted away from larger GPUs in favor of smaller GPUs in moderate-sized, fan-less, and rugged PCs like the RTX A2000 and L4. When it comes to the deployment of smaller GPUs at the edge, however, there are far fewer choices when it comes to physical size, memory, and price. GPUs based on the Mobile PCI Express Module (MXM) interconnect standard were introduced perhaps to address this need, but these devices have not really taken off. 

Newer processing devices could potentially address the size and space issues of modern GPUs. These devices including tensor processing units (TPUs), AI accelerator chips, and AI vision processors, which are paired with smaller GPUs for compute-intensive edge applications. These devices are still in early stages when it comes to ease of use and availability — with some experiencing significant supply delays. 

Rise of NVIDIA Jetson 

With these factors considered, many of our customers — and seemingly many in the industrial space — have shifted focus toward the increasingly popular NVIDIA Jetson platform for edge deployments where a moderately sized GPU may have previously been used. Designed specifically for edge AI and robotics, the NVIDIA Jetson platform is a compact high-performance system-on-module (SOM) that includes GPU, CPU, memory, power management, and high-speed interfaces.

Jetson offers a fast, low-power alternative to traditional computing platforms. It is offered in several models, listed below with required wattage and AI performance, which is measured in TOPS (tera [one trillion] operations per second), TFLOPS (tera [one trillion] floating point operations per second), or GFLOPS (giga [one billion] floating point operations per second).

All Jetson devices are supported by the same software stack, which is designed to accelerate AI applications and time-to-market by democratizing AI application development. The software stack includes NVIDIA JetPack SDK, which provides Jetson Linux, developer tools, and CUDA-X accelerated libraries, along with other NVIDIA technologies. The stack also includes Metropolis APIs and microservices for Jetson, enabling the development and deployment of applications on a Jetson device. Additionally, the stack includes NVIDIA TAO and pretrained AI models, the NVIDIA Triton Inference Server, NVIDIA Riva, NVIDIA DeepStream SDK, and NVIDIA Isaac. Learn more about the NVIDIA Jetson software stack. Industrial PCs based on Jetson technology offered by CoastIPC include MIC-711-OX with Jetson Orin NX, MIC-715 with Jetson Xavier NX, MIC-730 with Jetson Xavier AGX, and several others.

Edge Computing Solutions Going Forward

For many edge AI deployments, training and programming can be done in an office before deploying hardware into the field, which removes the need for devices with larger AI performance in terms of TOPS and TFLOPS. As a result, it seems that for the time being, many “lighter” edge AI deployments will be handled largely by Jetson products. Industrial PCs equipped with smaller GPUs or edge AI accelerator chips may gain further traction soon as well, and of course new and advancing technologies will always emerge. As for the future of larger GPUs in edge applications, these devices won’t be completely phased out, but it’s unclear what their role in the industrial space will be going forward. 

For more information about GPUs, including Jetson devices, please contact our product experts here or at 866-412-6278.