Cloudera adds NVIDIA GPU acceleration to Apache Spark workloads
Cloudera integrates NVIDIA GPU acceleration into Apache Spark 4.1 to reduce enterprise compute costs and speed up data processing across hybrid architectures.
Cloudera is introducing native NVIDIA GPU acceleration to Apache Spark 4.1 within Cloudera Data Engineering, enabling organisations to boost the performance of existing Spark workloads without modifying their PySpark or SQL code. Driven by NVIDIA’s cuDF library, this integration focuses on the intensive processing and preparation of vast datasets prior to downstream analytics and AI applications. The development allows data teams to speed up foundational pipeline tasks while keeping existing development routines intact.
Table Of Content
According to Cloudera, the implementation can deliver up to four times faster workload performance on NVIDIA GPUs compared with traditional CPU infrastructure. These reduced processing durations are also intended to lower cloud infrastructure expenditure by significantly cutting compute runtime during ETL stages, data staging and related Spark jobs.
GPU acceleration works with existing Spark applications
Built directly into Cloudera Data Engineering, the NVIDIA cuDF plug-in accelerates supported tasks while preserving original PySpark and SQL code. This native integration eliminates manual GPU driver configuration and substantially eases the transition of compatible pipelines onto high-performance hardware.
The acceleration runs natively inside the existing management environment for Spark pipelines. Through Cloudera’s Unified Data Fabric, enterprises maintain established security, compliance and governance policies across all active workloads. As a direct result, accelerated jobs remain fully aligned with existing operational procedures without requiring separate oversight workflows.
The deployment architecture extends across public cloud, private cloud, sovereign cloud and on-premises systems, reaching into edge environments as well. This expansive reach provides hybrid enterprises with a unified method to run GPU-accelerated Spark tasks across diverse physical locations under a single control layer.
“The fastest path to accelerating AI deployments is the one that aligns with how enterprises already operate today,” said Pat Lee, vice president, Strategic Enterprise Partnerships at NVIDIA. “With NVIDIA AI infrastructure and CUDA-X libraries now native to Cloudera Data Engineering, enterprises can lower costs and dramatically speed up Apache Spark pipelines without changing a single line of PySpark or SQL code, turning business data into a foundation for AI.”
Faster Spark jobs could reduce compute use
Complex Spark jobs frequently run for hours, consuming extensive compute resources and delaying the availability of prepared data for analytical models. By introducing GPU acceleration directly into these early stages, Cloudera aims to shorten turnaround times and curb the mounting consumption of compute capacity.
To highlight this operational challenge, Cloudera pointed to findings from its The Great Re-Architecture Survey, where 84% of respondents indicated that AI workloads had driven up their infrastructure expenses. Faster ETL processes and swifter data staging directly address these budget pressures by reducing the active compute hours needed to convert raw datasets into analytics-ready assets. The performance gains are therefore tightly connected to enterprise cost management, enabling businesses to process heavy data volumes without incurring escalating infrastructure fees.
“For many organisations, AI isn’t limited by models. It’s limited by how quickly they can turn raw data into trusted, usable insights,” said Leo Brunnick, Chief Product Officer at Cloudera. “Accelerating Spark inside Cloudera Data Engineering helps remove that bottleneck, allowing customers to move from data preparation to analytics and AI faster while keeping governance, security, and operational consistency at the centre of their strategy.”
Actual performance gains will ultimately vary based on specific workload characteristics and the underlying hardware environment. Nonetheless, Cloudera’s architecture ensures that qualifying Spark applications can harness NVIDIA GPU processing without necessitating any revisions to PySpark scripts or SQL queries.
Integration forms part of Cloudera Anywhere Cloud
GPU acceleration for Apache Spark will be made available within Cloudera Data Engineering as an integral component of Cloudera Anywhere Cloud. The new capability was officially unveiled during the EVOLVE Singapore conference on 20 August.
Cloudera Anywhere Cloud delivers a unified operational framework across public cloud, private cloud, sovereign cloud, on-premises and edge environments. Incorporating NVIDIA acceleration into this ecosystem brings higher data throughput to Spark pipelines while upholding the strict governance and security standards established across the wider Cloudera platform. This balance ensures that performance gains do not compromise enterprise compliance.
Looking ahead, Cloudera intends to showcase the integration in upcoming technical sessions and live demonstrations. These presentations are scheduled to take place at NVIDIA GTC Berlin and Cloudera EVOLVE New York later in 2026.
