Apache Spark

A retailer needs sales totals from thousands of large files. Processing everything on one computer takes too long, and a machine failure can interrupt the calculation. The work needs to be divided across machines, with partial results combined and failed tasks retried.

Apache Spark is an open-source engine for distributed data processing. It splits data into partitions that workers process in parallel. For sales totals, each worker calculates partial totals, then Spark combines them by product. A coordinator called the driver plans the work; executor processes run the tasks. Grouping or joining data may require workers to exchange records across the network, called a shuffle. Spark can reuse intermediate data in memory and recompute lost partitions after a worker fails. Teams use Spark for batch transformations, streaming calculations, and machine learning. Azure Synapse and Azure Databricks provide managed Spark environments. Starting compute and moving data add overhead, so a small file is often simpler to process with an ordinary SQL query.