If you’re looking for a high-level introduction about Spark on Kubernetes, check out The Pros And Cons of Running Spark on Kubernetes, and if you’re looking for a deeper technical dive, then read our guide Setting Up, Managing & Monitoring Spark on Kubernetes. Data Mechanics is a managed Spark platform…

Tutorial: How to build the right Docker image, start your Spark session, and run at scale! — R is a programming language for statistical computing. It is widely used among statisticians and data scientists. Running applications on a single machine has been sufficient for a long time, but it has become a limiting factor when more data and advanced analysis is required. That’s why the R community…

Achieve up to 65% performance gain using the latest S3 magic committer from Spark 3.2 and Hadoop 3.3! — Most Apache Spark users overlook the choice of an S3 committer (a protocol used by Spark when writing output results to S3), because it is quite complex and documentation about it is scarce. This choice has a major impact on performance whenever Spark writes data to S3. Since for AWS…

Main Features explained: PersistentVolumeClaim Reuse (k8s-specific), Koalas, Hadoop 3.3, RocksDB, and more! — Apache Spark 3.2 was released in October 2021(see release notes) and it is now available for Data Mechanics customers, and for anyone who wishes to run Spark on Kubernetes (or simply Spark on Docker) as we updated our DockerHub repository of optimized Docker images for Spark. In this article we’ll…

In this article we’re going to show you how to start running PySpark applications inside of Docker containers, by going through a step-by-step tutorial with code examples (see github repo). There are multiple motivations for running Spark application inside of Docker container (we covered them in an earlier article Spark…

Get started and do your work with all the common data sources supported by Spark. — Our optimized Docker images for Apache Spark are now freely available on our DockerHub repository, whether you’re a Data Mechanics customer or not. This is the result of a lot of work from our engineering team: We built a fleet of Docker Images combining various versions of Spark, Python, Scala…

In this article, the co-founder of Lingk tells the story of their migration from EMR to the Spark-on-Kubernetes platform managed by Data Mechanics: their goals, the architecture of the solution & challenges they had to address, and the results they obtained. Goals of this migration Lingk.io is a data loading, data pipelines, and integration…

With the Apache Spark 3.1 release in March 2021, the Spark on Kubernetes project is now officially declared as production-ready and Generally Available. This is the achievement of 3 years of fast-growing community contributions and adoptions of the project — since initial support for Spark-on-Kubernetes was added in Spark 2.3…

Recent developments with Spark 3.0, Spark-on-Kubernetes going GA, PySpark usability improvements, and more. — November 20th, 2020: I just attended the first edition of the Data + AI Summit — the new name of the Spark Summit conference organized twice a year by Databricks. This was the European edition, meaning the talks took place at a European-friendly time zone. …