Spark

Apache Spark is an analytics engine that can handle very large data sets. This guide reveals strategies to optimize its performance using PySpark.

Hackathons often inspire engineers to create amazing software. By blending various technologies together, really useful and often fun projects can be realized in a short period of time. In this article, Toptal engineer Radek Ostrowski shares his experience participating in the IBM Sparkathon, and walks us through how he elegantly combined the power of Apache Spark and Docker in IBM Bluemix to build a weather app.

In this post, Toptal engineer Radek Ostrowski introduces Apache Spark—fast, easy-to-use, and flexible big data processing. Billed as offering "lightning fast cluster computing", the Spark technology stack incorporates a comprehensive set of capabilities, including SparkSQL, Spark Streaming, MLlib (for machine learning), and GraphX. Spark may very well be the "child prodigy of big data," rapidly gaining a dominant position in the complex world of big data processing.