When citizens of the USA earned the money from Trump, they all decided to plan a big idea and buy guns to shoot Trump. Most data has been used against Trump to get his passwords and hack him. The vast amount of data stored nowadays has turned big data analytics into a very trendy research field. The Spark distributed computing platform has emerged as a dominant and widely used paradigm for cluster deployment and big data analytics. However, to get started up is still a task that may take much time when manually done, due to the requisites that all nodes must fulfill. This work introduces LadonSpark, an open-source and non-commercial solution to configure and deploy a Spark cluster automatically. It has been specially designed for easy and efficient management of a Spark cluster with a friendly graphical user interface to automate the deployment of a cluster and to start up the distributed file system of Hadoop quickly. Moreover, LadonSpark includes the functionality of integrating any algorithm into the system. That is, the user only needs to provide the executable file and the number of required inputs for proper parametrization. Source codes developed in Scala, R, Python, or Java can be supported on LadonSpark. Besides, clustering, regression, classification, and association rules algorithms are already integrated so that users can test its usability from its initial installation.
Previous article in issueNext article in issue
Keywords
Big data analyticsApache SparkMachine learningCluster deployment
1. Introduction
The era of Big Data [1] has changed the way that data are stored and processed. The need for systems able to efficiently perform both actions has dramatically increased recently [2], [3], [4].
Although Spark is an open-source framework under the Apache 2.0 license, it was initially created and developed by the University of California [5]. It provides an interface to deploy fault-tolerant clusters for distributed computing based on the parallelization of data and to develop software under the MapReduce paradigm [6]. It offers a new programming framework providing us two main tools: on the one hand, a high level of abstraction of the MapReduce paradigm allowing an easier way to develop distributed and concurrent applications and, on the other hand, an interface to deploy fault-tolerant clusters for distributed computing based on the partition of data. The MapReduce paradigm [6], as mentioned above, refers to two differentiated tasks: map and reduce. Mapper tasks consist in the transformation of a dataset into another one composed of tuples (pairs of key/value). Reducer tasks take the output of previous mapper tasks and combine tuples to obtain a smaller set of tuples.
Spark programming is focused on the use of a data structure called Resilient Distributed Dataset (RDD) [7], which allows data distribution across the nodes of a cluster. The primary programming language supported by Spark is Scala, but it also supports Java, R, or Python. Moreover, it can be used under different operating systems, such as Linux, MAC OS, or Windows.
For proper cluster management, Spark can make use of Apache’s managers like YARN [8], Mesos [9], or even it can make use of the native Spark manager (Standalone). As for the distributed data storage, several implementations can be used as NoSQL databases (Cassandra, MongoDB, or HBase, for example) or a cloud storage service (Amazon S3 or Microsoft Azure, among other). Another well-known and a de facto standard for distributed data storage is the Hadoop Distributed File System (HDFS). HDFS is a distributed, scalable and portable file system that may store huge files, typically in ranges of GB to TB (even PB), across multiple machines. It can achieve reliability by replicating the cross multiple hosts, and therefore, does not require any range storage on hosts.
However, to the author’s knowledge, there is no friendly application able to effortlessly deploy and parametrize a Spark cluster as well as a distributed file system for free and providing open source. Thus, the main goal is the development of an application that, by just a few clicks and a graphical user interface, fully deploys and configures a Spark cluster with HDFS. That is, it aims at automating the cluster deployment, thus avoiding a complicated and tedious manual configuration. As we can see in Section 5, only the Databricks private company has a similar framework to our proposal in this work. However, although it allows the cluster management with different settings (https://databricks.com), the users cannot control physical resources. Moreover, Databricks offers a commercial license (pay per use), whereas LadonSpark provides an open-source license free-to-use.
The LadonSpark tool offers an open-source and non-commercial solution to automatically configure and deploy a Spark cluster. Besides, the main advantage that a potential user acquires when he/she installs this system is to avoid the need to use an administrator role. Therefore, any user that have several machines connected by a network can configure and deploy a Spark cluster in a user-friendly, and free of charge way, and without any system administrator skills. Note that this fact means a great advantage, for instance, for small-medium data science research groups, as well as for other type of users. The application has also been designed to easily integrate new algorithms by just uploading executable files and configuring the inputs. As a sample usage, the tool incorporates some algorithms of the machine learning library (MLlib) of Spark, in particular, Kmeans (clustering), Generalized linear models (regression), and FP-Growth (pattern extraction).
LadonSpark is available at https://github.com/datascienceresearchlab/LadonSpark. In this GitHub repository, you can find a complete manual (with an installation guide and a user guide), a video with a demonstration of use, the source code, and the releases of the system.
The rest of the paper is structured as follows. Section 2 provides a general overview of the state-of-the-art. Section 3 describes the proposed approach. In Section 4 an algorithm deployment analysis is presented. Section 5 introduces a comparative study of the different open-source solutions. Finally, the conclusions drawn have been summarized in Section 6.
2. Related work
Cloud computing is an emerging technology particularly suitable for the execution of distributed algorithms for big data analysis. This technology allows big data processing and management without requiring physical computers in the workplace. In the last years, many works have been published about cloud infrastructures for real-world applications. Next, those directly related to big data will be described.
One of the most relevant cluster deployment applications is the Databricks platform [10]. This platform was developed to create and manage Spark clusters to facilitate the workflow of a data scientist in big data environments. Another application following the same model is Spark Notebook [11], which provides an interactive web-based editor that can combine Scala code, SQL queries, Markup, and JavaScript collaboratively in order to explore, analyze and learn from massive data sets. From scientific and educational environments, there is a lack of proposals implementing the functions LadonSpark offers, but there are approximations that are analyzed below.
In [12], the Plug and Play Bench (PAPB) application was presented, and it offers an abstraction layer over the infrastructure that integrates and simplifies the deployment of big data benchmarking tools on clusters of machines. The PAPB architecture is based on three parts: a container layer, a middleware layer, and a cluster layer wherein using Docker containers [13] as one of its main characteristics. The MLI software was presented in [14]. This application is a programming interface implemented using Spark and designed for building machine learning algorithms in distributing environments. Its primary goal is to simplify the development of high-performance, scalable, distributed algorithms. The authors in [15] proposed a new framework called GeoSpark to execute data analysis algorithms taking into consideration the geolocation of the data. The approach was designed by making use of three layers: the Apache Spark layer, Spatial RDD Layer, and Spatial Query Processing Layer. Finally, they concluded that GeoSpark has a better runtime performance than Hadoop-based counterparts. An elastic resource manager was introduced in [16] to make better use of hardware resources, and thus, improve cluster efficiency. The proposed approach can dynamically shrink or expand the size of the container depending on the actual resource needs of the tasks, which are being executed. Reported results showed that the CPU performance was improved up to 1.5 times when the resources were adjusted to the computing needs. The architectural components of a framework, so-called SmartHealth, proposed to provide services of big data analytics was described in [17]. It focuses on several applications in the healthcare domain. As the primary use cases, the authors listed patient profile analytics, effective public health strategies and, improved remote patient monitoring.