Using SHapley Additive exPlanations for value pruning in Bayesian Optimization

< Course:CPSC522

Using SHapley Additive exPlanations for hyperparameter value pruning in Bayesian Optimization

Here we test the hypothesis that using Interpretability of Machine Learning, Bayesian Optimization (BO) can quickly learn the expensive regions in the search-space.

Principal Author: Mayank Tiwary

Abstract

Big data systems like Apache Spark expose hundreds of knobs (like spark.executor.memory, spark.serializer) which need to be tuned for minimizing the execution time. The total search space which needs to be considered for tuning these knobs is extremely huge. In literatures AutoTuners have used BO based tuning setup for figuring out optimal values of the knobs. These tuners train their ML models for different workloads on a production scenario. Training on production becomes costly because the ML based tuners have to explore different regions in the search space. And one bad knob value can slow down the execution time by several factors when compared with the execution time achieved by setting the system with default knob values. For example, one bad knob value while executing KMeans workload on Spark increases the execution time by 17X. The tuning cost incurred on production can get amortized only when the tuner suggests good knob values which improves the execution time compared with the default execution time. The amortization of training cost becomes hard if a bad knob value slows down the total execution time by several minutes or hours. To minimize the training cost, the tuners need to effectively explore the search space. This work enables the tuner to effectively explore the search space by pruning the regions in the search space which cause a slower execution time.

Builds on

This work uses Bayesian Optimization as the core tuning engine for Apache Spark. It also uses Interpretability of Machine Learning (Explainable AI) to figure out and prune bad knob values which would lead to a very slow execution of the workload.

Related Pages

In this work, we use SHapley Additive exPlanations (Shapley values) for quantifying the ML-interpretability. This work is closely related to Tuneful (a Spark ML-tuner which uses BO) . The use of Shapley values was also been evaluated in an evaluation paper where the authors used SHapley Additive exPlanations (SHAP) to prune the entire knob instead of pruning the specific knob values while tuning knobs for databases instead of Apache Spark.

Content

Motivation and Background

Optimizing the behavior of Apache Spark applications is possible through a wide range of configuration options available, such as memory management, parallelism, and serialization. Commonly-tuned Spark configurations include spark.executor.memory, which allocates memory per executor process, increasing performance for memory-intensive applications, but also leading to out-of-memory errors if set too high. The spark.serializer configuration determines the serialization format for data transmission between Spark processes, with Java serialization as the default option, and other formats, such as Kryo, providing better performance for specific data types. Adjusting spark.sql.shuffle.partitions controls the number of partitions used for shuffling data in Spark SQL applications, improving performance for applications with high data shuffling requirements. spark.default.parallelism sets the default number of partitions for RDDs and dataframes, optimizing performance for applications with large datasets, albeit increasing the overhead of task coordination and scheduling. Lastly, spark.streaming.backpressure.enabled enables automatic rate control for Spark Streaming applications, with Spark dynamically adjusting the input rate based on processing rate to prevent system overloading.

Fine-tuning these and other Spark configurations can significantly impact Spark application performance, depending on the specific use case and workload. To streamline the process, there are available tools for automated configuration tuning, such as Tuneful . Overall, optimizing Apache Spark configurations is critical in enhancing the scalability and efficiency of Spark-based big data processing applications.

ML-based tuners (which use BO) are the widely adopted tuners because they can do a knowledge transfer of tuning experiences from one workload to other. However, the problem with these tuners are that they need huge set of observational data (training data). Generating the observational data involves changing the knob values and observing/measuring the execution time while executing a workload. It is important to note that for each workload the tuner needs to be seperately trained . It becomes challenging to collect training data specifically on production scenarios because playing around with knobs in production is often not permitted. This is mainly because if changing a knob configuration leads to a extreme slowdown of execution time when compared with the default execution time.

Limitations of tuning Spark knobs with BO based tuners

The biggest limitation of this class of tuners is that it needs to be trainied seperately for different workloads. This becomes more challenging on production scenario because its highly possible that changing a knobs value might degrade the performance. Authors in Tuneful propose an amortization model where the tuners training cost (cost incurred while gathering the training data on production) gets amortized after the training is completed only when the tuner starts recommending better knob values which leads to a better performance. However, this amortization gets more and more difficult if gathering training data takes a lot of time (this time delay is because the tuner picked a bad knob value which slowed down the execution by several factors).