Episode
mlrHyperopt: Effortless and collaborative hyperparameter optimization experiments
with Jakob Richter
useR!2017: mlrHyperopt: Effortless and collaborativ...
Keywords: machine learning, hyperparameter optimization, tuning, classification, networked science
Webpages: https://jakob-r.github.io/mlrHyperopt/
Most machine learning tasks demand hyperparameter tuning to achieve a good performance. For example, Support Vector Machines with radial basis functions are very sensitive to the choice of both kernel width and soft margin penalty C. However, for a wide range of machine learning algorithms these "search spaces" are less known. Even worse, experts for the particular methods might have conflicting views. The popular package caret (Jed Wing et al. 2016) approaches this problem by providing two simple optimizers grid search and random search and individual search spaces for all implemented methods. To prevent training on misconfigured methods a grid search is performed by default. Unfortunately it is only documented which parameters will be tuned but the exact bounds have to be obtained from the source code. As a counterpart mlr (Bischl et al. 2016) offers more flexible parameter tuning methods such as an interface to mlrMBO (Bischl et al. 2017) for conducting Bayesian optimization. Unfortunately mlr lacks of default search spaces and thus parameter tuning becomes difficult. Here mlrHyperopt steps in to make hyperparameter optimization as easy as in caret. As a matter of fact, for a developer of a machine learning package, it is unquestionable impossible to be an expert of all implemented methods and provide perfect search spaces. Hence mlrHyperopt aims at:
- improving the search spaces of caret with simple tricks.
- letting the users submit and download improved search spaces to a database.
- providing advanced tuning methods interfacing mlr and mlrMBO.
A study on selected data sets and numerous popular machine learning methods compares the performance of the grid and random search implemented in caret to the performance of mlrHyperopt for different budgets.
References Bischl, Bernd, Michel Lang, Lars Kotthoff, Julia Schiffner, Jakob Richter, Erich Studerus, Giuseppe Casalicchio, and Zachary M. Jones. 2016. "Mlr: Machine Learning in R." Journal of Machine Learning Research 17 (170): 1–5. https://CRAN.R-project.org/package=mlr.
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. 2017. "mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions." arXiv:1703.03373 [Stat], March. http://arxiv.org/abs/1703.03373.
useR!2017: mlrHyperopt: Effortless and collaborativ...
Keywords: machine learning, hyperparameter optimization, tuning, classification, networked science
Webpages: https://jakob-r.github.io/mlrHyperopt/
Most machine learning tasks demand hyperparameter tuning to achieve a good performance. For example, Support Vector Machines with radial basis functions are very sensitive to the choice of both kernel width and soft margin penalty C. However, for a wide range of machine learning algorithms these "search spaces" are less known. Even worse, experts for the particular methods might have conflicting views. The popular package caret (Jed Wing et al. 2016) approaches this problem by providing two simple optimizers grid search and random search and individual search spaces for all implemented methods. To prevent training on misconfigured methods a grid search is performed by default. Unfortunately it is only documented which parameters will be tuned but the exact bounds have to be obtained from the source code. As a counterpart mlr (Bischl et al. 2016) offers more flexible parameter tuning methods such as an interface to mlrMBO (Bischl et al. 2017) for conducting Bayesian optimization. Unfortunately mlr lacks of default search spaces and thus parameter tuning becomes difficult. Here mlrHyperopt steps in to make hyperparameter optimization as easy as in caret. As a matter of fact, for a developer of a machine learning package, it is unquestionable impossible to be an expert of all implemented methods and provide perfect search spaces. Hence mlrHyperopt aims at:
- improving the search spaces of caret with simple tricks.
- letting the users submit and download improved search spaces to a database.
- providing advanced tuning methods interfacing mlr and mlrMBO.
A study on selected data sets and numerous popular machine learning methods compares the performance of the grid and random search implemented in caret to the performance of mlrHyperopt for different budgets.
References Bischl, Bernd, Michel Lang, Lars Kotthoff, Julia Schiffner, Jakob Richter, Erich Studerus, Giuseppe Casalicchio, and Zachary M. Jones. 2016. "Mlr: Machine Learning in R." Journal of Machine Learning Research 17 (170): 1–5. https://CRAN.R-project.org/package=mlr.
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. 2017. "mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions." arXiv:1703.03373 [Stat], March. http://arxiv.org/abs/1703.03373.
Have feedback? Submit an issue here.