Hyperparameters, the tuning knobs of machine learning algorithms, are instrumental for the generation of high-performing models. The tedious task of hyperparameter optimization (HPO) is nonetheless often reduced to manual optimization, humorously called ‘graduate student descent’, or unsophisticated grid search and random search [1], a situation often leading to results that are highly sensitive to hyperparameters [2-6].

In an attempt to address this, we developed Oríon, based on a different approach centered on two main ideas: 1) Hyperparameter optimization should be effortless to execute in common machine learning workflows 2) New hyperparameter optimization algorithms should be readily available for practitioners.

In this talk, we will present Oríon and it’s core design principles. Oríon is a simple but powerful hyperparameter optimization platform, providing all the tools necessary for an efficient hyperparameter optimization, from state-of-the-art algorithms, to key visualizations and a unique version control system for experiments. Its intuitive and flexible user interface, seamless and fast integration with any research code, as well as its distributed and asynchronous master-less approach, make Oríon an accessible and versatile tool for creating precise and organized work. Moreover, its modular design, open source code base, and benchmarking tools makes it an ideal framework for the development of new algorithms.

Thanks to its master-less design, Oríon can easily be integrated with any type of distribution system. We will present an example of integration with Dask and demonstrate its simplicity of use. With a few lines of codes, Oríons can be parallelized with Dask providing multi-node asynchronous hyperparameter optimization seamlessly.

With community development in mind, we designed Oríon to be modular and support external contributions as plug-ins. Supporting contributions is an important part of our tool, as one of our main goals is to support research in the area of hyperparameter optimization. The integration with Dask is one of the first steps to extend our modular support at the level of the distributed computations.

[1] Bouthillier, X., & Varoquaux, G. (2020). Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020 (Diss., Inria Saclay Ile de France).

[2] Bouthillier, X., Laurent, C., & Vincent, P. (2019, May). Unreproducible research is reproducible. In International Conference on Machine Learning (pp. 725-734). PMLR.

[3] Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019). Show your work: Improved reporting of experimental results. arXiv preprint arXiv:1909.03004.

[4] Kadlec, R., Bajgar, O., & Kleindienst, J. (2017). Knowledge base completion: Baselines strike back. arXiv preprint arXiv:1705.10744.

[5] Lucic, M., Kurach, K., Michalski, M., Gelly, S., & Bousquet, O. (2018). Are gans created equal? a large-scale study. In Advances in neural information processing systems (pp. 700-709).

[6] Melis, G., Dyer, C., & Blunsom, P. (2017). On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589.