TL;DR
This NeurIPS 2011 paper proposes Tree-structured Parzen Estimators (TPE) and a Gaussian-process SMBO algorithm that outperform random search and previously published human-tuned configurations on deep belief network benchmarks.
Summary
Hyperparameter tuning is a major bottleneck in training high-performing neural networks, typically left to human intuition or grid search. Two sequential model-based optimization (SMBO) algorithms are proposed — one using a Gaussian process and one using Tree-structured Parzen Estimators (TPE) — that model the conditional structure of hyperparameter spaces to identify irrelevant dimensions. Both outperform random search on deep belief network benchmarks and surpass previously published human-tuned configurations on challenging classification tasks.
Key contributions
- Proposes TPE, which models p(x|y) rather than p(y|x), enabling efficient conditional hyperparameter search.
- Introduces response surface modeling that identifies and skips irrelevant hyperparameter combinations.
- Demonstrates that SMBO methods outperform the best previously reported human-tuned results on DBN benchmarks.
- Establishes random search as a strong baseline that should be run before heavier optimization methods.
When to cite
- When introducing Bayesian hyperparameter optimization or citing the origin of the TPE algorithm.
- When arguing that conditional hyperparameter structure should be modeled rather than treating all dimensions as independent.
- When claiming that automated HPO can systematically outperform human-tuned configurations.
- When discussing the expected improvement criterion as a sample-efficient black-box optimization objective.