17 - Conclusions and Future Directions

Summary

In this book we described a new approach to machine learning based on densityratio estimation. This density-ratio approach offers a novel research paradigm in the field of machine learning and data mining from theory and algorithms to application.

In Part II, various methods for density-ratio estimation were described, including methods based on separate estimations of numerator and denominator densities (Chapter 2), moment matching between numerator and denominator samples (Chapter 3), probabilistic classifications of numerator and denominator samples (Chapter 4), density fitting between numerator and denominator densities (Chapter 5), and direct fitting of a density-ratio model to the true densityratio (Chapter 6). We also gave a unified framework of density-ratio estimation in Chapter 7, which accommodates the various methods described above and is substantially more general – as an example, a robust density-ratio estimator was derived. Finally, in Chapter 8, we described methods that combine density-ratio estimation with dimensionality reduction. Among various density-ratio estimators, the unconstrained least-squares importance fitting (uLSIF) method described in Chapter 6 would be most useful practically because of its high computational efficiency by an analytic-form solution, the availability of cross-validation for model selection, its wide applicability to various machine learning tasks (Part III), and its superior convergence and numerical properties (Part IV).

In Part III we covered the usage of density-ratio estimators in various machine learning tasks that were categorized into four groups. In Chapter 9 we described applications of density ratios to importance sampling tasks such as non-stationarity/domain adaptation and multi-task learning.