Vowpal Wabbit
Vowpal Wabbit is an open-source fast online interactive machine learning system library and program developed originally at Yahoo! Research, and currently at Microsoft Research. It was started and is led by John Langford. Vowpal Wabbit's interactive learning support is particularly notable including Contextual Bandits, Active Learning, and forms of guided Reinforcement Learning. Vowpal Wabbit provides an efficient scalable out-of-core implementation with support for a number of machine learning reductions, importance weighting, and a selection of different loss functions and optimization algorithms.
Notable features
The VW program supports:- Multiple supervised learning problems:
- * Classification
- * Regression
- * Active learning for both regression and classification
- Multiple learning algorithms
- * OLS regression
- * Matrix factorization
- * Single layer neural net
- * Searn
- * Latent Dirichlet Allocation
- * Stagewise polynomial approximation
- * Recommend top-K out of N
- * One-against-all and cost-sensitive OAA reduction for multi-class
- * Weighted all pairs
- * Contextual-bandit
- Multiple loss functions:
- * squared error
- * quantile
- * hinge
- * logistic
- * poisson
- Multiple optimization algorithms
- * Stochastic gradient descent
- * BFGS
- * Conjugate gradient
- Regularization
- Flexible input - input features may be:
- * Binary
- * Numerical
- * Categorical
- * Can deal with missing values/sparse-features
- Other features
- * On the fly generation of feature interactions
- * On the fly generation of N-grams with optional skips
- * Automatic test-set holdout and early termination on multiple passes
- * bootstrapping
- * User settable online learning progress report + auditing of the model
- * Hyperparameter optimization
Scalability
Vowpal wabbit has been used to learn a tera-feature data-set on 1000 nodes in one hour. Its scalability is aided by several factors:- Out-of-core online learning: no need to load all data into memory
- The hashing trick: feature identities are converted to a weight index via a hash
- Exploiting multi-core CPUs: parsing of input and learning are done in separate threads.
- Compiled C++ code