machine-learning-mitchell

Machine Learning (Tom M. Mitchell)

Apply this skill to ground ML design, evaluation, and algorithm-selection questions in the classical framework — useful for foundational reasoning even when the actual implementation uses modern deep learning libraries, since the underlying tradeoffs (bias/variance, evaluation validity, inductive bias) haven't changed.

Core concepts

The well-posed learning problem (T, P, E) — Any learning problem should be stated precisely as: the Task (what the system should get better at), the Performance measure (how "better" is quantified), and the Experience (what data/interaction it learns from). Vague ML proposals ("build a model to predict churn") usually resolve once forced through this triple — especially P, which is often left implicit and later turns out to be the wrong thing to optimize.

Hypothesis space & inductive bias — A learner can only find hypotheses within the space its algorithm/representation can express (e.g. decision trees express axis-aligned splits; linear models express linear boundaries). Every learner has an inductive bias — the assumptions it uses to generalize beyond the training data (e.g. "prefer shorter trees," "prefer maximum-margin boundaries"). When a model generalizes badly, ask first whether the hypothesis space can even represent the true pattern, and second whether its bias matches the domain.

Overfitting & the bias-variance tradeoff — A hypothesis overfits if it fits training data better than the true underlying distribution, usually from a hypothesis space too expressive relative to the data available. High-bias (too simple) models underfit; high-variance (too flexible) models overfit. Diagnosing a performance problem should identify which side of this tradeoff it's on before reaching for "more data" or "a bigger model" as a default fix.

Proper evaluation methodology — Never evaluate on training data. Use held-out test sets or k-fold cross-validation to estimate true error, and treat that estimate as having a confidence interval, not a single fixed number — small test sets or single train/test splits produce noisy comparisons. When comparing two algorithms/models, check whether the observed difference is actually statistically significant given sample size, not just "which number is higher."