Mathematical Foundations of Machine Learning, Fall 2017

Mathematical Foundations of Machine Learning, Fall 2017


Justin Romberg
Office: Centergy 5227
Phone: (404) 894-3930

The purpose of this course is to provide first year PhD students in engineering and computing with a solid mathematical background for two of the pillars of modern data science: linear algebra and applied probability.

Download the syllabus


I. Vector space basics
a) linear vector spaces, linear independence
b) norms and inner products
c) bases and orthobases
d) examples: B­splines, cosines/Fourier, radial basis functions, etc.
e) linear approximation (closest point in a subspace, least­-squares I)

II. Linear Estimation
a) Examples: classical regression/recovering a function from point samples, imaging, etc.
b) the Singular Value Decomposition (SVD)
c) least-­squares solutions and the pseudo­inverse
d) stable inversion and regularization
e) kernels, Mercer’s theorem, RKHS, representer theorem
f) computing least-­squares solutions
i) matrix factorizations
ii) steepest descent and conjugate gradients
iii)low rank updates for online least-­squares

III. Statistical estimation and classification
a) Review: joint pdfs, random vectors, conditional probability, Bayes rule
b) multivariate Gaussian, Gaussian estimation
c) best linear unbiased estimator
d) maximum likelihood
e) Bayesian estimation
f) computing estimates: unconstrained optimization, stochastic gradient descent
g) classification (Bayes, nearest neighbor, linear, logistic)

IV. Modeling
a) geometric models
i) principal components analysis, low-rank approximation (Eckart-­Young theorem)
ii) sparsity, model selection
iii) structured matrix factorization (e.g. NNMF, dictionary learning)
iv) manifold models, nonlinear embeddings
b)  probabilistic models
i) hidden Markov models
ii) Gaussian graphical models
iii) message passing for inference