A batch of examples is a matrix. Matrix–vector multiplication applies one weight vector to every row, and least squares is the geometric statement that the residual is orthogonal to the feature columns.
References and further study
Optional further reading only — Wikipedia, OpenStax, author sites, and MIT OCW course pages. Prefer open encyclopedia / CC BY texts first. These links are not MIT endorsement, and MatheLinux lessons are not hosted copies of OCW.
Copyright rule: MatheLinux stores only the citation. It does not copy or host external books, papers, videos, or datasets. MIT OCW is CC BY-NC-SA (non-commercial); open it on ocw.mit.edu. Publisher books require a legal copy. Quote sparingly and cite.
Submit work
Before you continue
Lesson close · Math for Machine Learning
Use this checkpoint to recall the lesson before marking it complete. The concepts below come from this lesson's authored sections.
Stuck? Send a blocker
One sentence on what stopped you (definition, derivation, lab control, time). This feeds Module-1 product metrics — not a public comment thread.
Progress
Sequence
Continue to Probability for Learning, the next prerequisite step in this course.
Least squares chooses
Hypotheses. If has full column rank, is invertible and the minimizer is unique:
An intercept is a column of ones inside , not a special case of the algebra.
Expand . Differentiate in and set the gradient to zero:
Those are the normal equations. Geometrically, is the orthogonal projection of onto the column space of . The residual is orthogonal to every column, which is the same display .
If two columns are identical, is singular and many share the same . A pseudoinverse returns one solution; a ridge term replaces by and restores uniqueness. Centering and scaling columns does not change the column space when you keep an intercept, but it changes the conditioning of and the meaning of a penalty on .
A cross-sectional regression of next-period returns on a matrix of standardized characteristics is this least-squares problem. Collinear factors (two versions of the same momentum) make singular; ridge or dropping a column is the modeling response.
Interactive · design matrix shape
X shape n×p = 5×3 · params grow with p
Optional further reading (open references — MatheLinux does not host third-party course files):
Submit work
Optional human feedback—not required to mark the lesson complete. Pick a brief, submit it, or continue without submitting.
Human-graded work is a Pro course feature
The free lesson, quiz, and problem checks remain available. Pro members can choose a brief below, submit work, and receive a score and written feedback.
Previous submissions
No submitted work for this lesson yet.
