However, two common ways of solving LLS use similar lower/upper triangular factorization on derived systems. The version of LU factorization beaten to death in earlier notes was for general nonsymmetric n×n matrices. When A is symmetric, only half of the matrix needs to be stored, and the factorization is modified to give a lower triangular matrix L such that
where D is block diagonal with 1 x 1 and 2 x 2 blocks. Unsurprisingly this is called the LDL decomposition or LDL factorization.
In scientific and engineering computations, the most common special case for a symmetric matrix is when it is positive definite, defined as
for all nonzero n-vectors d. The reason this is common is because such linear systems arise in optimization. In first semester calculus, a function is minimized by taking its first derivative, setting it to zero, and solving for x. The Second Derivative Test says it is a minimum if the second derivative at that x is positive. For multidimensional calculus, the derivative is vector and the second derivative is a matrix. In this case the Second Derivative Test is exactly the definition of positive definite.
A symmetric positive definite (spd) matrix has a Cholesky factorization
where the diagonal entries of L are all positive values. Obviously (to me at least) the matrix L here is not the same as the matrix L in the LDLT factorization. However, the LDL factorization of a spd matrix will give a diagonal matrix D with all diagonal entries positive numbers. In that case, let S = sqrt(D) be the diagonal matrix with the positive square roots of the diagonal of D, then define E = L*S. Then A = EET is the Cholesky factorization of A.
Having a Cholesky factorization is also a necessary and sufficient condition for the real valued matrix A to be symmetric and positive definite. In fact, it seems the best way to numerically determine if a matrix is spd.
If the dT A d > 0 condition is relaxed to only requiring dT A d ≥ 0 for all nonzero d, the matrix A is positive semidefinite. In this case the LDL factorization exists, but some of the diagonal entries of D are equal to 0.
The easiest method is to solve the normal equations
Even though A is rectangular, AT A is square and for overdetermined LLS it is nonsingular. AT A is also symmetric (easy to prove) and positive definite (positive semidefinite is easy to prove for any matrix A, and positive definite for when A is full-rank).
The algorithm for solving LLS this way is
Solve (via LU factorization or its variants for symmetric matrices) the augmented system with block 2×2 coefficient matrix:
[ I A] [r] = [b] [AT 0] [x] = [0]
The augmented system is simply a rewriting of the normal equations together with the definition of the residual vector r = b − Ax. The block 2×2 coefficient matrix is symmetric, but guaranteed to not be positive definite. Still, the LDLT factorization can be used. Unfortunately the linear system is now of order (m+n)×(m+n), significantly larger. How much? You can answer that one:
Next: QR factorization via Givens rotations