Machine Learning

On the equivalency between frequentist Ridge (and LASSO) regression and hierarchial Bayesian regression

Introduction In this post, we will explore frequentist and Bayesian analogues of regularized/penalized linear regression models (e.g., LASSO [L1 penalty], Ridge regression [L2 penalty]), which are an extention of traditional linear regression models of the form: [y = \beta_{0}+X\beta + \epsilon\tag{1}] where (\epsilon) is the error, which is normally distributed as: [\epsilon \sim \mathcal{N}(0, \sigma)\tag{2}] Unlike these traditional linear regression models, regularized linear regression models produce biased estimates for the (\beta) weights.