> For the complete documentation index, see [llms.txt](https://ai.younglimit.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai.younglimit.com/machine-learning/general-linear-model-1.md).
# General Linear Model 1
Here are the notes for general linear regression.
## General Linear Model 1——Linear Regression
### 1. What is Linear Regression?
1. It is one of the most well known/understood algorithm in statistics and machine learning
2. Linear regression is a **linear model**, which a linear relationship between the input $$X$$ and output $$Y$$. More technical, we can consider $$Y$$ can be a **linear combination** of $$X$$
Quote: 我们的目的就是找到一条直线，使所有我们之前input的点到这条直线的距离最小。
3. The representation of simpliest linear regression can be written as
$$
Y= \beta\_0+\beta\_1X+\epsilon
$$
4. In statistics, this belongs to the parametric model, i.e. it has the parameter $$\beta\_0$$ and $$\beta\_1$$
(这里当然最好是$$Y= \beta\_0+\beta\_1X$$,，但是没办法我们有$\epsilon$,（但是我们知道它是normal）)
so, what is our target?
为了找 $$X$$和 $$Y$$的关系——Find the best $$\beta\_0$$ and $$\beta\_1$$ ——find total error 最小（loss function）——find Least square
### 2. The keys
#### 2.1 How to determine this model?(估计参数)
**2.1.1 Loss function？（we can link the more general case for the loss function）**
1. 如何我们要知道我们好不好呢？就需要看看error
$$Error=\vert y\_i-y\_i^{'}\vert$$
2. error 可以变化吗？可以, 但是为了好计算所以用squre
3. loss function=total error
**2.1.2.最小二乘法least square推导loss function**
$$
Loss=\sum\_{i=1}^{n} error^2=\sum\_{i=1}^{n}(y\_i-\beta\_0+\beta\_1X\_i)^2
$$
1. 这里不一定要平方，但是平方和可以找到某种意义上的最好值
1. 总的误差的平方最小的y就是真值，这个假设在误差是在随机波动下是最优的
2. 所以我们可以来找 $$Loss =\min\_{\beta\_0,\beta\_1} \sum\_{i=1}^{n}(y\_i-\beta\_0+\beta\_1X\_i)^2$$
2. 因着是找 $$\beta\_0$$ and $$\beta\_1$$符合 $$\arg \min\_{\beta\_0,\beta\_1} \sum\_{i=1}^{n}(y\_i-\beta\_0+\beta\_1X\_i)^2$$
3. 所以我是在找$$\beta\_0$$,$$\beta\_1$$使得$$\sum\_{i=1}^{n}(y\_i-\beta\_0+\beta\_1X\_i)^2$$最小，也就是loss的最小
4. 最小二乘法不永远是最优
**一些提醒：**
1. least square（最小二乘法）是从cost function的角度，利用**距离**的定义建立目标函数；(注意最小二乘法是方法整个loss体系的建立可以link到statsitcal learning theory)
2. 经典的参数估计方法是从**概率的角度**建立目标，比如说最大似然估计MLE（maximum likelihood estimation）
**2.1.3 最大似然估计MLE（maximum likelihood estimation）推导loss function**
1. mle是什么？
1. mle is a method of estimating the parameters of a statistical model given obersvation, by finding the parameter values that maximize the likelihood of making the observation given the parameters. 用参数估计的方法，在有了一定的观测值之后，来找parameter，让我们可以最有可能看到我们观测值，让我们可以最大程度放大我们的观测值
2. 如果对于linear regression 来说，相当与用一个方法，**去找穿过最大可能性（最大密度）(尽可能多的概率)的那些点的线上**
1. 同时这条线对于x来说是最大可能性分布所在的线（CLT）
2. 见图
1. 别忘了我们要使用model里的assumption： p(y|x) 是 $$mean=\mu=f(x)$$(和$$x$$有关), $$variance=\sigma^2$$（和$$x$$无关）的normal distribution（or we consider $$\varepsilon \sim N(0,\sigma^2)$$）
2. 推导\*\*(面试必考题)\*\*
1. we know that $$Y\vert X \sim N(\bar{\beta}\_0+\bar{\beta}\_1X, \sigma^2)$$
2. $$
p(Y\_i\vert X\_i)=\frac{1}{\sigma(2\pi)}e^{-\frac{1}{2\sigma^2}(Y\_i-\bar{\beta}\_0-\bar{\beta}\_1X\_i)^2}
$$
3. $$
L(\bar{\beta}\_0,\bar{\beta}*1,\sigma^2)=p(Y\_1,\cdots,Y\_n\vert X\_1,\cdots,X\_n)=\frac{1}{\sigma^{n}(2\pi)^{n/2}}e^{-\frac{1}{2\sigma^2}\sum*{i=1}^n(Y\_i-\bar{\beta}\_0-\bar{\beta}\_1X\_i)^2}
$$
under the assumption $$(X\_1,Y\_1),\cdots, (X\_n,Y\_n)$$ are independent
where $$L(\bar{\beta}\_0,\bar{\beta}\_1,\sigma^2)=p(Y\_1,\cdots,Y\_n\vert X\_1,\cdots,X\_n)=p(Y\_1\vert X\_1,\cdots,X\_n)\cdots p(Y\_n\vert X\_1,\cdots,X\_n) =p(Y\_1 \vert X\_1)p(Y\_2\vert X\_2)\cdots p(Y\_n\vert X\_n)$$$$L(\bar{\beta}\_0,\bar{\beta}\_1,\sigma^2)=p(Y\_1,\cdots,Y\_n\vert X\_1,\cdots,X\_n)=p(Y\_1\vert X\_1,\cdots,X\_n)\cdots p(Y\_n\vert X\_1,\cdots,X\_n) =p(Y\_1 \vert X\_1)p(Y\_2\vert X\_2)\cdots p(Y\_n\vert X\_n)$$
4. the corresponding the log function:(我只关心parameter，log函数不影响单调等数学性质)
$$
\log L(\bar{\beta}\_0,\bar{\beta}*1,\sigma^2)=-n\log(\sqrt{2\pi}\sigma)-\frac{1}{2\sigma^2}\sum*{i=1}^n(Y\_i-\bar{\beta}\_0-\bar{\beta}\_1X\_i)^2
$$
所以我们要找的就是
$$
\arg \max -\sum\_{i=1}^n(Y\_i-\bar{\beta}\_0-\bar{\beta}\_1X\_i)^2
$$
i.e.
$$
\arg \min \sum\_{i=1}^n(Y\_i-\bar{\beta}\_0-\bar{\beta}\_1X\_i)^2
$$
**两者关系？**
1. 当他们在linear regression下的assumption下，这两个方法得到结果是相通的
2. one is from statistics, and the other one is from optimization
**一点提醒：**
1. noise是数据造成的，是inherent bias. error是模型造成的，是人为的。是两个不同的概念
#### 2.2 How is the performance of this model?（这个是来看模型自己的好坏，评价自己的参数）
我们只能系统的保证其不会偏差（$E(Y)=\mu$)
Consider 这个问题，我们需要link到统计上的假设检验问题
null hypothesis : $$\beta\_1=0$$
1. 目的是从统计上来判断这组数据和population相差多少，assessing the accuracy of the coefficient estimation,可以使用 $$p$$-value 或者是 confidence interval $$\hat{\beta}\_1 \pm 2SE(\hat{\beta}\_1)$$
2. 选择的统计量 $$t=\frac{\hat{\beta}\_1-0}{SE(\hat{\beta}\_1)}$$ where $$t$$ distribution with $$n-2$$ degrees of freedom assuming $$\beta\_1=0$$
#### 2.3 How can we compare this model with others models?
(这个相当于模型外的判断模型的好坏 i.e. the extent to which the model fits the data)
1. assessing the overall accuracy
2. $$RSE=\sqrt{\frac{1}{n-2}RSS}=\sqrt{\frac{1}{n-2}\sum\_i^{n}(y\_i-\hat{y}\_i)^2}$$
3. $$R^2=\frac{TSS-RRR}{TSS}=1-\frac{RSS}{\sum\_i^{n}(y\_i-\hat{y}\_i)^2}$$,
4. where $$TSS$$ is total sum of squares, $RSS$ is the residual sum of squares(对误差的多少)
5. 当是simple regression时，他相同于correlation
6. 这里相当于 proportion of variability in $$Y$$ that can be explained using $$X$$ ($$Y$$ 的变化中能够被$X$解释的部分的比例 )
#### 2.4 GLM extra study with assumption？
## Multiple Linear Regression
We shall also put the notes in goodnotes here
1. interpreint regression coefficients:希望input 时uncorrelated；correlation 会影响；可以单独和output比较
2. RSS来判定好坏
3. Is at least one of the predictors $$X\_1,\cdots, X\_p$$ useful in predictiing the response? $$F$$ Statistic:
4. Do all the predictors help to explain $$Y$$, or is only a subset of the predictors useful? (不可能经过所有的input；所以基于最小化$RSS$选择一个$$X\_i$$，然后你基于最小化$$RSS$$选择第二个$$X\_j$$，直到选出来的$$p$$-value合格)（或者你可以采用全部放进去，基于$$p$$-value,然后一个个删掉）
5. How well does the model fit the data?
* systematic criteria for choosing an 'optimal' member in the path of models produced by forward or backward stepwise selection;
* 其他度量方式 Mallow's $$C\_p$$, Akaike information criterion(AIC), Bayesian information criterion(BIC), adjusted $$R^2$$, Cross-validation(CV)
6. Given a set of predictor values, what response value should we predict, and how accurate is our prediction
7. 小心qualitative data；可以换成binary $$x\_1=0&1$$在不同情况下，当然还可以有 $x\_2$
8. Removing the additive assumption: **interactions and nonlinearity**
* Interaction:市场造成的相互的影响，比如说你增加$$x\_1$$会影响$$x\_2$$;这时候刚增加一个 $$x\_1x\_2$$项
* hierarchy：hierarchy principle：if we include an interaction in a model, we should also include the main effects, even if the $p$-value associated with their coefficients are not significant.
9. outliers\&non-constant variance of error terms& high leverage points& collinearity section3.3
## Gradient descent简单的解释
Gradient descent is a commonly used optimization technique for other models as well, like neural networks, which we'll explore later in this track. Here's an overview of the gradient descent algorithm for a single-parameter linear regression model:
* select initial values for the parameter: $$a\_1$$
* repeat until convergence (usually implemented with a max number of iterations):
* calculate the error (MSE) of the model that uses the current parameter value: $$MSE(a\_1)=\frac{1}{n}\sum\_{i=1}^n({\hat{y}}^{(i)}-y^{(i)})^2$$
* calculate the derivative of the error (MSE) at the current parameter value: $\frac{d}{da\_1}MSE(a\_1)$
* update the parameter value by subtracting the derivative times a constant ($\alpha$, called the learning rate): $$a\_1=a\_1-\alpha \frac{d}{da\_1}MSE(a\_1)$$
Reference:
1. books: an introduction to statistical learning
2. notes in Good notes
3. lai notes
