> For the complete documentation index, see [llms.txt](https://ai.younglimit.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai.younglimit.com/machine-learning/linear-model-cheating-sheet.md).

# Linear Model Cheating Sheet

## Linear Regression

### Definition

* Linear regression is a linear model, which a linear relationship between the input and output . More technical, we can consider $$y$$​ can be a linear combination of $$X$$​

### Representation

* $$Y= \beta\_0+\beta\_1X+\epsilon$$

### How to determine this model?loss function？

* 最小二乘法least square推导loss function
  * $$Loss =\min\_{\beta\_0,\beta\_1} \sum\_{i=1}^{n}(y\_i-\beta\_0+\beta\_1X\_i)^2$$
* 最大似然估计MLE（maximum likelihood estimation）推导loss function
  * what is mle?
    * 用参数估计的方法，在有了一定的观测值之后，来找parameter，让我们可以最有可能看到我们观测值，让我们可以最大程度放大我们的观测值
    * method
    * estimating the parameters
    * statistical model
    * given observation
    * finding the parameter values
    * maximize the likelihood(making the observation given the parameters)
  * 推导过程（Y follws什么distribution？）
    * $$p(Y\_i\vert X\_i)=\frac{1}{\sigma(2\pi)}e^{-\frac{1}{2\sigma^2}(Y\_i-\bar{\beta}\_0-\bar{\beta}\_1X\_i)^2}$$
    * $$L(\bar{\beta}\_0,\bar{\beta}*1,\sigma^2)=p(Y\_1,\cdots,Y\_n\vert X\_1,\cdots,X\_n)=\frac{1}{\sigma^{n}(2\pi)^{n/2}}e^{-\frac{1}{2\sigma^2}\sum*{i=1}^n(Y\_i-\bar{\beta}\_0-\bar{\beta}\_1X\_i)^2}$$
  * ![](https://api2.mubu.com/v3/document_image/f5a43d32-9b61-4021-bfe6-0c3540a493f4-12267179.jpg)
* 两者的关系
  * 当他们在linear regression下的assumption下，这两个方法得到结果是相通的
  * one is from statistics, and the other one is from optimization
* 一些提醒
  * noise是数据造成的，是inherent bias. error是模型造成的，是人为的。是两个不同的概念

### How is the performance of this model?

* （这个是来看模型自己的好坏，评价自己的参数）
* 我们只能系统的保证其不会偏差（$$E(Y)=\mu$$​​​​​​​​）
* null hypothesis : $$\beta\_1=0$$
  * 目的是从统计上来判断这组数据和population相差多少，assessing the accuracy of the coefficient estimation,可以使用 ppp​-value 或者是 confidence interval
  * 选择的统计量$$t=\frac{\hat{\beta}\_1-0}{SE(\hat{\beta}\_1)}$$​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​distribution with $$n-2$$ degrees of freedom assuming $$\beta\_1=0$$​​​​​​

### How can we compare this model with others model?

* (这个相当于模型外的判断模型的好坏 i.e. the extent to which the model fits the data)
* assessing the overall accuracy
* $$RSE=\sqrt{\frac{1}{n-2}RSS}=\sqrt{\frac{1}{n-2}\sum\_i^{n}(y\_i-\hat{y}\_i)^2}$$
* $$R^2=\frac{TSS-RRR}{TSS}=1-\frac{RSS}{\sum\_i^{n}(y\_i-\hat{y}\_i)^2}$$
  * where TSSTSSTSS​​​ is total sum of squares, RSSRSSRSS​​​ is the residual sum of squares(对误差的多少)
  * 当是simple regression时，他相同于correlation
  * 这里相当于 proportion of variability in that can be explained using ( 的变化中能够被解释的部分的比例 )

## Multilinear Regression

## Logistic Regression

### 图形/值域的理解

* 采用这个方法：$$x \rightarrow p(x) \rightarrow y$$
  * 首先用 $$x$$​去拟合 概率$$p(x)$$​​​​
  * 然后用$$p$$​再去拟合$$y$$​ （采用threshold）
  * $$p(x)=\frac{e^{f(x)}}{1+e^{f(x)}}=\frac{1}{1+e^{-f(x)}}$$
* $$Y$$​是得病不得病，$$p$$​是相当于肿瘤的指数， $$X$$​是关于肿瘤的input（size，位置，etc）
  * $$p(Y=y\_i)=p^{y\_i}(1-p)^{1-y\_i},\ 0\<p<1,\ y\_i=0,1$$
  * $$log\frac{p(X)}{1-p(X)} = \beta\_0+\beta\_1X$$

### How to determine this model?loss function？

* Using MLE to get the loss function（面试必考题）
  * Key：Y fellows 什么distribution？
    * $$P(Y=y\_i\vert X) = p^{y\_i}(1-p)^{1-y\_i}, \ 0\<p<1,\ y\_i=0,1$$
  * $$p=h\_{\beta}(x\_i)=\frac{1}{1+e^{-\beta\_0+\beta\_1x}}$$​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​
  * $$L(\hat\beta\_0,\hat \beta\_1)=\Pi\_{i=1}^{n}P(Y\_i \vert X\_i)= \Pi\_{i=0}^n p^{y\_i}(1-p)^{1-y\_i}=\Pi\_{i=0}^n {h\_{\beta}(x\_i)}^{y\_i}(1-h\_{\beta}(x\_i))^{1-y\_i}$$
* 如何调参呢？有没有类似与linear的t distribution之类的？

### How is the performance of this model?

* t distribution？something？

### How can we compare this model with others model?

* link: Confusion matrix/AUC

### Compare with linear regression

* 关于y的assumption
* Step1： ‘Given ，得的distribution’——这是机器学习模型利用数据可以解决的问题
* Step2: ‘根据的distribution，得到的取值’——判断怎么用是你的问题

## Multiple Logistic Regression

* $$p(x)=h\_{\beta}(x\_i)=\frac{1}{1+e^{g(x)}}, \ \textit{where }\ g(x)=\beta\_0+\beta\_{1,1}x\_1+\beta\_{1,2}x\_1^2+\beta\_2x\_2$$
* 你会面对一个（overfit/underfit）的问题：
  * 模型复杂度对分类效果的影响
  * 模型复杂提到，可以更好的对应training data，但是对testing data不一定好
  * 过度拟合就是overfitting，但是过少可能就underfitting
* 很多时候，多分类问题可以比较成为多个两分类问题，两两二分类来做

## Regularization——Ridge regression/Lasso regression

## SVM

* $$E((w^Tx+b,0)\_{\max})$$
* Maximum margin classifier $$\max \frac{1}{\vert \vert w \vert \vert},\qquad s.t. y\_i(w^Tx\_i+b)\geq 0,\qquad i=1,\cdots$$
* $$\min \frac{1}{2}\vert\vert w \vert\vert^2,\qquad s.t. y\_i(w^Tx\_i+b)\geq 1,\qquad i=1,\cdots,n$$
  * dual $$\mathcal{L}(w,b,a)=\frac{1}{2}\vert \vert w\vert \vert^2-\sum\_{i=1}^n \alpha\_i(y\_i(w^Tx\_i+b)-1)$$
  * $$\mathcal{L}(w,b,a)=\sum\_{i=1}^n\alpha\_i-\frac{1}{2}\alpha\_i\alpha\_jy\_i y\_jx\_i^T x\_j$$ and $$w=\sum\_{i=1}^n\alpha\_iy\_ix\_i$$
* 换成核估计
  * $$f(x)=\sum\_{i=1}^Nw\_i\phi\_i(x)+b$$ 转换成为 $$f(x)=\sum\_{i=1}^l\alpha\_i y\_i \langle \phi(x\_i), \phi(x) \rangle+b$$
  * $$\alpha$$​​​​​​可以由dual 来求
    * $$\max\_{\alpha}\sum\_{i=1}^n \alpha\_i-\frac{1}{2}\sum\_{i,j=1}^n\alpha\_i\alpha\_jy\_iy\_j\langle \phi(x\_i)\phi(x\_j)\rangle \qquad s.t. \alpha\_i \geq 0,i=1,\cdots, n; \sum\_{i=1}^n\alpha\_iy\_i=0$$
    * $$\max\_{\alpha}\sum\_{i=1}^n \alpha\_i-\frac{1}{2}\sum\_{i,j=1}^n\alpha\_i\alpha\_jy\_iy\_jx\_ix\_j \qquad s.t. \alpha\_i \geq 0,i=1,\cdots, n; \sum\_{i=1}^n\alpha\_iy\_i=0$$


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://ai.younglimit.com/machine-learning/linear-model-cheating-sheet.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.