♎
Limited AI
  • Machine Learning
    • Linear Model Cheating Sheet
    • Nonlinear Model Cheating Sheet
    • General Linear Model 1
    • General Linear Model 2
    • General Linear Model 3
    • Tree Based Methods
    • Tree Based Methods Supplement
    • XG,Cat,Light__Boosting
    • KNN&PCA
    • Model Performance
    • Model Evaluation
    • Code Practice
      • KNN
      • Decision Tree Python Code
    • Data and Feature Engineering
      • Handle Bias Data
      • Cold Start Problem
  • Deep Learning
    • Summary v2
    • Basic Neural Network
      • From Linear to Deep
      • Perceptron and Activation Function
      • NN network Details
      • Backpropagation Details
      • Gradient Vanishing vs Gradient Exploding
    • Basic CNN
      • Why CNN
      • Filter/ Convolution Kernel and Its Operation
      • Padding& Stride
      • Layers
      • Extra:From Fully Connected Layers to Convolutions
      • Extra: Multiple Input and Multiple Output Channels
    • Advance CNN
      • Convolutional Neural Networks(LeNet)
      • Deep Convolution Neural Networks(AlexNet)
      • Networks Using Blocks (VGG)
      • Network in Network(NiN)
      • Multi-Branch Networks(GoogLeNet&I mageNet)
      • Residual Networks(ResNet) and ResNeXt
      • Densely Connected Networks(DenseNet)
      • Batch Normalization
    • Basic RNN
      • Seq Model
      • Raw Text to Seq
      • Language Models
      • Recurrent Neural Networks(RNN)
      • Backpropagation Through Time
    • Advance RNN
      • Gated Recurrent Units(GRU)
      • Long Short-Term Memory(LSTM)
      • Bidirectional Recurrent Neural Networks(BRNN)
      • Encoder-Decoder Architecture
      • Seuqence to Sequence Learning(Seq2Seq)
    • Attention Mechanisms and Transformers
      • Queries, Keys, and Values
      • Attention is all you need
        • Attention and Kernel
        • Attention Scoring Functions
        • The Bahdanau Attention Mechanism
        • Multi-Head Attention
        • Self-Attention
        • Attention的实现
      • The Transformer Architecture
        • Extra Reading
        • 最短的最大路径长度
      • Large-Scaling Pretraning with Transformers
        • BERT vs OpenAI GPT vs ELMo
        • Decoder Model框架
        • Bert vs XLNet
        • T5& GPT& Bert比较
        • 编码器-解码器架构 vs GPT 模型
        • Encoder vs Decoder Reference
      • Transformers for Vision
      • Transformer for Multiomodal
    • NLP Pretraining
      • Word Embedding(word2vec)
        • Extra Reading
      • Approximate Training
      • Word Embedding with Global Vectors(GloVe)
        • Extra Reading
        • Supplement
      • Encoder(BERT)
        • BERT
        • Extra Reading
      • Decoder(GPT&XLNet&Lamma)
        • GPT
        • XLNet
          • XLNet架构
          • XLNet特点与其他比较
      • Encoder-Decoder(BART& T5)
        • BART
        • T5
  • GenAI
    • Introduction
      • GenAI Paper Must Read
      • GenAI六个阶段
    • Language Models Pre-training
      • Encoder-Decoder Architecture
      • Encoder Deep Dive
      • Decoder Deep Dive
      • Encoder VS Decoder
      • Attention Mechanism
      • Transformers
    • Example: Llama 3 8B架构
    • Fine-Tuning Generation Models
    • RAG and Adavance RAG
    • AI Agent
  • Statistics and Optimization
    • A/B testing
    • Sampling/ABtesting/GradientMethod
    • Gradient Decent Deep Dive
  • Machine Learning System Design
    • Extra Reading
    • Introduction
  • Responsible AI
    • AI Risk and Uncertainty
      • What is AI risk
      • General Intro for Uncertainty Quantification
      • Calibration
      • Conformal Prediction
        • Review the linear regression
        • Exchangeability
        • Split Conformal Prediction
        • Conformalized Quantile Regression
        • Beyond marginal coverage
        • Split Conformal Classification
        • Full Conformal Coverage
        • Cross-Validation +
        • Conformal Histgram Regression
    • xAI
      • SHAP value
  • Extra Research
    • Paper Reading
    • Reference
Powered by GitBook
On this page
  • AlexNet: A Landmark in Deep Learning | AlexNet：深度学习的里程碑
  • Extra Reading
  1. Deep Learning
  2. Advance CNN

Deep Convolution Neural Networks(AlexNet)

AlexNet: A Landmark in Deep Learning | AlexNet：深度学习的里程碑

1. What is AlexNet? | 什么是 AlexNet？

AlexNet is a deep convolutional neural network (CNN) architecture that revolutionized computer vision and deep learning. It was developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, and won the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) by a significant margin. AlexNet's victory popularized the use of deep learning and CNNs for large-scale image classification tasks. AlexNet 是一种深度卷积神经网络（CNN）架构，它彻底改变了计算机视觉和深度学习。由 Alex Krizhevsky、Ilya Sutskever 和 Geoffrey Hinton 开发，并在 2012 年的 ImageNet 大规模视觉识别挑战赛（ILSVRC）中取得了压倒性胜利。AlexNet 的胜利使得深度学习和 CNN 在大规模图像分类任务中的应用得到了广泛认可。

2. Architecture of AlexNet | AlexNet 的架构

AlexNet consists of 8 layers: 5 convolutional layers followed by 3 fully connected layers. It introduced several innovations, such as the use of Rectified Linear Unit (ReLU) activations, dropout for regularization, and overlapping max-pooling. Below is an overview of the architecture:

  1. Input Layer | 输入层:

    • The input to AlexNet is a 224x224x3 RGB image.

    • 输入 AlexNet 的是一个 224x224x3 的 RGB 图像。

  2. First Convolutional Layer (Conv1) | 第一卷积层 (Conv1):

    • 96 filters of size 11x11 with a stride of 4, resulting in 96 feature maps of size 55x55.

    • This layer captures low-level features such as edges and textures.

    • 使用 96 个大小为 11x11 的滤波器，步幅为 4，生成 96 个大小为 55x55 的特征图。

    • 这一层捕捉低级特征，如边缘和纹理。

  3. Max Pooling Layer (Pool1) | 最大池化层 (Pool1):

    • 3x3 max pooling with a stride of 2, reducing the spatial size to 27x27.

    • 3x3 的最大池化，步幅为 2，将空间尺寸减少到 27x27。

  4. Second Convolutional Layer (Conv2) | 第二卷积层 (Conv2):

    • 256 filters of size 5x5 with padding, resulting in 256 feature maps of size 27x27.

    • 使用 256 个 5x5 的滤波器，带有填充，生成 256 个大小为 27x27 的特征图。

  5. Max Pooling Layer (Pool2) | 最大池化层 (Pool2):

    • Similar to the first pooling layer, a 3x3 max pooling with a stride of 2, reducing the size to 13x13.

    • 与第一个池化层类似，3x3 最大池化，步幅为 2，将尺寸减少到 13x13。

  6. Third, Fourth, and Fifth Convolutional Layers (Conv3, Conv4, Conv5) | 第三、第四和第五卷积层 (Conv3, Conv4, Conv5):

    • Conv3: 384 filters of size 3x3, producing 384 feature maps.

    • Conv4: 384 filters of size 3x3, connected to the output of Conv3.

    • Conv5: 256 filters of size 3x3, followed by another max pooling layer.

    • Conv3: 384 个 3x3 滤波器，生成 384 个特征图。

    • Conv4: 384 个 3x3 滤波器，连接到 Conv3 的输出。

    • Conv5: 256 个 3x3 滤波器，之后还有一个最大池化层。

  7. Fully Connected Layers (FC6, FC7, FC8) | 全连接层 (FC6, FC7, FC8):

    • FC6: A fully connected layer with 4096 neurons, receiving the flattened output from the previous layers.

    • FC7: Another fully connected layer with 4096 neurons.

    • FC8: The final fully connected layer with 1000 neurons, corresponding to the 1000 classes in the ImageNet dataset.

    • FC6: 具有 4096 个神经元的全连接层，接收前几层的展平输出。

    • FC7: 另一个具有 4096 个神经元的全连接层。

    • FC8: 最后的全连接层具有 1000 个神经元，对应 ImageNet 数据集中的 1000 个类别。

  8. Output Layer | 输出层:

    • The output is a probability distribution over 1000 classes, obtained by applying a softmax activation.

    • 输出是对 1000 个类别的概率分布，应用 softmax 激活函数获得。

3. Key Innovations of AlexNet | AlexNet 的关键创新

  • ReLU Activation | ReLU 激活函数: AlexNet was one of the first architectures to use ReLU (Rectified Linear Unit), which introduced non-linearity more effectively than traditional activation functions like Sigmoid or Tanh. ReLU also helped with faster training by mitigating the vanishing gradient problem. ReLU 激活函数： AlexNet 是最早使用 ReLU（修正线性单元） 的架构之一，比传统的 Sigmoid 或 Tanh 激活函数更有效地引入了非线性。ReLU 还通过缓解梯度消失问题帮助实现了更快的训练。

  • Dropout for Regularization | Dropout 正则化: AlexNet used dropout in the fully connected layers to prevent overfitting. Dropout randomly turns off neurons during training to ensure the network does not become overly dependent on any specific features. Dropout 正则化： AlexNet 在全连接层使用 dropout 来防止过拟合。Dropout 在训练期间随机关闭神经元，确保网络不过度依赖任何特定特征。

  • Overlapping Max Pooling | 重叠最大池化: Instead of non-overlapping pooling (like in LeNet), AlexNet used overlapping max pooling, which allows pooling windows to overlap, resulting in better generalization. 重叠最大池化： 与 LeNet 中的不重叠池化不同，AlexNet 使用了重叠最大池化，允许池化窗口重叠，从而带来了更好的泛化能力。

  • Data Augmentation | 数据增强: AlexNet utilized data augmentation techniques, such as random cropping and horizontal flipping, to artificially increase the size of the training set and improve the network's ability to generalize to new data. 数据增强： AlexNet 使用了 数据增强 技术，例如随机裁剪和水平翻转，以人为地增加训练集的大小，并提高网络泛化到新数据的能力。

  • GPU Utilization | 使用 GPU: AlexNet was one of the first models to fully exploit GPU power for training, significantly speeding up the training process and enabling the training of deep networks. 使用 GPU： AlexNet 是首批充分利用 GPU 计算能力进行训练的模型之一，大大加快了训练过程，并使深层网络的训练成为可能。

4. Why Was AlexNet Important? | 为什么 AlexNet 很重要？

  • Performance Breakthrough | 性能突破: AlexNet achieved a top-5 error rate of 15.3% on ImageNet, significantly outperforming all previous methods and marking a substantial leap in the accuracy of deep learning models for image recognition. 性能突破： AlexNet 在 ImageNet 上取得了 15.3% 的 top-5 错误率，远超此前的所有方法，标志着深度学习图像识别模型准确率的显著飞跃。

  • Deep Learning Boom | 深度学习的兴起: AlexNet's success popularized the use of deep convolutional neural networks and sparked the rapid development of deep learning in both academic and industry settings. 深度学习的兴起： AlexNet 的成功推广了深度卷积神经网络的使用，并在学术界和工业界引发了深度学习的快速发展。

  • Foundation for Modern CNNs | 现代 CNN 的基础: AlexNet’s architecture and techniques (ReLU, dropout, data augmentation) have become standard in most modern deep learning architectures like VGG, GoogLeNet, ResNet, and more. 现代 CNN 的基础： AlexNet 的架构和技术（如 ReLU、dropout、数据增强）已经成为现代深度学习架构的标准，如 VGG、GoogLeNet、ResNet 等。

5. Example of AlexNet in Action | AlexNet 的操作示例

Consider an example of classifying an image of a dog using AlexNet:

  • The input is a 224x224 RGB image of the dog.

  • AlexNet processes the image through convolutional and pooling layers, progressively extracting more complex features.

  • After passing through the fully connected layers, the output is a probability distribution over 1000 classes.

  • The class with the highest probability might be "Labrador" or "Golden Retriever."

In this case, AlexNet can recognize the breed of the dog from the ImageNet dataset, outputting the most likely label based on its learned features. 以使用 AlexNet 对狗的图像进行分类为例：

  • 输入是一张 224x224 的 RGB 狗图像。

  • AlexNet 通过卷积层和池化层处理图像，逐步提取更复杂的特征。

  • 通过全连接层后，输出是对 1000 个类别的概率分布。

  • 概率最高的类别可能是“拉布拉多犬”或“金毛猎犬”。

在这种情况下，AlexNet 能够从 ImageNet 数据集中识别狗的品种，并根据其学习到的特征输出最有可能的标签。

6. Limitations of AlexNet | AlexNet 的局限性

  • Size and Complexity | 大小和复杂性: AlexNet is relatively large, with about 60 million parameters, making it computationally expensive compared to more modern architectures like ResNet or MobileNet. 大小和复杂性： AlexNet 相对较大，具有大约 6000 万个参数，与 ResNet 或 MobileNet 等现代架构相比，其计算开销较大。

  • Manual Design | 手工设计: The architecture of AlexNet required manual tuning and design, whereas more recent models use techniques like automatic architecture search to design more efficient models. 手工设计： AlexNet 的架构需要手动调整和设计，而最近的模型使用了自动架构搜索等技术来设计更高效的模型。

AlexNet 是一个开创性的架构，它推广了 CNN 在大规模图像分类中的应用。它引入了 ReLU、dropout 和重叠最大池化等关键创新，使其在 ImageNet 挑战赛中取得了巨大的成功。虽然 AlexNet 已被更高效的架构所取代，但它在现代深度学习中的影响力仍然非常重要。

Extra Reading

  • 当人们和机器学习研究人员交谈时，会发现机器学习研究人员相信机器学习既重要又美丽：优雅的理论去证明各种模型的性质。机器学习是一个正在蓬勃发展、严谨且非常有用的领域。然而，当人们和计算机视觉研究人员交谈，会听到一个完全不同的故事。计算机视觉研究人员会告诉一个诡异事实————推动领域进步的是数据特征，而不是学习算法。计算机视觉研究人员相信，从对最终模型精度的影响来说，更大或更干净的数据集、或是稍微改进的特征提取，比任何学习算法带来的进步要大得多。

  • 有趣的是，在网络的最底层，模型学习到了一些类似于传统滤波器的特征抽取器。

  • 相比于CPU，GPU由100∼1000100∼1000100∼1000个小的处理单元组成（NVIDIA、ATI、ARM和其他芯片供应商之间的细节稍有不同），通常被分成更大的组（NVIDIA称之为warps）。 虽然每个GPU核心都相对较弱，有时甚至以低于1GHz的时钟频率运行，但庞大的核心数量使GPU比CPU快几个数量级。

  • AlexNet使用了8层卷积神经网络，并以很大的优势赢得了2012年ImageNet图像识别挑战赛。

PreviousConvolutional Neural Networks(LeNet)NextNetworks Using Blocks (VGG)

Last updated 7 months ago

一组研究人员，包括Yann LeCun、Geoff Hinton、Yoshua Bengio、Andrew Ng、Shun ichi Amari和Juergen Schmidhuber，想法则与众不同：他们认为特征本身应该被学习。此外，他们还认为，在合理地复杂性前提下，特征应该由多个共同学习的神经网络层组成，每个层都有可学习的参数。在机器视觉中，最底层可能检测边缘、颜色和纹理。事实上，Alex Krizhevsky、Ilya Sutskever和Geoff Hinton提出了一种新的卷积神经网络变体AlexNet。在2012年ImageNet挑战赛中取得了轰动一时的成绩。AlexNet以Alex Krizhevsky的名字命名，他是论文 ()的第一作者。

Krizhevsky et al., 2012