♎
Limited AI
  • Machine Learning
    • Linear Model Cheating Sheet
    • Nonlinear Model Cheating Sheet
    • General Linear Model 1
    • General Linear Model 2
    • General Linear Model 3
    • Tree Based Methods
    • Tree Based Methods Supplement
    • XG,Cat,Light__Boosting
    • KNN&PCA
    • Model Performance
    • Model Evaluation
    • Code Practice
      • KNN
      • Decision Tree Python Code
    • Data and Feature Engineering
      • Handle Bias Data
      • Cold Start Problem
  • Deep Learning
    • Summary v2
    • Basic Neural Network
      • From Linear to Deep
      • Perceptron and Activation Function
      • NN network Details
      • Backpropagation Details
      • Gradient Vanishing vs Gradient Exploding
    • Basic CNN
      • Why CNN
      • Filter/ Convolution Kernel and Its Operation
      • Padding& Stride
      • Layers
      • Extra:From Fully Connected Layers to Convolutions
      • Extra: Multiple Input and Multiple Output Channels
    • Advance CNN
      • Convolutional Neural Networks(LeNet)
      • Deep Convolution Neural Networks(AlexNet)
      • Networks Using Blocks (VGG)
      • Network in Network(NiN)
      • Multi-Branch Networks(GoogLeNet&I mageNet)
      • Residual Networks(ResNet) and ResNeXt
      • Densely Connected Networks(DenseNet)
      • Batch Normalization
    • Basic RNN
      • Seq Model
      • Raw Text to Seq
      • Language Models
      • Recurrent Neural Networks(RNN)
      • Backpropagation Through Time
    • Advance RNN
      • Gated Recurrent Units(GRU)
      • Long Short-Term Memory(LSTM)
      • Bidirectional Recurrent Neural Networks(BRNN)
      • Encoder-Decoder Architecture
      • Seuqence to Sequence Learning(Seq2Seq)
    • Attention Mechanisms and Transformers
      • Queries, Keys, and Values
      • Attention is all you need
        • Attention and Kernel
        • Attention Scoring Functions
        • The Bahdanau Attention Mechanism
        • Multi-Head Attention
        • Self-Attention
        • Attention的实现
      • The Transformer Architecture
        • Extra Reading
        • 最短的最大路径长度
      • Large-Scaling Pretraning with Transformers
        • BERT vs OpenAI GPT vs ELMo
        • Decoder Model框架
        • Bert vs XLNet
        • T5& GPT& Bert比较
        • 编码器-解码器架构 vs GPT 模型
        • Encoder vs Decoder Reference
      • Transformers for Vision
      • Transformer for Multiomodal
    • NLP Pretraining
      • Word Embedding(word2vec)
        • Extra Reading
      • Approximate Training
      • Word Embedding with Global Vectors(GloVe)
        • Extra Reading
        • Supplement
      • Encoder(BERT)
        • BERT
        • Extra Reading
      • Decoder(GPT&XLNet&Lamma)
        • GPT
        • XLNet
          • XLNet架构
          • XLNet特点与其他比较
      • Encoder-Decoder(BART& T5)
        • BART
        • T5
  • GenAI
    • Introduction
      • GenAI Paper Must Read
      • GenAI六个阶段
    • Language Models Pre-training
      • Encoder-Decoder Architecture
      • Encoder Deep Dive
      • Decoder Deep Dive
      • Encoder VS Decoder
      • Attention Mechanism
      • Transformers
    • Example: Llama 3 8B架构
    • Fine-Tuning Generation Models
    • RAG and Adavance RAG
    • AI Agent
  • Statistics and Optimization
    • A/B testing
    • Sampling/ABtesting/GradientMethod
    • Gradient Decent Deep Dive
  • Machine Learning System Design
    • Extra Reading
    • Introduction
  • Responsible AI
    • AI Risk and Uncertainty
      • What is AI risk
      • General Intro for Uncertainty Quantification
      • Calibration
      • Conformal Prediction
        • Review the linear regression
        • Exchangeability
        • Split Conformal Prediction
        • Conformalized Quantile Regression
        • Beyond marginal coverage
        • Split Conformal Classification
        • Full Conformal Coverage
        • Cross-Validation +
        • Conformal Histgram Regression
    • xAI
      • SHAP value
  • Extra Research
    • Paper Reading
    • Reference
Powered by GitBook
On this page
  • Encoder-Decoder Architecture | 编码器-解码器架构
  • Extra Reading
  • Encoder& Decoder
  1. Deep Learning
  2. Advance RNN

Encoder-Decoder Architecture

Encoder-Decoder Architecture | 编码器-解码器架构

1. What is Encoder-Decoder Architecture? | 什么是编码器-解码器架构？

Encoder-Decoder Architecture is a deep learning framework commonly used for tasks where the input and output are sequences of different lengths, such as machine translation, text summarization, and image captioning. The architecture consists of two main components:

  • Encoder: Processes the input sequence and compresses it into a fixed-length representation (known as the context or hidden state).

  • Decoder: Takes the compressed representation from the encoder and generates the output sequence step-by-step.

The Encoder-Decoder architecture is particularly powerful for handling tasks that require a sequence-to-sequence (seq2seq) transformation. The encoder processes the entire input sequence, and the decoder generates the output sequence based on the compressed information provided by the encoder. 编码器-解码器架构 是一种常用于输入输出序列长度不同的任务的深度学习框架，如机器翻译、文本摘要和图像描述。该架构主要包括两个部分：

  • 编码器：处理输入序列并将其压缩为固定长度的表示（通常称为上下文或隐藏状态）。

  • 解码器：接收编码器生成的压缩表示，逐步生成输出序列。

编码器-解码器 架构在需要序列到序列（seq2seq）转换的任务中特别强大。编码器处理整个输入序列，解码器基于编码器提供的压缩信息生成输出序列。

2. How Does Encoder-Decoder Work? | 编码器-解码器是如何工作的？

Encoder | 编码器

The encoder is responsible for processing the input sequence (e.g., a sentence) and transforming it into a fixed-length vector. The encoder typically consists of a stack of recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or gated recurrent units (GRUs).

At each time step, the encoder updates its hidden state based on the current input and the previous hidden state. After processing the entire input sequence, the final hidden state (also called the context vector) represents a compressed summary of the input sequence. This context vector is passed to the decoder. 编码器 负责处理输入序列（如一句话）并将其转换为固定长度的向量。编码器通常由一组递归神经网络 (RNN)、长短期记忆网络 (LSTM) 或门控循环单元 (GRU) 组成。

在每个时间步，编码器根据当前输入和先前的隐藏状态更新其隐藏状态。处理完整个输入序列后，最终的隐藏状态（也称为 上下文向量）表示输入序列的压缩摘要。该上下文向量将传递给解码器。

Decoder | 解码器

The decoder is responsible for generating the output sequence based on the context vector provided by the encoder. Like the encoder, the decoder is often implemented using RNNs, LSTMs, or GRUs. At each time step, the decoder generates the next output token based on the previous hidden state and the current context vector.

The decoder typically uses a softmax layer to predict the probability distribution over the vocabulary, and the word with the highest probability is selected as the next output. The decoder continues this process until it generates an end-of-sequence token or reaches the desired length of the output. 解码器 负责基于编码器提供的上下文向量生成输出序列。与编码器类似，解码器通常由 RNN、LSTM 或 GRU 实现。在每个时间步，解码器基于先前的隐藏状态和当前的上下文向量生成下一个输出标记。

解码器通常使用 softmax 层预测词汇表上的概率分布，并选择概率最高的词作为下一个输出。解码器会继续这个过程，直到生成 序列结束 标记或达到所需的输出长度。

Training Process | 训练过程

During training, the encoder processes the entire input sequence, and the decoder learns to generate the corresponding output sequence. In a supervised learning setting, the decoder is trained using teacher forcing, where the ground truth output at each time step is provided as input to the decoder, ensuring that it learns to generate the correct sequence. 在训练过程中，编码器处理整个输入序列，解码器学习生成对应的输出序列。在监督学习环境中，解码器使用 教师强制 进行训练，即每个时间步都提供真实的输出作为解码器的输入，以确保其学习生成正确的序列。

3. Applications of Encoder-Decoder Architecture | 编码器-解码器架构的应用

  • Machine Translation | 机器翻译: The most common application of the Encoder-Decoder architecture is in machine translation. For example, the encoder processes a sentence in English, and the decoder generates the corresponding sentence in French. 机器翻译：编码器-解码器架构最常见的应用是机器翻译。例如，编码器处理一段英语句子，解码器生成对应的法语句子。

  • Text Summarization | 文本摘要: In text summarization, the encoder processes a long document, and the decoder generates a shorter summary. 文本摘要：在文本摘要中，编码器处理一篇长文档，解码器生成简短摘要。

  • Image Captioning | 图像描述: For image captioning, a convolutional neural network (CNN) can be used as the encoder to process the image, and an RNN-based decoder generates a caption for the image. 图像描述：在图像描述任务中，卷积神经网络 (CNN) 可用作编码器处理图像，而基于 RNN 的解码器生成图像的描述。

  • Speech Recognition | 语音识别: In speech recognition systems, the encoder processes an audio signal, and the decoder generates the corresponding transcript. 语音识别：在语音识别系统中，编码器处理音频信号，解码器生成相应的转录文本。

4. Limitations of Encoder-Decoder Architecture | 编码器-解码器架构的局限性

  • Bottleneck Problem | 瓶颈问题: The primary limitation of the basic Encoder-Decoder architecture is the bottleneck caused by compressing the entire input sequence into a single fixed-length vector (the context vector). This can result in information loss, especially for long input sequences. 瓶颈问题：基本的编码器-解码器架构的主要局限是将整个输入序列压缩为一个固定长度的向量（上下文向量）导致的 瓶颈。这可能导致信息丢失，尤其是对于长输入序列。

  • Difficulty with Long Sequences | 处理长序列的困难: For tasks involving long sequences, the fixed-length context vector might not be sufficient to capture all relevant information, resulting in degraded performance. 处理长序列的困难：对于包含长序列的任务，固定长度的上下文向量可能不足以捕捉所有相关信息，从而导致性能下降。

5. Attention Mechanism in Encoder-Decoder | 编码器-解码器中的注意力机制

To address the bottleneck problem, the attention mechanism was introduced as an enhancement to the Encoder-Decoder architecture. In an attention-based model, instead of relying on a single fixed-length context vector, the decoder selectively focuses on different parts of the input sequence at each time step. This allows the model to dynamically retrieve relevant information from the input, making it more effective for handling long and complex sequences.

The attention mechanism calculates a weighted sum of the encoder’s hidden states, where the weights are determined based on how important each hidden state is for the current decoding step. 为了解决瓶颈问题，引入了 注意力机制 作为对编码器-解码器架构的增强。在基于注意力的模型中，解码器不再依赖单一的固定长度上下文向量，而是每个时间步 选择性地关注 输入序列的不同部分。这使得模型能够动态地从输入中提取相关信息，从而更有效地处理长序列和复杂序列。

注意力机制计算编码器隐藏状态的 加权和，其中权重根据每个隐藏状态对当前解码步骤的重要性来确定。

6. Example of Encoder-Decoder in Action | 编码器-解码器架构的操作示例

Consider an example of machine translation using an Encoder-Decoder architecture:

  1. Encoder: The encoder processes an English sentence, "The weather is nice today," and compresses it into a context vector that summarizes the sentence.

  2. Decoder: The decoder takes the context vector and generates the French translation "Il fait beau aujourd'hui" one word at a time.

During training, the decoder learns to generate each French word based on the context vector and the previously generated words, using teacher forcing to ensure it learns the correct sequence.

Example with attention mechanism:

  1. At each decoding step, the decoder selectively focuses on specific parts of the input sentence, retrieving more relevant information from the encoder’s hidden states to generate accurate translations.

This dynamic attention helps the model better handle long sentences by focusing on the relevant parts of the input. 以 机器翻译 使用编码器-解码器架构为例：

  1. 编码器：编码器处理一句英语句子 “The weather is nice today”，并将其压缩为一个总结句子含义的上下文向量。

  2. 解码器：解码器接收上下文向量并逐字生成法语翻译 "Il fait beau aujourd'hui"。

在训练过程中，解码器通过教师强制学习在每个时间步生成正确的法语单词，确保其学会生成正确的序列。

使用注意力机制的示例：

  1. 在每个解码步骤，解码器选择性地关注输入句子的特定部分，从编码器的隐藏状态中提取更多相关信息以生成准确的翻译。

这种动态注意力机制帮助模型通过关注输入中的相关部分更好地处理长句子。

编码器-解码器架构 是用于序列到序列任务的强大框架，编码器处理输入，解码器生成输出。其优势在于适用于机器翻译、文本摘要和其他基于序列的任务。局限性是存在瓶颈问题，可以通过 注意力机制 缓解。该架构的应用包括机器翻译、文本摘要、图像描述和语音识别等。

Extra Reading

In general sequence-to-sequence problems like machine translation, inputs and outputs are of varying lengths that are unaligned. The standard approach to handling this sort of data is to design an encoder-decoder architecture consisting of two major components: an encoder that takes a variable-length sequence as input, and a decoder that acts as a conditional language model, taking in the encoded input and the leftwards context of the target sequence and predicting the subsequent token in the target sequence.

  • 机器翻译是序列转换模型的一个核心问题， 其输入和输出都是长度可变的序列。

  • 我们可以设计一个包含两个主要组件的架构：

  • 第一个组件是一个编码器（encoder）

    • 它接受一个长度可变的序列作为输入， 并将其转换为具有固定形状的编码状态。

    • 编码器通常由一系列的神经网络层组成，比如循环神经网络（RNN）、长短期记忆网络（LSTM）或卷积神经网络（CNN）。编码器负责处理输入数据，将其转换成一个固定维度的隐含向量，捕捉输入序列中的语义信息。

  • 第二个组件是解码器（decoder）

    • 它将固定形状的编码状态映射到长度可变的序列。 这被称为编码器-解码器（encoder-decoder）架构

    • 解码器接收来自编码器的隐含向量，并生成输出序列。解码器与编码器的结构类似，通常也是由RNN、LSTM或CNN构成。解码器的任务是从隐含表示中逐步生成目标序列

  • 隐含向量（Context Vector / Latent Vector）

    • 编码器将输入序列转换为一个固定长度的向量，称为上下文向量。这个向量包含了输入序列的语义信息，解码器利用它来生成目标序列。

  • 我们以英语到法语的机器翻译为例子

    • 比如说给定一个英文的输入序列，They are watching

    • 首先使用一个encoder和decoder，把这个输入的序列变成一个“状态”

    • 然后对这个状态进行解码，一个词元接着一个词元地生成翻译后的序列作为输出：IIs regordent

Encoder& Decoder

PreviousBidirectional Recurrent Neural Networks(BRNN)NextSeuqence to Sequence Learning(Seq2Seq)

Last updated 7 months ago