update readme
parent
6f00fc7954
commit
bb0ec7a420
|
|
@ -311,7 +311,7 @@ GPT模型根据现有token预测输出下一个下下一个下下下一个token
|
|||
和LlaVA、Qwen-VL等视觉语言模型类似,MiniMind-V同样选用开源Clip系列模型作为Visual Encoder。
|
||||
具体使用[clip-vit-base-patch16](https://huggingface.co/openai/clip-vit-base-patch16),
|
||||
一种基于 ViT-B/16 架构的经典Visual Encoder用于描述图像文本信息。
|
||||
输入的图像尺寸为224x224,因为划分的Patch是16×16,所以会产生16*16=196个token作为encoder编码层的输入,
|
||||
输入的图像尺寸为224x224,因为划分的Patch是16×16,所以会产生14*14=196个token作为encoder编码层的输入,
|
||||
最终产生1×768维的嵌入向量用于和文本对计算误差。
|
||||
我们并不需要最终嵌入表示,因此只取encoder层的输出,也就是VIT核心主干的输出特征即可。
|
||||
它拿到前一层维度196×768大小的特征,我们把它作为196个visual token输入MiniMind-V。
|
||||
|
|
|
|||
|
|
@ -338,7 +338,7 @@ Like LlaVA, Qwen-VL, and other visual language models, MiniMind-V also uses the
|
|||
Visual Encoder.
|
||||
Specifically, we use [clip-vit-base-patch16](https://huggingface.co/openai/clip-vit-base-patch16), a classic Visual
|
||||
Encoder based on the ViT-B/16 architecture for describing image-text information.
|
||||
The input image size is 224x224, and because the Patch size is 16×16, it generates 16*16=196 tokens as the input to the
|
||||
The input image size is 224x224, and because the Patch size is 16×16, it generates 14*14=196 tokens as the input to the
|
||||
encoder layer, which produces a 1×768 dimensional embedding vector for calculating error with the text.
|
||||
We don’t need the final embedding representation, so we only take the output from the encoder layer, which is the output
|
||||
feature from the core ViT backbone.
|
||||
|
|
|
|||
Loading…
Reference in New Issue