From b1744b7858453fc3de4dc8302d68fc4465bf8d2f Mon Sep 17 00:00:00 2001 From: bglglzd Date: Sun, 12 Jul 2026 22:42:59 +0300 Subject: [PATCH] Fix typo in minimind-v --- CODE_OF_CONDUCT.md | 2 +- README.md | 8 ++++---- README_en.md | 36 ++++++++++++++++++------------------ model/model_init.md | 2 +- model/tokenizer.json | 2 +- model/tokenizer_config.json | 2 +- 6 files changed, 26 insertions(+), 26 deletions(-) diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index f95d690..18c9147 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -125,4 +125,4 @@ enforcement ladder](https://github.com/mozilla/diversity). For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at -https://www.contributor-covenant.org/translations. \ No newline at end of file +https://www.contributor-covenant.org/translations. diff --git a/README.md b/README.md index 27810ab..28364ca 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ -
+
2026-04-01 - 新增 minimind-3v (67M) 和 minimind-3v-moe (201M-A67M) 模型 @@ -97,7 +97,7 @@
-
+
2025-10-24 - bug修复：模型权重不对应 @@ -623,9 +623,9 @@ MoE 版本的场景描述更丰富，对背景环境（城市街道、城市天 ## 😊鸣谢 -@xinyanghuang7: 多图vlm分支 | 仓库截至此版本提供 +@xinyanghuang7: 多图vlm分支 | 仓库截至此版本提供 -
+
参考链接 & 感谢以下优秀的论文或项目 - 排名不分任何先后顺序 diff --git a/README_en.md b/README_en.md index 40fe602..b4fdbaa 100644 --- a/README_en.md +++ b/README_en.md @@ -86,7 +86,7 @@ Is the training process difficult? Now, let's explore the answers and feel the j
-
+
2026-04-01 - Added minimind-3v (67M) and minimind-3v-moe (201M-A67M) models @@ -99,7 +99,7 @@ Is the training process difficult? Now, let's explore the answers and feel the j
-
+
2025-10-24 - Bug fix: model weights mismatch @@ -357,21 +357,21 @@ Let's take a moment to think about two questions: * What is a **Large Language Model (LLM)**? * What is a multimodal model? -[This article](https://www.jiqizhixin.com/articles/2024-09-15-3) perfectly aligns with my thoughts: +[This article](https://www.jiqizhixin.com/articles/2024-09-15-3) perfectly aligns with my thoughts: Although the name "large language model" (LLM) contains the word "language," they are actually not closely related to language; this is just a historical issue. A more accurate name would be self-regressive Transformer or something else. LLMs are more of a general statistical modeling technology, mainly using a self-regressive Transformer to simulate token -flows. These tokens can represent text, images, audio, action choices, and even molecules—anything, really. +flows. These tokens can represent text, images, audio, action choices, and even molecules—anything, really. Therefore, as long as the problem can be converted into a process of simulating a series of discrete tokens, LLM can theoretically solve it. In fact, with the increasing maturity of large language model technologies, we may see more and more problems falling under this modeling paradigm. In other words, the problem is fixed in using LLM to "predict the next token," but the role and meaning of the tokens differ in each domain. -[ZJU-LiXi](https://person.zju.edu.cn/xilics#694283) has also mentioned a similar viewpoint (roughly stated below): +[ZJU-LiXi](https://person.zju.edu.cn/xilics#694283) has also mentioned a similar viewpoint (roughly stated below): Text, video, audio, actions, etc., are considered "multimodal" signals in human perception, but the term "modality" is essentially just a classification concept based on how humans store information. Just like `.txt` and `.png` files, though they differ in visual presentation and higher-level forms, they are fundamentally the same. The concept of " -multimodal" arose simply because humans need to categorize these signals based on different sensory dimensions. +multimodal" arose simply because humans need to categorize these signals based on different sensory dimensions. However, for machines, regardless of the signal's "modality," they are ultimately presented as a sequence of binary " monomodal" numbers. Machines do not differentiate the origin of these signals; they just process and analyze the information contained within these sequences. @@ -398,15 +398,15 @@ outputs the end token; here, the "token" doesn’t necessarily have to be text! 2. Fine-tune the LLM so that it and the **"foreign language dictionary"** go through a period of adaptation, thereby better understanding images. -The "foreign language dictionary" is referred to as the Visual Encoder model. +The "foreign language dictionary" is referred to as the Visual Encoder model. Like LlaVA, Qwen-VL, and other visual language models, MiniMind-V now uses the open-source SigLIP2 series models as the -Visual Encoder. +Visual Encoder. Specifically, we use [siglip2-base-p32-256-ve](https://huggingface.co/jingyaogong/siglip2-base-p32-256-ve), a Visual -Encoder based on the ViT-B/32 architecture for describing image-text information. +Encoder based on the ViT-B/32 architecture for describing image-text information. The current SigLIP2 NaFlex vision encoder generates 64 patch tokens (256×256 image / patch_size 32 = 8×8 = 64) from the processor output as the input to the -encoder layer, which produces a 1×768 dimensional embedding vector for calculating error with the text. +encoder layer, which produces a 1×768 dimensional embedding vector for calculating error with the text. We don't need the final embedding representation, so we only take the output from the encoder layer, which is the output -feature from the core ViT backbone. +feature from the core ViT backbone. It receives 64×768 features from the previous layer, which are projected to the LLM's hidden dimension via LayerNorm + a 2-layer MLP (Linear→GELU→Linear), resulting in 64 visual tokens fed into MiniMind-V — this step is exactly cross-modal feature alignment: the native visual features are brought into the semantic space where text tokens live, so that the two can interact in the same space. [LlaVA-1](https://arxiv.org/pdf/2304.08485) achieves good alignment with a simple linear transformation, [LlaVA-1.5](https://arxiv.org/pdf/2310.03744) upgrades to a 2-layer MLP. MiniMind-V adopts the same MLP Projection approach as LlaVA-1.5 (P32 natively outputs 64 tokens, no additional reshape compression needed). @@ -422,16 +422,16 @@ With that, the internal structural changes of MiniMind-V are now fully presented Next, let's briefly discuss the changes in the external input and output of MiniMind-V. -The input to the VLM is still a segment of text containing special `` placeholders. +The input to the VLM is still a segment of text containing special `` placeholders. After computing the text embedding, the vector generated by the image encoder can be projected onto the corresponding -embedding part of the placeholder, replacing the original placeholder embedding. +embedding part of the placeholder, replacing the original placeholder embedding. For example: ```text \nWhat is in this image? ``` -In `minimind-v`, the image is replaced by 64 `<|image_pad|>` tokens as placeholder (SigLIP2 P32 directly outputs 64 patch tokens, projected to 64 visual tokens via MLP), +In `minimind-v`, the image is replaced by 64 `<|image_pad|>` tokens as placeholder (SigLIP2 P32 directly outputs 64 patch tokens, projected to 64 visual tokens via MLP), thus the `minimind-v` prompt becomes: ```text @@ -525,10 +525,10 @@ We therefore use `--freeze_llm 1`: **only the Projection and the first & last LL On a single NVIDIA 3090, SFT takes ~2 hours per `epoch` in practice; dense and MoE finish in similar time (activated parameters are on the same order, with the gap mostly coming from the extra memory traffic of expert routing). Pretrain data volume is ~45% of SFT's, so one Pretrain epoch can be roughly scaled by that ratio. At a typical cloud price of ~1.5 RMB/hour for a 3090, a full SFT round costs about 3 RMB. -Pretrain [768+8] (dense & moe) +Pretrain [768+8] (dense & moe) ![input](./images/pretrain_loss.jpg) -SFT [768+8] (dense & moe) +SFT [768+8] (dense & moe) ![input](./images/sft_loss.jpg) ## Ⅲ Model Weights @@ -637,9 +637,9 @@ Visual signals act as a special "foreign language" to the LLM, so the ceiling of ## 😊 Acknowledgments -@xinyanghuang7: Multi-image VLM branch | Repository provided up to this version +@xinyanghuang7: Multi-image VLM branch | Repository provided up to this version -
+
Reference Links & Thanks to the following excellent papers or projects - No particular order diff --git a/model/model_init.md b/model/model_init.md index a558519..3ac0a29 100644 --- a/model/model_init.md +++ b/model/model_init.md @@ -25,4 +25,4 @@ model/ ├── model_vlm.py # VLM structure ├── tokenizer.json └── tokenizer_config.json -``` \ No newline at end of file +``` diff --git a/model/tokenizer.json b/model/tokenizer.json index e6ca86c..1afdc17 100644 --- a/model/tokenizer.json +++ b/model/tokenizer.json @@ -31188,4 +31188,4 @@ ] ] } -} \ No newline at end of file +} diff --git a/model/tokenizer_config.json b/model/tokenizer_config.json index e46c14a..d37b5e8 100644 --- a/model/tokenizer_config.json +++ b/model/tokenizer_config.json @@ -332,4 +332,4 @@ "audio_eos_token": "<|audio_end|>", "chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0].role == 'system' %}\n {{- messages[0].content + '\\n\\n' }}\n {%- endif %}\n {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within XML tags:\\n\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n\\n\\nFor each function call, return a json object with function name and arguments within XML tags:\\n\\n{\\\"name\\\": , \\\"arguments\\\": }\\n<|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0].role == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == \"user\" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n {%- if message.content is string %}\n {%- set content = message.content %}\n {%- else %}\n {%- set content = '' %}\n {%- endif %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {%- set reasoning_content = '' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if '' in content %}\n {%- set reasoning_content = content.split('')[0].rstrip('\\n').split('')[-1].lstrip('\\n') %}\n {%- set content = content.split('')[-1].lstrip('\\n') %}\n {%- endif %}\n {%- endif %}\n {%- if true %}\n {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }}\n {%- endif %}\n {%- if message.tool_calls %}\n {%- for tool_call in message.tool_calls %}\n {%- if (loop.first and content) or (not loop.first) %}\n {{- '\\n' }}\n {%- endif %}\n {%- if tool_call.function %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {%- if tool_call.arguments is string %}\n {{- tool_call.arguments }}\n {%- else %}\n {{- tool_call.arguments | tojson }}\n {%- endif %}\n {{- '}\\n' }}\n {%- endfor %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n\\n' }}\n {{- content }}\n {{- '\\n' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- if open_thinking is defined and open_thinking is true %}\n {{- '\\n' }}\n {%- else %}\n {{- '\\n\\n\\n\\n' }}\n {%- endif %}\n{%- endif %}", "tokenizer_class": "PreTrainedTokenizerFast" -} \ No newline at end of file +}