网站排名掉了该怎么做,枫泾网站建设,做淘宝差不多的网站,手机淘宝网页Qwen2-MOE-57B-A14B模型结构解读 模型代码文件下载
该模型总的参数为57B#xff0c;激活参数为14B#xff0c;推理速度比32B的快#xff0c;而且性能更好。
Qwen2-MOE-57B-A14B模型总体结构
class transformers.models.qwen2_moe.modeling_qwen2_moe.Qwen2MoeForCaus… Qwen2-MOE-57B-A14B模型结构解读
模型代码文件下载
该模型总的参数为57B激活参数为14B推理速度比32B的快而且性能更好。
Qwen2-MOE-57B-A14B模型总体结构
class transformers.models.qwen2_moe.modeling_qwen2_moe.Qwen2MoeForCausalLM
Qwen2MoeForCausalLM((model): Qwen2MoeModel((embed_tokens): Embedding(151936, 3584)(layers): ModuleList((0-27): 28 x Qwen2MoeDecoderLayer((self_attn): Qwen2MoeSdpaAttention((q_proj): Linear(in_features3584, out_features3584, biasTrue)(k_proj): Linear(in_features3584, out_features512, biasTrue)(v_proj): Linear(in_features3584, out_features512, biasTrue)(o_proj): Linear(in_features3584, out_features3584, biasFalse)(rotary_emb): Qwen2MoeRotaryEmbedding())(mlp): Qwen2MoeSparseMoeBlock((gate): Linear(in_features3584, out_features64, biasFalse)(experts): ModuleList((0-63): 64 x Qwen2MoeMLP((gate_proj): Linear(in_features3584, out_features2560, biasFalse)(up_proj): Linear(in_features3584, out_features2560, biasFalse)(down_proj): Linear(in_features2560, out_features3584, biasFalse)(act_fn): SiLU()))(shared_expert): Qwen2MoeMLP((gate_proj): Linear(in_features3584, out_features20480, biasFalse)(up_proj): Linear(in_features3584, out_features20480, biasFalse)(down_proj): Linear(in_features20480, out_features3584, biasFalse)(act_fn): SiLU())(shared_expert_gate): Linear(in_features3584, out_features1, biasFalse))(input_layernorm): Qwen2MoeRMSNorm()(post_attention_layernorm): Qwen2MoeRMSNorm()))(norm): Qwen2MoeRMSNorm())(lm_head): Linear(in_features3584, out_features151936, biasFalse)
)Qwen2-MOE-57B-A14B模型详细结构下面是从输入到输出的顺序输出的每层的参数量
#输入的Embedding层
model.embed_tokens.weight: torch.Size([151936, 3584])
#主体的layer层model.layers.0是第一层共有28层
#下面是model.layers.0的attention层
model.layers.0.self_attn.q_proj.weight: torch.Size([3584, 3584])
model.layers.0.self_attn.q_proj.bias: torch.Size([3584])
model.layers.0.self_attn.k_proj.weight: torch.Size([512, 3584])
model.layers.0.self_attn.k_proj.bias: torch.Size([512])
model.layers.0.self_attn.v_proj.weight: torch.Size([512, 3584])
model.layers.0.self_attn.v_proj.bias: torch.Size([512])
model.layers.0.self_attn.o_proj.weight: torch.Size([3584, 3584])
model.layers.0.mlp.gate.weight: torch.Size([64, 3584])#下面是model.layers.0的moe结构的mlp层
model.layers.0.mlp.experts.0.gate_proj.weight: torch.Size([2560, 3584])
model.layers.0.mlp.experts.0.up_proj.weight: torch.Size([2560, 3584])
model.layers.0.mlp.experts.0.down_proj.weight: torch.Size([3584, 2560])
model.layers.0.mlp.experts.1.gate_proj.weight: torch.Size([2560, 3584])
model.layers.0.mlp.experts.1.up_proj.weight: torch.Size([2560, 3584])
model.layers.0.mlp.experts.1.down_proj.weight: torch.Size([3584, 2560])
model.layers.0.mlp.experts.2.gate_proj.weight: torch.Size([2560, 3584])
model.layers.0.mlp.experts.2.up_proj.weight: torch.Size([2560, 3584])
model.layers.0.mlp.experts.2.down_proj.weight: torch.Size([3584, 2560])...有64个model.layers.0.mlp.experts层这里省略model.layers.0.mlp.experts.3----model.layers.0.mlp.experts.62model.layers.0.mlp.experts.63.gate_proj.weight: torch.Size([2560, 3584])
model.layers.0.mlp.experts.63.up_proj.weight: torch.Size([2560, 3584])
model.layers.0.mlp.experts.63.down_proj.weight: torch.Size([3584, 2560])#下面是model.layers.0的shared moe结构的mlp层
model.layers.0.mlp.shared_expert.gate_proj.weight: torch.Size([20480, 3584])
model.layers.0.mlp.shared_expert.up_proj.weight: torch.Size([20480, 3584])
model.layers.0.mlp.shared_expert.down_proj.weight: torch.Size([3584, 20480])
model.layers.0.mlp.shared_expert_gate.weight: torch.Size([1, 3584])#下面是是model.layers.0的Qwen2MoeRMSNorm层
model.layers.0.input_layernorm.weight: torch.Size([3584])
model.layers.0.post_attention_layernorm.weight: torch.Size([3584])...这里省略model.layers.1---model.layers.27它们的结构与model.layers.0一样#下面是马上要输出前的归一化norm层
model.norm.weight: torch.Size([3584])#下面是输出到最后的151936个token概率分布的mlp层
lm_head.weight: torch.Size([151936, 3584])
