深圳网站建设 利科技有限公司,企业信用信息公示系统年报怎么填,济南网站建设免费,项目外包和人力外包的区别主流的LLM都需要通过CUDA才能高效的运行在本地#xff0c;但是随着Github上出现了Llama.cpp这个神器#xff0c;一切都改变了。它通过AVX指令和MPI来实现CPU上并行计算#xff0c;从而在本地计算机高效地运行各种主流的类Llama模型。同时它也支持metal#xff0c;使得Apple…主流的LLM都需要通过CUDA才能高效的运行在本地但是随着Github上出现了Llama.cpp这个神器一切都改变了。它通过AVX指令和MPI来实现CPU上并行计算从而在本地计算机高效地运行各种主流的类Llama模型。同时它也支持metal使得Apple Silicon的系统也能部署LLM。然而他的架构偏向于编译安装部署较为复杂于是衍生了Ollama之类的傻瓜式工具。而我要介绍的是Mozilla公司推出了Llamafile工具。
这个工具强大之处在于可以像Nvidia推出了一款 “Chat with RTX” ——原生支持Windows环境运行而Ollama只有Windows10和11的预览版但同时非常轻。
ModelSizeLicensellamafileLLaVA 1.53.97 GBLLaMA 2llava-v1.5-7b-q4.llamafileMistral-7B-Instruct5.15 GBApache 2.0mistral-7b-instruct-v0.2.Q5_K_M.llamafileMixtral-8x7B-Instruct30.03 GBApache 2.0mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafileWizardCoder-Python-34B22.23 GBLLaMA 2wizardcoder-python-34b-v1.0.Q5_K_M.llamafileWizardCoder-Python-13B7.33 GBLLaMA 2wizardcoder-python-13b.llamafileTinyLlama-1.1B0.76 GBApache 2.0TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafileRocket-3B1.89 GBcc-by-sa-4.0rocket-3b.Q5_K_M.llamafilePhi-21.96 GBMITphi-2.Q5_K_M.llamafile
支持的系统
Linux 2.6.18 (i.e. every distro since RHEL5 c. 2007)Darwin (macOS) 23.1.0 [1] (GPU is only supported on ARM64)Windows 8 (AMD64 only)FreeBSD 13NetBSD 9.2 (AMD64 only)OpenBSD 7 (AMD64 only) 演示
文本生成
./mistral-7b-instruct-v0.2.Q5_K_M.llamafile -ngl 9999 --temp 0.7 -p [INST]Write a story about llamas[/INST]代码生成
./wizardcoder-python-13b.llamafile -ngl 9999 --temp 0 -e -r \n -p c\nvoid *memcpy_sse2(char *dst, const char *src, size_t size) {\nVQA
./llava-v1.5-7b-q4.llamafile -ngl 9999 --temp 0.2 --image lemurs.jpg -e -p ### User: What do you see?\n### Assistant:WEB 服务
./mistral-7b-instruct-v0.2.Q5_K_M.llamafile -ngl 9999访问 http://localhost:8080
想要只用CPU运行只需要-ngl 0 或 --gpu disable
支持 Python版的OpenAI SDK
#!/usr/bin/env python3
from openai import OpenAI
client OpenAI(base_urlhttp://localhost:8080/v1, # http://Your api-server IP:portapi_key sk-no-key-required
)
completion client.chat.completions.create(modelLLaMA_CPP,messages[{role: system, content: You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.},{role: user, content: Write a limerick about python exceptions}]
)
print(completion.choices[0].message)或是 REST API请求
curl http://localhost:8080/v1/chat/completions \
-H Content-Type: application/json \
-H Authorization: Bearer no-key \
-d {model: LLaMA_CPP,messages: [{role: system,content: You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.},{role: user,content: Write a limerick about python exceptions}]
} | python3 -c
import json
import sys
json.dump(json.load(sys.stdin), sys.stdout, indent2)
print()’
存在问题
Windows的exe只能支持4GB大小所以5G以上需要使用github上的release文件和LLM的Guff文件。
原文 Unfortunately, Windows users cannot make use of many of these example llamafiles because Windows has a maximum executable file size of 4GB, and all of these examples exceed that size. (The LLaVA llamafile works on Windows because it is 30MB shy of the size limit.) But don’t lose heart: llamafile allows you to use external weights; this is described later in this document.
我在WSL中运行会被杀毒误杀 在Windows环境中GPU运行会乱码输出但是CPU运行能正常显示尚不清楚原因。
