\u672c\u5730\u90e8\u7f72\u5927\u8bed\u8a00\u6a21\u578b\uff08LLM\uff09\u8d8a\u6765\u8d8a\u6d41\u884c\u3002\u4f18\u52bf\u5305\u62ec\uff1a\u6570\u636e\u9690\u79c1\u3001\u65e0 API \u8d39\u7528\u3001\u79bb\u7ebf\u53ef\u7528\u3001\u53ef\u5b9a\u5236\u3002\u672c\u6587\u5c06\u8be6\u7ec6\u4ecb\u7ecd\u5982\u4f55\u5728\u672c\u5730\u90e8\u7f72\u548c\u4f18\u5316 LLM\u3002<\/p>\n
## \u4e3a\u4ec0\u4e48\u9009\u62e9\u672c\u5730\u90e8\u7f72\uff1f<\/p>\n
### \u4f18\u52bf
\n– **\u6570\u636e\u9690\u79c1\uff1a** \u6570\u636e\u4e0d\u79bb\u5f00\u4f60\u7684\u8bbe\u5907
\n– **\u96f6 API \u8d39\u7528\uff1a** \u4e00\u6b21\u90e8\u7f72\uff0c\u65e0\u9650\u4f7f\u7528
\n– **\u79bb\u7ebf\u53ef\u7528\uff1a** \u4e0d\u4f9d\u8d56\u7f51\u7edc
\n– **\u5b8c\u5168\u63a7\u5236\uff1a** \u53ef\u81ea\u5b9a\u4e49\u3001\u5fae\u8c03\u3001\u91cf\u5316
\n– **\u5ef6\u8fdf\u4f4e\uff1a** \u672c\u5730\u63a8\u7406\uff0c\u54cd\u5e94\u5feb\u901f<\/p>\n
### \u52a3\u52bf
\n– \u786c\u4ef6\u8981\u6c42\uff1a\u9700\u8981\u8f83\u597d\u7684 GPU
\n– \u6a21\u578b\u5927\u5c0f\uff1a\u672c\u5730\u5b58\u50a8\u5360\u7528
\n– \u63a8\u7406\u901f\u5ea6\uff1a\u53ef\u80fd\u4e0d\u5982\u4e91\u7aef\u670d\u52a1<\/p>\n
## \u786c\u4ef6\u8981\u6c42
\n– **\u6700\u4f4e\u914d\u7f6e\uff1a** CPU 8\u6838+\u3001\u5185\u5b58 16GB\u3001\u5b58\u50a8 50GB SSD\uff08GPU \u53ef\u9009\uff09
\n– **\u63a8\u8350\u914d\u7f6e\uff1a** CPU 16\u6838+\u3001\u5185\u5b58 32GB+\u3001\u5b58\u50a8 100GB+ NVMe SSD\u3001GPU RTX 3060+
\n– **\u9ad8\u7aef\u914d\u7f6e\uff1a** CPU 32\u6838+\u3001\u5185\u5b58 64GB+\u3001\u5b58\u50a8 200GB+\u3001GPU RTX 4090<\/p>\n
## \u6a21\u578b\u9009\u62e9<\/p>\n
### \u8f7b\u91cf\u7ea7\uff08\u9002\u5408 CPU \/ \u5c0f GPU\uff09
\n| \u6a21\u578b | \u53c2\u6570 | \u63a8\u8350\u7528\u9014 |
\n|——|——|———-|
\n| Llama-3.2-3B | 3B | \u5bf9\u8bdd\u3001\u7b80\u5355\u4efb\u52a1 |
\n| Phi-3-mini | 3.8B | \u901a\u7528\u3001\u63a8\u7406 |
\n| Qwen-2.5-3B | 3B | \u4e2d\u82f1\u53cc\u8bed |<\/p>\n
### \u4e2d\u7b49\u89c4\u6a21\uff08\u9700\u8981 GPU\uff09
\n| \u6a21\u578b | \u53c2\u6570 | \u63a8\u8350\u7528\u9014 |
\n|——|——|———-|
\n| Llama-3.1-8B | 8B | \u901a\u7528\u5bf9\u8bdd |
\n| Qwen-2.5-7B | 7B | \u4e2d\u82f1\u3001\u4ee3\u7801 |
\n| Mistral-7B | 7B | \u901a\u7528\u3001\u6307\u4ee4\u8ddf\u968f |
\n| Gemma-2-9B | 9B | \u591a\u4efb\u52a1 |<\/p>\n
### \u5927\u89c4\u6a21\uff08\u9700\u8981\u5927\u663e\u5b58\uff09
\n| \u6a21\u578b | \u53c2\u6570 | \u63a8\u8350\u7528\u9014 |
\n|——|——|———-|
\n| Llama-3.1-70B | 70B | \u9ad8\u8d28\u91cf\u751f\u6210 |
\n| Qwen-2.5-72B | 72B | \u4e2d\u82f1\u3001\u590d\u6742\u4efb\u52a1 |
\n| Mixtral-8x7B | 47B | MoE \u67b6\u6784 |<\/p>\n
## \u90e8\u7f72\u6846\u67b6<\/p>\n
### 1. Ollama\uff08\u6700\u7b80\u5355\uff09
\n“`bash
\n# \u5b89\u88c5
\ncurl -fsSL https:\/\/ollama.ai\/install.sh | sh<\/p>\n
# \u62c9\u53d6\u6a21\u578b
\nollama pull llama3.2<\/p>\n
# \u8fd0\u884c\u804a\u5929
\nollama run llama3.2<\/p>\n
# API \u670d\u52a1\uff08\u9ed8\u8ba4\u7aef\u53e3 11434\uff09
\nollama serve
\n“`<\/p>\n
API \u8c03\u7528\uff1a
\n“`python
\nimport ollama
\nresponse = ollama.chat(model=’llama3.2′, messages=[
\n {‘role’: ‘user’, ‘content’: ‘Why is the sky blue?’}
\n])
\nprint(response[‘message’][‘content’])
\n“`<\/p>\n
### 2. vLLM\uff08\u9ad8\u6027\u80fd\uff09
\n“`bash
\npip install vllm
\npython -m vllm.entrypoints.openai.api_server
\n –model Qwen\/Qwen2.5-7B-Instruct
\n –host 0.0.0.0 –port 8000
\n“`<\/p>\n
### 3. Text Generation WebUI\uff08\u56fe\u5f62\u754c\u9762\uff09
\n“`bash
\ngit clone https:\/\/github.com\/oobabooga\/text-generation-webui
\ncd text-generation-webui
\npip install -r requirements.txt
\npython server.py –model Qwen\/Qwen2.5-7B-Instruct
\n“`<\/p>\n
### 4. LocalAI\uff08OpenAI \u517c\u5bb9\uff09
\n“`bash
\ndocker run -p 8080:8080 -v \/path\/to\/models:\/models localai\/localai:latest
\n“`<\/p>\n
## \u91cf\u5316\u6280\u672f
\n| \u7cbe\u5ea6 | \u5927\u5c0f | \u8d28\u91cf | \u901f\u5ea6 |
\n|——|——|——|——|
\n| FP16 | 100% | \u6700\u597d | \u6162 |
\n| Q8_0 | 60% | \u5f88\u597d | \u8f83\u5feb |
\n| Q6_K | 50% | \u597d | \u5feb |
\n| Q4_K_M | 35% | \u53ef\u63a5\u53d7 | \u5f88\u5feb |
\n| Q3_K_S | 25% | \u8f83\u5dee | \u6700\u5feb |<\/p>\n
## \u591a\u6a21\u6001\u6a21\u578b
\n– **LLaVA\uff08\u89c6\u89c9+\u8bed\u8a00\uff09\uff1a** `ollama pull llava`
\n– **Qwen-VL\uff08\u591a\u6a21\u6001\uff09\uff1a** OpenAI \u517c\u5bb9 API<\/p>\n
## \u90e8\u7f72\u6700\u4f73\u5b9e\u8df5
\n– \u5bb9\u5668\u5316\uff1a\u4f7f\u7528 Docker \u4e00\u952e\u90e8\u7f72
\n– \u8d1f\u8f7d\u5747\u8861\uff1aNginx \u4ee3\u7406\u591a\u8282\u70b9
\n– \u76d1\u63a7\uff1aPrometheus + Grafana<\/p>\n","protected":false},"excerpt":{"rendered":"
\u672c\u5730\u90e8\u7f72\u5927\u8bed\u8a00\u6a21\u578b\uff08LLM\uff09\u8d8a\u6765\u8d8a\u6d41\u884c\u3002\u4f18\u52bf\u5305\u62ec\uff1a\u6570\u636e\u9690\u79c1\u3001\u65e0 API \u8d39\u7528\u3001\u79bb\u7ebf\u53ef\u7528\u3001\u53ef\u5b9a\u5236\u3002\u672c\u6587\u5c06\u8be6\u7ec6\u4ecb\u7ecd […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"zakra_page_container_layout":"customizer","zakra_page_sidebar_layout":"customizer","zakra_remove_content_margin":false,"zakra_sidebar":"customizer","zakra_transparent_header":"customizer","zakra_logo":0,"zakra_main_header_style":"default","zakra_menu_item_color":"","zakra_menu_item_hover_color":"","zakra_menu_item_active_color":"","zakra_menu_active_style":"","zakra_page_header":true,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-11","post","type-post","status-publish","format-standard","hentry","category-llm-"],"_links":{"self":[{"href":"https:\/\/gmrea.com\/index.php?rest_route=\/wp\/v2\/posts\/11","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gmrea.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gmrea.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/gmrea.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11"}],"version-history":[{"count":0,"href":"https:\/\/gmrea.com\/index.php?rest_route=\/wp\/v2\/posts\/11\/revisions"}],"wp:attachment":[{"href":"https:\/\/gmrea.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gmrea.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gmrea.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}