LLM USE FAQ
Q1:how to use openai chatgpt service
change your LLM_MODEL
LLM_MODEL=proxyllm
set your OPENAPI KEY
PROXY_API_KEY={your-openai-sk}
PROXY_SERVER_URL=https://api.openai.com/v1/chat/completions
make sure your openapi API_KEY is available
Q2 What difference between python dbgpt_server --light and python dbgpt_server
python dbgpt_server --lightdbgpt_server does not start the llm service. Users can deploy the llm service separately by usingpython llmserver`, and dbgpt_server accesses the llm service through set the LLM_SERVER environment variable in .env. The purpose is to allow for the separate deployment of dbgpt's backend service and llm service.
python dbgpt_server service and the llm service are deployed on the same instance. when dbgpt_server starts the service, it also starts the llm service at the same time.
Q3 how to use MultiGPUs
DB-GPT will use all available gpu by default. And you can modify the setting CUDA_VISIBLE_DEVICES=0,1 in .env file
to use the specific gpu IDs.
Optionally, you can also specify the gpu ID to use before the starting command, as shown below:
# Specify 1 gpu
CUDA_VISIBLE_DEVICES=0 python3 dbgpt/app/dbgpt_server.py
# Specify 4 gpus
CUDA_VISIBLE_DEVICES=3,4,5,6 python3 dbgpt/app/dbgpt_server.py
You can modify the setting MAX_GPU_MEMORY=xxGib in .env file to configure the maximum memory used by each GPU.
Q4 Not Enough Memory
DB-GPT supported 8-bit quantization and 4-bit quantization.
You can modify the setting QUANTIZE_8bit=True or QUANTIZE_4bit=True in .env file to use quantization(8-bit quantization is enabled by default).
Llama-2-70b with 8-bit quantization can run with 80 GB of VRAM, and 4-bit quantization can run with 48 GB of VRAM.
Note: you need to install the latest dependencies according to requirements.txt. Note: you need to install the latest dependencies according to requirements.txt.