llm Configuration
This document provides an overview of all configuration classes in llm type.
Configuration Classes
| Class | Description | Documentation |
|---|---|---|
BaichuanDeployModelParameters | Baichuan Proxy LLM | View Details |
BitsandbytesQuantization | Bits and bytes quantization parameters. | View Details |
BitsandbytesQuantization4bits | Bits and bytes quantization 4 bits parameters. | View Details |
BitsandbytesQuantization8bits | Bits and bytes quantization 8 bits parameters. | View Details |
ClaudeDeployModelParameters | Claude Proxy LLM | View Details |
DeepSeekDeployModelParameters | Deepseek proxy LLM configuration. | View Details |
GeminiDeployModelParameters | Google Gemini proxy LLM configuration. | View Details |
GiteeDeployModelParameters | Gitee proxy LLM configuration. | View Details |
HFLLMDeployModelParameters | Local deploy model parameters. | View Details |
LlamaCppModelParameters | LlamaCppModelParameters(name: str, provider: str = 'llama.cpp', verbose: Optional[bool] = False, concurrency: Optional[int] = 5, backend: Optional[str] = None, prompt_template: Optional[str] = None, context_length: Optional[int] = None, reasoning_model: Optional[bool] = None, path: Optional[str] = None, device: Optional[str] = None, seed: Optional[int] = -1, n_threads: Optional[int] = None, n_batch: Optional[int] = 512, n_gpu_layers: Optional[int] = 1000000000, n_gqa: Optional[int] = None, rms_norm_eps: Optional[float] = 5e-06, cache_capacity: Optional[str] = None, prefer_cpu: Optional[bool] = False) | View Details |
LlamaServerParameters | LlamaServerParameters(name: str, provider: str = 'llama.cpp.server', verbose: Optional[bool] = False, concurrency: Optional[int] = 20, backend: Optional[str] = None, prompt_template: Optional[str] = None, context_length: Optional[int] = None, reasoning_model: Optional[bool] = None, path: Optional[str] = None, model_hf_repo: Optional[str] = None, model_hf_file: Optional[str] = None, device: Optional[str] = None, server_bin_path: Optional[str] = None, server_host: str = '127.0.0.1', server_port: int = 0, temperature: float = 0.8, seed: int = 42, debug: bool = False, model_url: Optional[str] = None, model_draft: Optional[str] = None, threads: Optional[int] = None, n_gpu_layers: Optional[int] = None, batch_size: Optional[int] = None, ubatch_size: Optional[int] = None, ctx_size: Optional[int] = None, grp_attn_n: Optional[int] = None, grp_attn_w: Optional[int] = None, n_predict: Optional[int] = None, slot_save_path: Optional[str] = None, n_slots: Optional[int] = None, cont_batching: bool = False, embedding: bool = False, reranking: bool = False, metrics: bool = False, slots: bool = False, draft: Optional[int] = None, draft_max: Optional[int] = None, draft_min: Optional[int] = None, api_key: Optional[str] = None, lora_files: List[str] = <factory>, no_context_shift: bool = False, no_webui: Optional[bool] = None, startup_timeout: Optional[int] = None) | View Details |
MoonshotDeployModelParameters | Moonshot proxy LLM configuration. | View Details |
OllamaDeployModelParameters | Ollama proxy LLM configuration. | View Details |
OpenAICompatibleDeployModelParameters | OpenAI Compatible Proxy LLM | View Details |
AimlapiDeployModelParameters | AI/ML API proxy LLM configuration. | View Details |
SiliconFlowDeployModelParameters | SiliconFlow proxy LLM configuration. | View Details |
SparkDeployModelParameters | Xunfei Spark proxy LLM configuration. | View Details |
TongyiDeployModelParameters | Tongyi proxy LLM configuration. | View Details |
VLLMDeployModelParameters | Local deploy model parameters. | View Details |
VolcengineDeployModelParameters | Volcengine proxy LLM configuration. | View Details |
WenxinDeployModelParameters | Baidu Wenxin proxy LLM configuration. | View Details |
YiDeployModelParameters | Yi proxy LLM configuration. | View Details |
ZhipuDeployModelParameters | Zhipu proxy LLM configuration. | View Details |