BitsandbytesQuantization Configuration
Bits and bytes quantization parameters.
Parameters
| Name | Type | Required | Description |
|---|---|---|---|
load_in_8bits | boolean | ❌ | Whether to load the model in 8 bits(LLM.int8() algorithm), default is False. Defaults： False |
load_in_4bits | boolean | ❌ | Whether to load the model in 4 bits, default is False. Defaults： False |