bitsandbytes is an open-source PyTorch library that shrinks large language models with 8-bit and 4-bit quantisation so they need far less memory.
In plain words
Model weights are usually stored as 16-bit numbers. bitsandbytes stores them in 8 or 4 bits instead. Its 8-bit mode, LLM.int8(), roughly halves inference memory, keeping a few unusual large values in 16-bit to protect quality. Its 4-bit mode powers QLoRA, which trains small LoRA add-ons on top of a 4-bit model. It also offers 8-bit optimisers that cut training memory.
A quick example
Imagine a model that is too big for your GPU in 16-bit. Loading it in 4-bit with bitsandbytes lets it fit, and you can then fine-tune it with QLoRA.
Where you'll meet it
- Loading big models on smaller GPUs.
- QLoRA fine-tuning.
Related
Sources
- Docsbitsandbytes README, bitsandbytes Foundation (GitHub) · read 29 Sept 2026
- Docsbitsandbytes, Hugging Face docs · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.