thunder.plugins.QuantizeInt4

class thunder.plugins.QuantizeInt4

Bases: Plugin

Plugin for 4-bit integer quantization using BitsAndBytes.

This plugin applies a 4-bit linear quantization transform to model weights, reducing memory footprint and improving throughput for both training and inference.

See https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/functional.py#L889 for more details.

__init__()

Methods

__init__()
setup_executors()	Fetches the BitsAndBytes quantization executor.
setup_lookasides()	rtype: Optional[list[Lookaside]]
setup_transforms()	Fetches the BitsAndBytes quantization transform.

Attributes

policy

setup_executors()

Fetches the BitsAndBytes quantization executor.

Returns:

A list containing the Transformer Engine executor.

Return type:

list[Executor]

setup_transforms()

Fetches the BitsAndBytes quantization transform.

Returns:

A list containing the Transformer Engine executor.

Return type:

list[Transform]