thunder.distributed.FSDPBucketingStrategy

class thunder.distributed.FSDPBucketingStrategy(value)

Bases: Enum

Specify how we group parameters into a bucket for collective communication in fsdp.

NONE

Disables bucketing.

LAYER

Creates buckets per layer such as torch.nn.Linear and torch.nn.LayerNorm.

BLOCK

Creates buckets per block such as Transformer block.

__init__()

Attributes

NONE
LAYER
BLOCK