Distributed
ddp— Thunder’s Distributed Data Parallel.fsdp— Convert model into Fully Sharded Data Parallel.FSDPType— Specifies the sharding strategy to be used for FSDP in Thunder.FSDPBucketingStrategy— Specify how we group parameters into a bucket for collective communication in fsdp.set_skip_data_parallel_grad_sync— Set whether to skip data parallel grad sync.reset_skip_data_parallel_grad_sync— Reset whether to skip data parallel grad sync.get_skip_data_parallel_grad_sync— Get whether to skip data parallel grad sync.skip_data_parallel_grad_sync— A context manager to skip data parallel grad sync.column_parallel— Convert specified modules into column-wise parallel ones.row_parallel— Convert specified modules into row-wise parallel ones.
Was this page helpful?
Thanks for the feedback.