Minimal fabric project with config file, resume and experiment tracker?

vincentme · December 19, 2023, 8:24am

Hi

I am new to fabric and would like to start a new project with some example code. I noticed the official example, but would like to have one with more features (config file, resume, experiment tracker, etc…).

Could you point out one? Thanks

github.com

Lightning-AI/pytorch-lightning/blob/master/examples/fabric/image_classifier/train_fabric.py

# Copyright The Lightning AI team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Here are 4 easy steps to use Fabric in your PyTorch code.

1. Create the Lightning Fabric object at the beginning of your script.

2. Remove all ``.to`` and ``.cuda`` calls since Fabric will take care of it.

3. Apply ``setup`` over each model and optimizers pair, ``setup_dataloaders`` on all your dataloaders,

This file has been truncated. show original

awaelchli · December 23, 2023, 7:34pm

Hey there

One example I can give is from our Lit-GPT repository, it uses Fabric everywhere. Here is an example script that uses a lot of different features, including experiment tracking (TB or WandB):

github.com

Lightning-AI/lit-gpt/blob/main/pretrain/tinyllama.py

"""
This script is adapted from TinyLlama:
https://github.com/jzhang38/TinyLlama/blob/main/pretrain/tinyllama.py
"""

import math
import os
import sys
import time
from functools import partial
from pathlib import Path
from typing import Tuple, Union

import lightning as L
import torch
import torch.nn as nn
from lightning.fabric.loggers import CSVLogger, TensorBoardLogger
from lightning.fabric.strategies import FSDPStrategy
from lightning.fabric.utilities.throughput import ThroughputMonitor, measure_flops
from lightning.pytorch.loggers import WandbLogger

This file has been truncated. show original

It also also has all the logic to properly resume training from any checkpoint. I am confident that you can take a lot of this and just apply it to your setting very easily.

Let me know how it goes.

Happy holidays

vincentme · December 25, 2023, 3:18am

Thank you for pointing this project. It is very helpful!