When you’re ready to train your models at scale, you can use Runs. A Run is a collection of experiments.<\/span><\/p>\n Runs allow you to scale your machine learning code to hundreds of GPUs and model configurations without needing to change a single line of code. Grid Runs support all major machine learning frameworks, enabling full hyperparameter sweeps, native logging, artifacts, and Spot Instances all out of the box without the need to modify a single line of machine learning code.<\/span><\/p>\n Runs are “serverless”, which means that you only pay for the time your scripts are actually running. When running on your own infrastructure, this results in massive cost savings.<\/span><\/p>\n Grid Runs respect the use of .ignore files, which are used to tell a program which files it should ignore during execution. Grid gives preference to the .gridignore file. In the absence of a .gridignore file, Grid will concatenate the .gitignore and .dockerignore files to determine which files should be ignored. When creating a run, you do not have to provide these files to the CLI or UI – they are just expected to reside in the project root directory.<\/span><\/p>\n Note: <\/span><\/i>The examples used in this tutorial assume you have already installed and set up Grid. If you haven’t done this already, please visit <\/span><\/i>The First TIme You Use Grid<\/span><\/i><\/a> to learn more.\u00a0<\/span><\/i><\/p>\n Runs are customizable and provide serverless compute. Here, we cover all available methods to customize Runs for any specific use case. The examples in this tutorial cover the following:<\/span><\/p>\n A “vanilla” Run is a basic Run that only executes a script. This hello_world <\/span>repo<\/span><\/a> will be used in the following example.<\/span><\/p>\n The above code is passing a script named hello.py to the Run. The script will print out ‘hello_world’.<\/span><\/p>\n For instructions on how to view logs, check out <\/span>viewing logs produced by Runs<\/span><\/a>.<\/span><\/p>\n If you\u2019ve taken a look at the grid-tutorials repo, you may have noticed three things:<\/span><\/p>\n Let\u2019s quickly discuss how Grid handles requirements before touching on each of these topics.<\/span><\/p>\n Handling Requirements<\/b><\/p>\n Any time you create a Run, Grid attempts to resolve as many dependencies as it can automatically for you. Nested requirements are not currently supported.<\/span><\/p>\n We do, however, recommend that your projects have a <\/span> Runs with Specified requirements.txt<\/b><\/p>\n Runs allow you to specify which requirements.txt you want to use for package installation. This is especially useful when your directory isn’t ordered in such a way that the In these cases, you can use the below example as a template for specifying which You may have noticed that we did something different here than in prior examples: we used the <\/span> Runs with Specified environment.yml<\/b><\/p>\n Runs allow you to specify the When running on a non-Linux machine, we recommend using <\/span> You can use the example below as a template for specifying which To speed up training iteration time, you can store your data in a Grid Datastore. Datastores are high-performance, low-latency, versioned datasets. If you have large-scale data, Datastores can resolve blockers in your workflow by eliminating the need to download the large dataset every time your script runs.\u00a0<\/span><\/p>\n If you haven’t done so already, create a Datastore from the cifar5 dataset using the following commands:<\/span><\/p>\n Now let’s mount this Datastore to a Run:<\/span><\/p>\n This code passes a script named datastore.py to the Run. The script prints the contents of the Datastore to the root directory. You should see the following output in your stdout logs:<\/span><\/p>\n Interruptible Runs<\/span><\/a> powered by spot instances are 50-90% cheaper than on-demand instances, but they can be interrupted at any time if the machine gets taken away. Here is how you launch a Run with spot instances:<\/span><\/p>\n What happens to your models if the Run gets interrupted?\u00a0<\/span><\/i><\/p>\n Grid keeps all the artifacts that you saved during training, including logs, checkpoints and other files. This means that if you write your training script such that it periodically saves checkpoint files with all the states needed to resume your training, you can restart the Grid Run from where it was interrupted:<\/span><\/p>\n Writing the logic for checkpointing and resuming the training loop correctly, however, can be difficult and time consuming.\u00a0<\/span><\/p>\n PyTorch Lightning<\/span><\/a> removes the need to write all this boilerplate code. In fact, if you implement your training script with PyTorch Lightning, you will have to change <\/span>zero<\/b> lines of code to use interruptible Runs in Grid. All you have to do is add the <\/span> If this Run gets interrupted, PyTorch Lightning will save a fault-tolerant checkpoint automatically, Grid will collect it, provision a new machine, restart the Run for you and let PyTorch Lightning restore the training state where it left off. Mind-blowing! Learn more about <\/span>auto-resuming experiments in Grid<\/span><\/a> or the <\/span>fault-tolerance feature in PyTorch Lightning<\/span><\/a>.<\/span><\/p>\n And that\u2019s it! You can check out other Grid tutorials, or browse the <\/span>Grid Docs<\/span><\/a> to learn more about anything not covered in this tutorial.<\/span><\/p>\nHow to Create Runs<\/h3>\n
\n
\n
Creating Vanilla Runs<\/h4>\n
git clone https:\/\/github.com\/PyTorchLightning\/grid-tutorials.git\r\ncd features-intro\/runs\r\ngrid run --name hello hello.py<\/pre>\n
Creating Runs with Script Dependencies<\/h4>\n
\n
requirements.txt<\/code> file in the root.<\/span><\/p>\nrequirements.txt<\/code> resides at the root project level, or if you have more than one requirements.txt<\/code> file.\u00a0<\/span><\/p>\nrequirements.txt<\/code> file should be used for package installation.<\/span><\/p>\ngit clone https:\/\/github.com\/PyTorchLightning\/grid-tutorials.git\r\ncd features-intro\/runs\r\ngrid run --name specified-requirements-pip --dependency_file .\/pip\/requirements.txt hello.py<\/pre>\n
--dependency_file<\/code> flag. This flag tells Grid what file should be used for package installation in the Run.<\/span><\/p>\nenvironment.yml<\/code> you want to use for package installation. This is the only way to get Runs to use the Conda package manager without using a config file.\u00a0<\/span><\/p>\nconda env export --from-history<\/code> before creating a Run, as mentioned in the <\/span>official Conda documentation<\/span><\/a>. This is because <\/span>conda export<\/code> will output dependencies specifically for your operating system.\u00a0<\/span><\/p>\nenvironment.yml<\/code> file should be used for package installation:<\/span><\/p>\ngit clone <\/span>https:\/\/github.com\/PyTorchLightning\/grid-tutorials.git<\/span><\/a>\r\ncd features-intro\/runs<\/span>\r\ngrid run --name specified-requirements-conda --dependency_file .\/conda\/environemnt.yml hello.py<\/span><\/pre>\nAttaching Datastores to Runs<\/h4>\n
# download<\/span>\r\ncurl https:\/\/pl-flash-data.s3.amazonaws.com\/cifar5.zip -o cifar5.zip<\/span>\r\n# unzip<\/span>\r\nunzip cifar5.zip<\/span>\r\ngrid datastore create cifar5\/ --name cifar5<\/span><\/pre>\ngit clone https:\/\/github.com\/PyTorchLightning\/grid-tutorials.git\r\ncd features-intro\/runs\r\ngrid run --name attaching-datastore --datastore_name cifar5 --datastore_version 1 datastore.py --data_dir \/datastores\/cifar5\/1<\/pre>\n
['test', 'train']<\/span><\/pre>\nInterruptible Runs\u200b<\/b><\/a><\/h4>\n
grid run --use_spot train.py<\/pre>\n
grid run --use_spot train.py --checkpoint_path \"https:\/\/grid.ai\/url\/to\/ckpt\"<\/pre>\n
--auto-resume<\/code> flag to the <\/span>grid run<\/code> command and to make your experiments fault-tolerant:<\/span><\/p>\ngrid run --use_spot --auto_resume train.py<\/pre>\n