{"id":5647522,"date":"2023-03-17T10:44:36","date_gmt":"2023-03-17T14:44:36","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5647522"},"modified":"2023-03-20T12:37:11","modified_gmt":"2023-03-20T16:37:11","slug":"training-compiled-pytorch-2.0-with-pytorch-lightning","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/","title":{"rendered":"Training Compiled PyTorch 2.0 with PyTorch Lightning"},"content":{"rendered":"<div class=\"takeaways card-glow p-4 my-4\"><h3 class=\"w-100 d-block\">Key Takeaways<\/h3>How to use PyTorch 2.0 and train a compiled model with PyTorch Lightning 2.0.<\/p>\n<p>Find the full code used in this tutorial <a href=\"https:\/\/gist.github.com\/aniketmaurya\/bdaf373eba4913065fd68cfbcfb4187e\">here<\/a>. <\/div>\n<h2>Higher Performance and as Pythonic as ever<\/h2>\n<p>PyTorch 2.0 aims to achieve higher levels of performance with <code>torch.compile<\/code>. This is done by taking advantage of the latest packages that perform optimizations at the compiler level. By using <code>torch.compile<\/code>, you can train your model 51% faster on average with AMP on an NVIDIA A100 GPU, according to an experiment with 163 open-source models.<\/p>\n<p>PyTorch 2.x aims to push the performance with model compilation further while maintaining its Pythonic nature and backward compatibility.<\/p>\n<div id=\"attachment_5647523\" style=\"width: 946px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-5647523\" class=\"size-full wp-image-5647523\" src=\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/bar-graph.png\" alt=\"\" width=\"936\" height=\"536\" srcset=\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/bar-graph.png 936w, https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/bar-graph-300x172.png 300w, https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/bar-graph-300x172@2x.png 600w\" sizes=\"(max-width: 936px) 100vw, 936px\" \/><p id=\"caption-attachment-5647523\" class=\"wp-caption-text\">Speedups for torch.compile against eager mode on an NVIDIA A100 GPU. Source: PyTorch<\/p><\/div>\n<p>&nbsp;<\/p>\n<h2>Lightning \u2764\ufe0f PyTorch 2.x<\/h2>\n<blockquote><p><span class=\"notion-enable-hover\" data-token-index=\"0\">\u201cPyTorch 2.0 embodies the future of deep learning frameworks. The possibility to capture a PyTorch program with effectively no user intervention and get massive on-device speedups and program manipulation out of the box unlocks a whole new dimension for AI developers.\u201d &#8211; Luca Antiga, CTO Lightning AI<\/span><\/p><\/blockquote>\n<p>This week, Lightning also launched <a href=\"https:\/\/lightning.ai\/pages\/open-source\/\">version 2.0 of PyTorch Lightning<\/a>, that is compatible with PyTorch 2.x. This version supports model compilation out of the box:<\/p>\n<pre class=\"code-shortcode dark-theme window- collapse-false \" style=\"--height:falsepx\"><code class=\"language-python\">\n\nimport lightning as L<br \/>\nimport torch\n\nclass LitModel(L.LightningModule):<br \/>\n    ...\n\nmodel = LitModel()<br \/>\ncompiled_model = torch.compile(model) # compiles the model and *step (training\/validation\/prediction)\n\ntrainer = L.Trainer()<br \/>\ntrainer.fit(compiled_model)\n\n<\/code><div class=\"copy-button\"><button class=\"expand-button\">Expand<\/button><button class=\"copy\">Copy<\/button><\/div><\/pre>\n<p>&nbsp;<\/p>\n<h2>Is PyTorch compile fast?<\/h2>\n<p>To confirm this, let&#8217;s compile a model and run some benchmarks. We&#8217;ll use a ResNet-50 and a ResNet-152 model from <span class=\"notion-enable-hover\" spellcheck=\"false\" data-token-index=\"1\">TorchVision<\/span>, and pass a tensor of size <span class=\"notion-enable-hover\" spellcheck=\"false\" data-token-index=\"3\">1x3x224x224<\/span> 10 times.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-5647527\" src=\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/compile-table-1.png\" alt=\"\" width=\"1650\" height=\"1100\" srcset=\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/compile-table-1.png 1650w, https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/compile-table-1-300x200.png 300w, https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/compile-table-1-1024x683.png 1024w, https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/compile-table-1-1536x1024.png 1536w, https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/compile-table-1-300x200@2x.png 600w\" sizes=\"(max-width: 1650px) 100vw, 1650px\" \/><\/p>\n<p>We observe that ResNet-151 had a performance gain of 74% and ResNet-50 gained 69.2%.<\/p>\n<p>What you can do with just a single line of change in your codebase is really impressive! \ud83e\udd2f <a href=\"https:\/\/github.com\/aniketmaurya\/deep-learning-examples\/blob\/main\/src\/pytorch-compile\/benchmark.py\">Here<\/a> is the code if you want to reproduce this.<\/p>\n<p>&nbsp;<\/p>\n<h2>Training a Compiled PyTorch LightningModule<\/h2>\n<p>We&#8217;ll train the ResNet model using the CIFAR-10 dataset to show how you can train a model with PyTorch 2.x compile feature. You can take this tutorial as an inspiration to build and train further compiled models with high performance.<\/p>\n<p>We&#8217;ll take the following steps:<\/p>\n<ol>\n<li>Load dataset<\/li>\n<li>Write training code using LightningModule<\/li>\n<li>Compile the LightningModule<\/li>\n<li>Train the model<\/li>\n<\/ol>\n<p>&nbsp;<\/p>\n<h3>Load CIFAR-10 dataset<\/h3>\n<p>First, let&#8217;s use <span class=\"notion-enable-hover\" spellcheck=\"false\" data-token-index=\"1\">torchvision<\/span> to load the CIFAR-10 dataset and build training and validation dataloaders:<\/p>\n<pre class=\"code-shortcode dark-theme window- collapse-false \" style=\"--height:falsepx\"><code class=\"language-python\">\n\nimport torch.nn.functional as F<br \/>\nimport torchvision<br \/>\nimport torchvision.transforms as transforms\n\ndef load_data():<br \/>\n    transform = transforms.Compose(<br \/>\n        [transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]<br \/>\n    )\n\n    batch_size = 32\n\n    train_set = torchvision.datasets.CIFAR10(<br \/>\n        root=\"~\/data\", train=True, download=True, transform=transform<br \/>\n    )<br \/>\n    train_loader = torch.utils.data.DataLoader(<br \/>\n        train_set, batch_size=batch_size, shuffle=True, num_workers=4<br \/>\n    )\n\n    val_set = torchvision.datasets.CIFAR10(<br \/>\n        root=\"~\/data\", train=False, download=True, transform=transform<br \/>\n    )<br \/>\n    val_loader = torch.utils.data.DataLoader(<br \/>\n        val_set, batch_size=batch_size, shuffle=False, num_workers=4<br \/>\n    )\n\n    return train_loader, val_loader\n\n<\/code><div class=\"copy-button\"><button class=\"expand-button\">Expand<\/button><button class=\"copy\">Copy<\/button><\/div><\/pre>\n<p>&nbsp;<\/p>\n<h3>Organize the Training Code with LightningModule<\/h3>\n<p>To train a ResNet18 model, we create a <a href=\"https:\/\/lightning.ai\/pages\/open-source\/pytorch-lightning\/\"><span class=\"notion-enable-hover\" spellcheck=\"false\" data-token-index=\"1\">LightningModule<\/span><\/a> and instantiate the model in the <span class=\"notion-enable-hover\" spellcheck=\"false\" data-token-index=\"3\">__init__<\/span> method. We then define the training and validation steps and configure our optimizer. Lightning automatically handles moving the data to the correct devices, calling the optimizer update, and zeroing out the gradients.<\/p>\n<pre class=\"code-shortcode dark-theme window- collapse-false \" style=\"--height:falsepx\"><code class=\"language-python\">\n\n# pip install lightning timm\n\nimport lightning as L<br \/>\nfrom timm import create_model\n\nclass LitModel(L.LightningModule):<br \/>\n    def __init__(self):<br \/>\n        super().__init__()<br \/>\n        self.model = create_model(\"resnet18\", num_classes=10)<br \/>\n        self.criterion = nn.CrossEntropyLoss()\n\n    def forward(self, x):<br \/>\n        return self.model(x)\n\n    def training_step(self, batch, batch_idx):<br \/>\n        x, y = batch<br \/>\n        logits = self(x)<br \/>\n        loss = self.criterion(logits, y)<br \/>\n        return loss\n\n    def validation_step(self, batch, batch_idx):<br \/>\n        x, y = batch<br \/>\n        logits = self(x)<br \/>\n        loss = self.criterion(logits, y)<br \/>\n        return loss\n\n    def configure_optimizers(self):<br \/>\n        return torch.optim.Adam(self.model.parameters(), lr=0.001)\n\n<\/code><div class=\"copy-button\"><button class=\"expand-button\">Expand<\/button><button class=\"copy\">Copy<\/button><\/div><\/pre>\n<p>&nbsp;<\/p>\n<h3>Compile our code<\/h3>\n<p>To compile the <code>LightningModule<\/code>, you simply call <code>torch.compile<\/code>. This will automatically compile the model and its *_steps (training\/validation). The <code>torch.compile<\/code> function has several arguments, but in this tutorial, we will only use the <code>mode<\/code> argument and leave the rest as default. To learn more about these arguments, <a href=\"https:\/\/pytorch.org\/get-started\/pytorch-2.0\/#user-experience\">visit the official docs<\/a>.<\/p>\n<p>There are three available modes: <code>default<\/code>, <code>reduce-overhead<\/code>, and <code>max-autotune<\/code>. During the benchmarks above, we tried both <code>default<\/code> and <code>reduce-overhead<\/code> and found that <code>reduce-overhead<\/code> leads to higher performance.<\/p>\n<pre class=\"code-shortcode dark-theme window- collapse-false \" style=\"--height:falsepx\"><code class=\"language-python\">\n\nimport torch\n\ncompiled_model = torch.compile(LitModel(), mode=\"reduce-overhead\")\n\n<\/code><div class=\"copy-button\"><button class=\"expand-button\">Expand<\/button><button class=\"copy\">Copy<\/button><\/div><\/pre>\n<p>&nbsp;<\/p>\n<h3>Train the compiled model<\/h3>\n<pre class=\"code-shortcode dark-theme window- collapse-false \" style=\"--height:falsepx\"><code class=\"language-python\">\n\ntrainer = L.Trainer(max_epochs=1)<br \/>\ntrainer.fit(model, train_loader)\n\n<\/code><div class=\"copy-button\"><button class=\"expand-button\">Expand<\/button><button class=\"copy\">Copy<\/button><\/div><\/pre>\n<p>The Lightning 2.0 <code>Trainer<\/code> automatically selects the appropriate accelerator (MPS\/Cuda\/TPUs) and the number of devices for you. You can provide additional flags to configure where and how to save your checkpoints, log metrics, and more. For more information about the <code>Trainer<\/code>, check out our <a href=\"https:\/\/lightning.ai\/docs\/pytorch\/latest\/common\/trainer.html\">docs<\/a>.<\/p>\n<p>If you are undertaking complex tasks like reinforcement learning or training an LLM and need more control over the training loop while leveraging PyTorch Lightning APIs for accelerators and distributed strategies then check out <a href=\"https:\/\/lightning.ai\/docs\/fabric\/stable\/\">Fabric<\/a>: a fast and lightweight way to scale PyTorch models.<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: center;\"><div class=\"takeaways card-glow p-4 my-4\"><h3 class=\"w-100 d-block\">Chat with us<\/h3> Lightning&#8217;s Discord server is a great place to meet members of our team as well as other open-source and AI enthusiasts.<\/p>\n<p style=\"text-align: center;\"><\/div>\n<p style=\"text-align: center;\"><a target=\"blank\" href=\"https:\/\/discord.gg\/tfXFetEZxv\" class=\"d-inline-block btn btn-\">Join the Lightning Discord!<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Higher Performance and as Pythonic as ever PyTorch 2.0 aims to achieve higher levels of performance with torch.compile. This is done by taking advantage of the latest packages that perform optimizations at the compiler level. By using torch.compile, you can train your model 51% faster on average with AMP on an NVIDIA A100 GPU, according<a class=\"excerpt-read-more\" href=\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/\" title=\"ReadTraining Compiled PyTorch 2.0 with PyTorch Lightning\">&#8230; Read more &raquo;<\/a><\/p>\n","protected":false},"author":16,"featured_media":5647526,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":"","_links_to":"","_links_to_target":""},"categories":[29,41],"tags":[96,97,51,176],"glossary":[],"acf":{"additional_authors":false,"hide_from_archive":false,"content_type":"Blog Post","sticky":false,"custom_styles":""},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Training Compiled PyTorch 2.0 with PyTorch Lightning<\/title>\n<meta name=\"description\" content=\"Learn how to use PyTorch 2.0 and train a compiled model with PyTorch Lightning 2.0.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Training Compiled PyTorch 2.0 with PyTorch Lightning\" \/>\n<meta property=\"og:description\" content=\"Learn how to use PyTorch 2.0 and train a compiled model with PyTorch Lightning 2.0.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/\" \/>\n<meta property=\"og:site_name\" content=\"Lightning AI\" \/>\n<meta property=\"article:published_time\" content=\"2023-03-17T14:44:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-03-20T16:37:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1160\" \/>\n\t<meta property=\"og:image:height\" content=\"600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"JP Hennessy\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@LightningAI\" \/>\n<meta name=\"twitter:site\" content=\"@LightningAI\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"JP Hennessy\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/\"},\"author\":{\"name\":\"JP Hennessy\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6\"},\"headline\":\"Training Compiled PyTorch 2.0 with PyTorch Lightning\",\"datePublished\":\"2023-03-17T14:44:36+00:00\",\"dateModified\":\"2023-03-20T16:37:11+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/\"},\"wordCount\":871,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#organization\"},\"image\":{\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png\",\"keywords\":[\"ai\",\"ml\",\"pytorch\",\"torch compile\"],\"articleSection\":[\"Blog\",\"Tutorials\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/\",\"url\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/\",\"name\":\"Training Compiled PyTorch 2.0 with PyTorch Lightning\",\"isPartOf\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png\",\"datePublished\":\"2023-03-17T14:44:36+00:00\",\"dateModified\":\"2023-03-20T16:37:11+00:00\",\"description\":\"Learn how to use PyTorch 2.0 and train a compiled model with PyTorch Lightning 2.0.\",\"breadcrumb\":{\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#primaryimage\",\"url\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png\",\"contentUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png\",\"width\":1160,\"height\":600},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/lightning.ai\/pages\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Training Compiled PyTorch 2.0 with PyTorch Lightning\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/lightning.ai\/pages\/#website\",\"url\":\"https:\/\/lightning.ai\/pages\/\",\"name\":\"Lightning AI\",\"description\":\"The platform for teams to build AI.\",\"publisher\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/lightning.ai\/pages\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/lightning.ai\/pages\/#organization\",\"name\":\"Lightning AI\",\"url\":\"https:\/\/lightning.ai\/pages\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png\",\"contentUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png\",\"width\":1744,\"height\":856,\"caption\":\"Lightning AI\"},\"image\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/LightningAI\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6\",\"name\":\"JP Hennessy\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g\",\"caption\":\"JP Hennessy\"},\"url\":\"https:\/\/lightning.ai\/pages\/author\/jplightning-ai\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Training Compiled PyTorch 2.0 with PyTorch Lightning","description":"Learn how to use PyTorch 2.0 and train a compiled model with PyTorch Lightning 2.0.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/","og_locale":"en_US","og_type":"article","og_title":"Training Compiled PyTorch 2.0 with PyTorch Lightning","og_description":"Learn how to use PyTorch 2.0 and train a compiled model with PyTorch Lightning 2.0.","og_url":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/","og_site_name":"Lightning AI","article_published_time":"2023-03-17T14:44:36+00:00","article_modified_time":"2023-03-20T16:37:11+00:00","og_image":[{"width":1160,"height":600,"url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png","type":"image\/png"}],"author":"JP Hennessy","twitter_card":"summary_large_image","twitter_creator":"@LightningAI","twitter_site":"@LightningAI","twitter_misc":{"Written by":"JP Hennessy","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#article","isPartOf":{"@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/"},"author":{"name":"JP Hennessy","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6"},"headline":"Training Compiled PyTorch 2.0 with PyTorch Lightning","datePublished":"2023-03-17T14:44:36+00:00","dateModified":"2023-03-20T16:37:11+00:00","mainEntityOfPage":{"@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/"},"wordCount":871,"commentCount":0,"publisher":{"@id":"https:\/\/lightning.ai\/pages\/#organization"},"image":{"@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#primaryimage"},"thumbnailUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png","keywords":["ai","ml","pytorch","torch compile"],"articleSection":["Blog","Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/","url":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/","name":"Training Compiled PyTorch 2.0 with PyTorch Lightning","isPartOf":{"@id":"https:\/\/lightning.ai\/pages\/#website"},"primaryImageOfPage":{"@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#primaryimage"},"image":{"@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#primaryimage"},"thumbnailUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png","datePublished":"2023-03-17T14:44:36+00:00","dateModified":"2023-03-20T16:37:11+00:00","description":"Learn how to use PyTorch 2.0 and train a compiled model with PyTorch Lightning 2.0.","breadcrumb":{"@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#primaryimage","url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png","contentUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/03\/FAQs-1.png","width":1160,"height":600},{"@type":"BreadcrumbList","@id":"https:\/\/lightning.ai\/pages\/blog\/training-compiled-pytorch-2.0-with-pytorch-lightning\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/lightning.ai\/pages\/"},{"@type":"ListItem","position":2,"name":"Training Compiled PyTorch 2.0 with PyTorch Lightning"}]},{"@type":"WebSite","@id":"https:\/\/lightning.ai\/pages\/#website","url":"https:\/\/lightning.ai\/pages\/","name":"Lightning AI","description":"The platform for teams to build AI.","publisher":{"@id":"https:\/\/lightning.ai\/pages\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/lightning.ai\/pages\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/lightning.ai\/pages\/#organization","name":"Lightning AI","url":"https:\/\/lightning.ai\/pages\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/","url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png","contentUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png","width":1744,"height":856,"caption":"Lightning AI"},"image":{"@id":"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/LightningAI"]},{"@type":"Person","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6","name":"JP Hennessy","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g","caption":"JP Hennessy"},"url":"https:\/\/lightning.ai\/pages\/author\/jplightning-ai\/"}]}},"_links":{"self":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/posts\/5647522"}],"collection":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/comments?post=5647522"}],"version-history":[{"count":0,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/posts\/5647522\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/media\/5647526"}],"wp:attachment":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/media?parent=5647522"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/categories?post=5647522"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/tags?post=5647522"},{"taxonomy":"glossary","embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/glossary?post=5647522"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}