{"id":5646308,"date":"2021-05-29T10:31:19","date_gmt":"2021-05-29T14:31:19","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5646308"},"modified":"2022-07-25T10:32:05","modified_gmt":"2022-07-25T14:32:05","slug":"exploding-and-vanishing-gradients","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/","title":{"rendered":"Exploding And Vanishing Gradients in Lightning"},"content":{"rendered":"<h2>How Lightning Identifies Exploding and Vanishing Gradients<\/h2>\n<p>Lightning is a lightweight PyTorch wrapper for high-performance AI research that reduces the boilerplate without limiting flexibility. In this series, we are covering all the tricks Lightning offers to supercharge your machine learning training.<\/p>\n<p>Training very deep networks can make your derivatives get very small or very large quickly. This problem is referred to as vanishing or exploding gradients, which makes training unstable. In this video we introduce two flags, <code>track_grad_norm<\/code> to identify vanishing and exploding gradients, and <code>gradient_clip_val<\/code>, which will clip the gradient norm computed over all model parameters together.<\/p>\n<p>Follow along with this notebook: <a href=\"https:\/\/bit.ly\/33YzC1P\" target=\"_blank\" rel=\"noopener\"><u>https:\/\/bit.ly\/33YzC1P<\/u><\/a><\/p>\n<p><iframe loading=\"lazy\" title=\"YouTube video player\" src=\"https:\/\/www.youtube.com\/embed\/YZ-vJ2phDCo\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How Lightning Identifies Exploding and Vanishing Gradients Lightning is a lightweight PyTorch wrapper for high-performance AI research that reduces the boilerplate without limiting flexibility. In this series, we are covering all the tricks Lightning offers to supercharge your machine learning training. Training very deep networks can make your derivatives get very small or very large<a class=\"excerpt-read-more\" href=\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/\" title=\"ReadExploding And Vanishing Gradients in Lightning\">&#8230; Read more &raquo;<\/a><\/p>\n","protected":false},"author":16,"featured_media":5646309,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":"","_links_to":"","_links_to_target":""},"categories":[41],"tags":[65,61,51,62],"glossary":[],"acf":{"hide_from_archive":null,"content_type":null,"code_embed":null,"code_shortcode":null,"custom_styles":null,"sticky":null,"additional_authors":null,"mathjax":null,"default_editor":null,"sections":null,"show_table_of_contents":null,"table_of_contents":null,"tabs":null,"tab_group":null},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Exploding And Vanishing Gradients in Lightning - Lightning AI<\/title>\n<meta name=\"description\" content=\"In this video, learn how to let Lightning help you identify exploding and vanishing gradients to make your training more stable.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Exploding And Vanishing Gradients in Lightning - Lightning AI\" \/>\n<meta property=\"og:description\" content=\"In this video, learn how to let Lightning help you identify exploding and vanishing gradients to make your training more stable.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/\" \/>\n<meta property=\"og:site_name\" content=\"Lightning AI\" \/>\n<meta property=\"article:published_time\" content=\"2021-05-29T14:31:19+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-07-25T14:32:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"JP Hennessy\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@LightningAI\" \/>\n<meta name=\"twitter:site\" content=\"@LightningAI\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"JP Hennessy\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/\"},\"author\":{\"name\":\"JP Hennessy\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6\"},\"headline\":\"Exploding And Vanishing Gradients in Lightning\",\"datePublished\":\"2021-05-29T14:31:19+00:00\",\"dateModified\":\"2022-07-25T14:32:05+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/\"},\"wordCount\":112,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#organization\"},\"image\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png\",\"keywords\":[\"gradients\",\"lightning\",\"pytorch\",\"pytorch lightning\"],\"articleSection\":[\"Tutorials\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/\",\"url\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/\",\"name\":\"Exploding And Vanishing Gradients in Lightning - Lightning AI\",\"isPartOf\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png\",\"datePublished\":\"2021-05-29T14:31:19+00:00\",\"dateModified\":\"2022-07-25T14:32:05+00:00\",\"description\":\"In this video, learn how to let Lightning help you identify exploding and vanishing gradients to make your training more stable.\",\"breadcrumb\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#primaryimage\",\"url\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png\",\"contentUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png\",\"width\":1920,\"height\":1080,\"caption\":\"How Lightning can identify exploding and vanishing gradients\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/lightning.ai\/pages\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Exploding And Vanishing Gradients in Lightning\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/lightning.ai\/pages\/#website\",\"url\":\"https:\/\/lightning.ai\/pages\/\",\"name\":\"Lightning AI\",\"description\":\"The platform for teams to build AI.\",\"publisher\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/lightning.ai\/pages\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/lightning.ai\/pages\/#organization\",\"name\":\"Lightning AI\",\"url\":\"https:\/\/lightning.ai\/pages\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png\",\"contentUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png\",\"width\":1744,\"height\":856,\"caption\":\"Lightning AI\"},\"image\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/LightningAI\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6\",\"name\":\"JP Hennessy\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g\",\"caption\":\"JP Hennessy\"},\"url\":\"https:\/\/lightning.ai\/pages\/author\/jplightning-ai\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Exploding And Vanishing Gradients in Lightning - Lightning AI","description":"In this video, learn how to let Lightning help you identify exploding and vanishing gradients to make your training more stable.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/","og_locale":"en_US","og_type":"article","og_title":"Exploding And Vanishing Gradients in Lightning - Lightning AI","og_description":"In this video, learn how to let Lightning help you identify exploding and vanishing gradients to make your training more stable.","og_url":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/","og_site_name":"Lightning AI","article_published_time":"2021-05-29T14:31:19+00:00","article_modified_time":"2022-07-25T14:32:05+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png","type":"image\/png"}],"author":"JP Hennessy","twitter_card":"summary_large_image","twitter_creator":"@LightningAI","twitter_site":"@LightningAI","twitter_misc":{"Written by":"JP Hennessy","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#article","isPartOf":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/"},"author":{"name":"JP Hennessy","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6"},"headline":"Exploding And Vanishing Gradients in Lightning","datePublished":"2021-05-29T14:31:19+00:00","dateModified":"2022-07-25T14:32:05+00:00","mainEntityOfPage":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/"},"wordCount":112,"commentCount":0,"publisher":{"@id":"https:\/\/lightning.ai\/pages\/#organization"},"image":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#primaryimage"},"thumbnailUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png","keywords":["gradients","lightning","pytorch","pytorch lightning"],"articleSection":["Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/","url":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/","name":"Exploding And Vanishing Gradients in Lightning - Lightning AI","isPartOf":{"@id":"https:\/\/lightning.ai\/pages\/#website"},"primaryImageOfPage":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#primaryimage"},"image":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#primaryimage"},"thumbnailUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png","datePublished":"2021-05-29T14:31:19+00:00","dateModified":"2022-07-25T14:32:05+00:00","description":"In this video, learn how to let Lightning help you identify exploding and vanishing gradients to make your training more stable.","breadcrumb":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#primaryimage","url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png","contentUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Exploding-And-Vanishing-Gradients.png","width":1920,"height":1080,"caption":"How Lightning can identify exploding and vanishing gradients"},{"@type":"BreadcrumbList","@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/exploding-and-vanishing-gradients\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/lightning.ai\/pages\/"},{"@type":"ListItem","position":2,"name":"Exploding And Vanishing Gradients in Lightning"}]},{"@type":"WebSite","@id":"https:\/\/lightning.ai\/pages\/#website","url":"https:\/\/lightning.ai\/pages\/","name":"Lightning AI","description":"The platform for teams to build AI.","publisher":{"@id":"https:\/\/lightning.ai\/pages\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/lightning.ai\/pages\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/lightning.ai\/pages\/#organization","name":"Lightning AI","url":"https:\/\/lightning.ai\/pages\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/","url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png","contentUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png","width":1744,"height":856,"caption":"Lightning AI"},"image":{"@id":"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/LightningAI"]},{"@type":"Person","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6","name":"JP Hennessy","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g","caption":"JP Hennessy"},"url":"https:\/\/lightning.ai\/pages\/author\/jplightning-ai\/"}]}},"_links":{"self":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/posts\/5646308"}],"collection":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/comments?post=5646308"}],"version-history":[{"count":0,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/posts\/5646308\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/media\/5646309"}],"wp:attachment":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/media?parent=5646308"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/categories?post=5646308"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/tags?post=5646308"},{"taxonomy":"glossary","embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/glossary?post=5646308"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}