{"id":5646310,"date":"2021-05-30T10:46:29","date_gmt":"2021-05-30T14:46:29","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5646310"},"modified":"2022-07-25T10:48:53","modified_gmt":"2022-07-25T14:48:53","slug":"truncated-back-propogation-through-time","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/","title":{"rendered":"Truncated Back-propogation Through Time"},"content":{"rendered":"<h2>How Lightning Helps You With Long Sequences<\/h2>\n<p>Lightning is a lightweight PyTorch wrapper for high-performance AI research that reduces the boilerplate without limiting flexibility. In this series, we are covering all the tricks Lightning offers to supercharge your machine learning training.<\/p>\n<p>Too many recursive steps can lead to unstable gradients. Truncating the sum of gradients of only the last N steps can eliminate this issue. In this video we explain how you can automatically truncate a time sequence with Lightning&#8217;s implementation of Truncated Back-propogation Through Time using the <code>truncated_bptt_steps<\/code> flag.<\/p>\n<p>Follow along with this notebook: <a href=\"https:\/\/bit.ly\/33YzC1P\" target=\"_blank\" rel=\"noopener\"><u>https:\/\/bit.ly\/33YzC1P<\/u><\/a><\/p>\n<p><iframe loading=\"lazy\" title=\"YouTube video player\" src=\"https:\/\/www.youtube.com\/embed\/2K_jUv8aKN4\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How Lightning Helps You With Long Sequences Lightning is a lightweight PyTorch wrapper for high-performance AI research that reduces the boilerplate without limiting flexibility. In this series, we are covering all the tricks Lightning offers to supercharge your machine learning training. Too many recursive steps can lead to unstable gradients. Truncating the sum of gradients<a class=\"excerpt-read-more\" href=\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/\" title=\"ReadTruncated Back-propogation Through Time\">&#8230; Read more &raquo;<\/a><\/p>\n","protected":false},"author":16,"featured_media":5646311,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":"","_links_to":"","_links_to_target":""},"categories":[41],"tags":[65,61,51,62,66],"glossary":[],"acf":{"hide_from_archive":null,"content_type":null,"code_embed":null,"code_shortcode":null,"custom_styles":null,"sticky":null,"additional_authors":null,"mathjax":null,"default_editor":null,"sections":null,"show_table_of_contents":null,"table_of_contents":null,"tabs":null,"tab_group":null},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Truncated Back-propogation Through Time - Lightning AI<\/title>\n<meta name=\"description\" content=\"In this video, learn how to let Lightning help you deal with long sequences using Truncated Back-propogation Through Time.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Truncated Back-propogation Through Time - Lightning AI\" \/>\n<meta property=\"og:description\" content=\"In this video, learn how to let Lightning help you deal with long sequences using Truncated Back-propogation Through Time.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/\" \/>\n<meta property=\"og:site_name\" content=\"Lightning AI\" \/>\n<meta property=\"article:published_time\" content=\"2021-05-30T14:46:29+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-07-25T14:48:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"JP Hennessy\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@LightningAI\" \/>\n<meta name=\"twitter:site\" content=\"@LightningAI\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"JP Hennessy\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/\"},\"author\":{\"name\":\"JP Hennessy\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6\"},\"headline\":\"Truncated Back-propogation Through Time\",\"datePublished\":\"2021-05-30T14:46:29+00:00\",\"dateModified\":\"2022-07-25T14:48:53+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/\"},\"wordCount\":104,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#organization\"},\"image\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png\",\"keywords\":[\"gradients\",\"lightning\",\"pytorch\",\"pytorch lightning\",\"truncating\"],\"articleSection\":[\"Tutorials\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/\",\"url\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/\",\"name\":\"Truncated Back-propogation Through Time - Lightning AI\",\"isPartOf\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png\",\"datePublished\":\"2021-05-30T14:46:29+00:00\",\"dateModified\":\"2022-07-25T14:48:53+00:00\",\"description\":\"In this video, learn how to let Lightning help you deal with long sequences using Truncated Back-propogation Through Time.\",\"breadcrumb\":{\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#primaryimage\",\"url\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png\",\"contentUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png\",\"width\":1920,\"height\":1080,\"caption\":\"Truncated Back-propogation Through Time in Lightning\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/lightning.ai\/pages\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Truncated Back-propogation Through Time\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/lightning.ai\/pages\/#website\",\"url\":\"https:\/\/lightning.ai\/pages\/\",\"name\":\"Lightning AI\",\"description\":\"The platform for teams to build AI.\",\"publisher\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/lightning.ai\/pages\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/lightning.ai\/pages\/#organization\",\"name\":\"Lightning AI\",\"url\":\"https:\/\/lightning.ai\/pages\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png\",\"contentUrl\":\"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png\",\"width\":1744,\"height\":856,\"caption\":\"Lightning AI\"},\"image\":{\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/LightningAI\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6\",\"name\":\"JP Hennessy\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/lightning.ai\/pages\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g\",\"caption\":\"JP Hennessy\"},\"url\":\"https:\/\/lightning.ai\/pages\/author\/jplightning-ai\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Truncated Back-propogation Through Time - Lightning AI","description":"In this video, learn how to let Lightning help you deal with long sequences using Truncated Back-propogation Through Time.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/","og_locale":"en_US","og_type":"article","og_title":"Truncated Back-propogation Through Time - Lightning AI","og_description":"In this video, learn how to let Lightning help you deal with long sequences using Truncated Back-propogation Through Time.","og_url":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/","og_site_name":"Lightning AI","article_published_time":"2021-05-30T14:46:29+00:00","article_modified_time":"2022-07-25T14:48:53+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png","type":"image\/png"}],"author":"JP Hennessy","twitter_card":"summary_large_image","twitter_creator":"@LightningAI","twitter_site":"@LightningAI","twitter_misc":{"Written by":"JP Hennessy","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#article","isPartOf":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/"},"author":{"name":"JP Hennessy","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6"},"headline":"Truncated Back-propogation Through Time","datePublished":"2021-05-30T14:46:29+00:00","dateModified":"2022-07-25T14:48:53+00:00","mainEntityOfPage":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/"},"wordCount":104,"commentCount":0,"publisher":{"@id":"https:\/\/lightning.ai\/pages\/#organization"},"image":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#primaryimage"},"thumbnailUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png","keywords":["gradients","lightning","pytorch","pytorch lightning","truncating"],"articleSection":["Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/","url":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/","name":"Truncated Back-propogation Through Time - Lightning AI","isPartOf":{"@id":"https:\/\/lightning.ai\/pages\/#website"},"primaryImageOfPage":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#primaryimage"},"image":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#primaryimage"},"thumbnailUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png","datePublished":"2021-05-30T14:46:29+00:00","dateModified":"2022-07-25T14:48:53+00:00","description":"In this video, learn how to let Lightning help you deal with long sequences using Truncated Back-propogation Through Time.","breadcrumb":{"@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#primaryimage","url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png","contentUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2022\/07\/Truncated-Back-propogation-Through-Time.png","width":1920,"height":1080,"caption":"Truncated Back-propogation Through Time in Lightning"},{"@type":"BreadcrumbList","@id":"https:\/\/lightning.ai\/pages\/community\/tutorial\/truncated-back-propogation-through-time\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/lightning.ai\/pages\/"},{"@type":"ListItem","position":2,"name":"Truncated Back-propogation Through Time"}]},{"@type":"WebSite","@id":"https:\/\/lightning.ai\/pages\/#website","url":"https:\/\/lightning.ai\/pages\/","name":"Lightning AI","description":"The platform for teams to build AI.","publisher":{"@id":"https:\/\/lightning.ai\/pages\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/lightning.ai\/pages\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/lightning.ai\/pages\/#organization","name":"Lightning AI","url":"https:\/\/lightning.ai\/pages\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/","url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png","contentUrl":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/02\/image-17.png","width":1744,"height":856,"caption":"Lightning AI"},"image":{"@id":"https:\/\/lightning.ai\/pages\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/LightningAI"]},{"@type":"Person","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/person\/2518f4d5541f8e98016f6289169141a6","name":"JP Hennessy","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/lightning.ai\/pages\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/28ade268218ae45f723b0b62499f527a?s=96&d=mm&r=g","caption":"JP Hennessy"},"url":"https:\/\/lightning.ai\/pages\/author\/jplightning-ai\/"}]}},"_links":{"self":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/posts\/5646310"}],"collection":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/comments?post=5646310"}],"version-history":[{"count":0,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/posts\/5646310\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/media\/5646311"}],"wp:attachment":[{"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/media?parent=5646310"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/categories?post=5646310"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/tags?post=5646310"},{"taxonomy":"glossary","embeddable":true,"href":"https:\/\/lightning.ai\/pages\/wp-json\/wp\/v2\/glossary?post=5646310"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}