{"version":"1.0","provider_name":"Lightning AI","provider_url":"https:\/\/lightning.ai\/pages","author_name":"JP Hennessy","author_url":"https:\/\/lightning.ai\/pages\/author\/jplightning-ai\/","title":"Scaling Large (Language) Models with PyTorch Lightning - Lightning AI","type":"rich","width":600,"height":338,"html":"<blockquote class=\"wp-embedded-content\" data-secret=\"qn8AtOY7Kh\"><a href=\"https:\/\/lightning.ai\/pages\/blog\/scaling-large-language-models-with-pytorch-lightning\/\">Scaling Large (Language) Models with PyTorch Lightning<\/a><\/blockquote><iframe sandbox=\"allow-scripts\" security=\"restricted\" src=\"https:\/\/lightning.ai\/pages\/blog\/scaling-large-language-models-with-pytorch-lightning\/embed\/#?secret=qn8AtOY7Kh\" width=\"600\" height=\"338\" title=\"&#8220;Scaling Large (Language) Models with PyTorch Lightning&#8221; &#8212; Lightning AI\" data-secret=\"qn8AtOY7Kh\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" class=\"wp-embedded-content\"><\/iframe><script type=\"text\/javascript\">\n\/* <![CDATA[ *\/\n\/*! This file is auto-generated *\/\n!function(d,l){\"use strict\";l.querySelector&&d.addEventListener&&\"undefined\"!=typeof URL&&(d.wp=d.wp||{},d.wp.receiveEmbedMessage||(d.wp.receiveEmbedMessage=function(e){var t=e.data;if((t||t.secret||t.message||t.value)&&!\/[^a-zA-Z0-9]\/.test(t.secret)){for(var s,r,n,a=l.querySelectorAll('iframe[data-secret=\"'+t.secret+'\"]'),o=l.querySelectorAll('blockquote[data-secret=\"'+t.secret+'\"]'),c=new RegExp(\"^https?:$\",\"i\"),i=0;i<o.length;i++)o[i].style.display=\"none\";for(i=0;i<a.length;i++)s=a[i],e.source===s.contentWindow&&(s.removeAttribute(\"style\"),\"height\"===t.message?(1e3<(r=parseInt(t.value,10))?r=1e3:~~r<200&&(r=200),s.height=r):\"link\"===t.message&&(r=new URL(s.getAttribute(\"src\")),n=new URL(t.value),c.test(n.protocol))&&n.host===r.host&&l.activeElement===s&&(d.top.location.href=t.value))}},d.addEventListener(\"message\",d.wp.receiveEmbedMessage,!1),l.addEventListener(\"DOMContentLoaded\",function(){for(var e,t,s=l.querySelectorAll(\"iframe.wp-embedded-content\"),r=0;r<s.length;r++)(t=(e=s[r]).getAttribute(\"data-secret\"))||(t=Math.random().toString(36).substring(2,12),e.src+=\"#?secret=\"+t,e.setAttribute(\"data-secret\",t)),e.contentWindow.postMessage({message:\"ready\",secret:t},\"*\")},!1)))}(window,document);\n\/* ]]> *\/\n<\/script>\n","thumbnail_url":"https:\/\/lightningaidev.wpengine.com\/wp-content\/uploads\/2023\/10\/fsdp-blog.png","thumbnail_width":2048,"thumbnail_height":1213,"description":"In this blog, you will learn about techniques to train large models like Llama (or any LLM) and Stable Diffusion using distributed training strategy FSDP with PyTorch Lightning. In recent times, there has been a notable shift in the scale of models, particularly in the realm of language models such as GPT 4, Llama, and... Read more &raquo;"}