Decoding the brain by training 1000 AI models to

Understanding the brain through movement

One of neuroscience’s great challenges is connecting what we see, ie: behavior, to what we can’t see, neural activity. For decades, researchers have asked: how does the brain produce movement? The answers aren’t just academic. They shape how we understand injury recovery, mental health, motor disorders, and even animal intelligence.

At Columbia University’s Paninski Lab, researchers are tackling this question head-on by building AI tools that decode behavior with unprecedented precision. Their latest effort, Lightning Pose, uses deep learning to track animal movement from video more robustly, more accurately, and crucially, much faster than traditional methods.

But building and scaling a tool like this posed serious challenges.

Training 1000s of models was slow

The team knew that a better pose estimation algorithm would improve nearly every downstream neuroscience study involving movement. But to get there, they had to test thousands of model variations across species, datasets, and experimental conditions.

Fitting a single model could take hours. Fitting them all? Their estimates put the timeline at 3–4 months and that was just the model tuning phase.

And that wasn’t the only obstacle. Once the model was ready, they wanted other labs, many without deep software expertise, to actually use it. That meant packaging it, hosting it, and making it reproducible.

For an academic lab, those were major asks.

“At our scale,” said Matt Whiteway, Associate Research Scientist at the lab, “this might’ve been infeasible entirely.”

Thousand models in 2 weeks

By shifting their work into Lightning AI Studios, the team suddenly had access to GPU-powered workspaces, reproducible environments, and a seamless interface for running and managing large-scale sweeps.

They launched 1,000 model training jobs in just two weeks - an order of magnitude faster than expected.

Instead of building infrastructure, they focused on what mattered: improving pose estimation accuracy, shortening training time, and testing more hypotheses.

Lightning Pose: a new foundation

Lightning Pose is a video-centric, semi-supervised deep learning framework for animal pose estimation. Built in PyTorch Lightning, it departs from traditional image-based models by learning from both labeled frames and unlabeled videos, using the physics of motion to regularize predictions over time. It also generalizes better - especially in real-world conditions where occlusions, new camera angles, and unexpected behaviors are common.

On a widely used mouse pupil tracking dataset, Lightning Pose outperformed traditional models like DeepLabCut, producing smoother predictions, lower pixel error, and most importantly, better correlations with neural data.

"When the team regressed neural recordings against pupil diameter predicted by Lightning Pose, they found significantly stronger fits. Better tracking didn’t just look nicer. It revealed more about how the brain works."

Faster science

The Paninski Lab wasn’t just trying to improve pose tracking. They were working to speed up the entire scientific feedback loop - from training and tuning models to sharing reproducible results with collaborators around the world.

With Lightning AI, they cut training time by 90%, bringing what once took months down to just two weeks. New researchers could get started in hours instead of days, launching experiments without setup hurdles or dev environment issues. When it came time to share their work for peer review, the team didn’t send around GitHub links or installation instructions - they shared a live Studio workspace, complete with data, code, logs, and results.

That same workspace now powers open collaboration across institutions. Today, their Lightning Pose framework is helping neuroscience consortia like the International Brain Lab analyze behavior at scale - across more than 30,000 experiments and counting.

Movement helps understand the brain

Science moves on ideas, but progress depends on iteration. The faster researchers can try things, test them, and learn, the faster discovery happens.

That’s especially true in neuroscience, where each experiment helps answer questions that go to the heart of what it means to move, to perceive, to act. How does the brain generate motion? What happens when movement breaks down? And how can we build therapies or brain-machine interfaces that restore it?

Lightning Pose isn’t just a better model. It’s a new pace for neuroscience, one where teams can ask bigger questions, try more hypotheses, and get to answers faster.

When the barriers to scale disappear, what’s left is pure science. And that’s where breakthroughs begin.

Decoding the brain by training 1000 AI models to predict movement