Very slow performance with trainer.fit

1

Hi, I have moved my 3D u-net like model to the lightning.
I have a single GPU (i.e. K40) and was hoping for a better utilisation of it.
It looks like my code does not utilizes it’s full potential.
However, after running trainer.fit() I can see 2x slow down :frowning:
The whole framework looks quite complex and I wanted to ask for some tips on where should I look to get it tuned up?
Are there some unnecessary data transmissions or logging etc.
Thank you in advance for all the help.
G.

2

Hi, you can try Profiler to see what’s taking most of the time. If your data loader is taking more time, you can try adjusting the number of workers in your dataloader.

Additionally, you can try some of these plugins, though I am not sure if they would be helpful in reducing execution time.