Hi,
I am new to PyTorch Lightning and I am loving it so far!
I was wondering, when using the track_grad_norm flag as so:
trainer = pl.Trainer(
gpus=gpu_count,
#auto_scale_batch_size=auto_scale_batch_size,
precision=16,
auto_lr_find=auto_lr_find,
min_epochs=10,
max_epochs=100,
num_sanity_val_steps=5,
track_grad_norm=2, #L2 Norm
callbacks=[early_stopping, model_checkpoint]
)
I am getting following output:
Epoch 2: 100%|██████████| 74/74 [14:29<00:00, 16.41s/it, loss=0.335, v_num=1, train_loss=0.553, grad_2.0_norm_net.features.conv0.weight_step=268.0, grad_2.0_norm_net.features.norm0.weight_step=0.248, grad_2.0_norm_net.features.norm0.bias_step=13.50, grad_2.0_norm_net.features.denseblock1.denselayer1.layers.norm1.weight_step=17.90, grad_2.0_norm_net.features.denseblock1.denselayer1.layers.norm1.bias_step=15.90, grad_2.0_norm_net.features.denseblock1.denselayer1.layers.conv1.weight_step=98.40, grad_2.0_norm_net.features.denseblock1.denselayer1.layers.norm2.weight_step=13.70, grad_2.0_norm_net.features.denseblock1.denselayer1.layers.norm2.bias_step=9.170, grad_2.0_norm_net.features.denseblock1.denselayer1.layers.conv2.weight_step=208.0, grad_2.0_norm_net.features.denseblock1.denselayer2.layers.norm1.weight_step=10.40, grad_2.0_norm_net.features.denseblock1.denselayer2.layers.norm1.bias_step=7.020, grad_2.0_norm_net.features.denseblock1.denselayer2.layers.conv1.weight_step=72.80, grad_2.0_norm_net.features.denseblock1.denselayer2.layers.norm2.weight_step=8.920, grad_2.0_norm_net.features.denseblock1.denselayer2.layers.norm2.bias_step=6.110, grad_2.0_norm_net.features.denseblock1.denselayer2.layers.conv2.weight_step=179.0, grad_2.0_norm_net.features.denseblock1.denselayer3.layers.norm1.weight_step=19.80, grad_2.0_norm_net.features.denseblock1.denselayer3.layers.norm1.bias_step=14.70, grad_2.0_norm_net.features.denseblock1.denselayer3.layers.conv1.weight_step=140.0, grad_2.0_norm_net.features.denseblock1.denselayer3.layers.norm2.weight_step=12.20, grad_2.0_norm_net.features.denseblock1.denselayer3.layers.norm2.bias_step=11.40, grad_2.0_norm_net.features.denseblock1.denselayer3.layers.conv2.weight_step=199.0, grad_2.0_norm_net.features.denseblock1.denselayer4.layers.norm1.weight_step=9.990, grad_2.0_norm_net.features.denseblock1.denselayer4.layers.norm1.bias_step=6.520, grad_2.0_norm_net.features.denseblock1.denselayer4.layers.conv1.weight_step=86.70, grad_2.0_norm_net.features.denseblock1.denselayer4.layers.norm2.weight_step=8.500, grad_2.0_norm_net.features.denseblock1.denselayer4.layers.norm2.bias_step=5.570, grad_2.0_norm_net.features.denseblock1.denselayer4.layers.conv2.weight_step=167.0, grad_2.0_norm_net.features.denseblock1.denselayer5.layers.norm1.weight_step=15.20, grad_2.0_norm_net.features.denseblock1.denselayer5.layers.norm1.bias_step=9.560, grad_2.0_norm_net.features.denseblock1.denselayer5.layers.conv1.weight_step=129.0, grad_2.0_norm_net.features.denseblock1.denselayer5.layers.norm2.weight_step=10.80, grad_2.0_norm_net.features.denseblock1.denselayer5.layers.norm2.bias_step=8.860, grad_2.0_norm_net.features.denseblock1.denselayer5.layers.conv2.weight_step=197.0, grad_2.0_norm_net.features.denseblock1.denselayer6.layers.norm1.weight_step=18.00, grad_2.0_norm_net.features.denseblock1.denselayer6.layers.norm1.bias_step=8.890, grad_2.0_norm_net.features.denseblock1.denselayer6.layers.conv1.weight_step=136.0, grad_2.0_norm_net.features.denseblock1.denselayer6.layers.norm2.weight_step=10.90, grad_2.0_norm_net.features.denseblock1.denselayer6.layers.norm2.bias_step=7.730, grad_2.0_norm_net.features.denseblock1.denselayer6.layers.conv2.weight_step=174.0, grad_2.0_norm_net.features.transition1.norm.weight_step=40.10, grad_2.0_norm_net.features.transition1.norm.bias_step=25.80, grad_2.0_norm_net.features.transition1.conv.weight_step=379.0, grad_2.0_norm_net.features.denseblock2.denselayer1.layers.norm1.weight_step=30.90, grad_2.0_norm_net.features.denseblock2.denselayer1.layers.norm1.bias_step=15.00, grad_2.0_norm_net.features.denseblock2.denselayer1.layers.conv1.weight_step=207.0, grad_2.0_norm_net.features.denseblock2.denselayer1.layers.norm2.weight_step=18.90, grad_2.0_norm_net.features.denseblock2.denselayer1.layers.norm2.bias_step=11.10, grad_2.0_norm_net.features.denseblock2.denselayer1.layers.conv2.weight_step=424.0, grad_2.0_norm_net.features.denseblock2.denselayer2.layers.norm1.weight_step=30.30, grad_2.0_norm_net.features.denseblock2.denselayer2.layers.norm1.bias_step=16.90, grad_2.0_norm_net.features.denseblock2.denselayer2.layers.conv1.weight_step=212.0, grad_2.0_norm_net.features.denseblock2.denselayer2.layers.norm2.weight_step=19.20, grad_2.0_norm_net.features.denseblock2.denselayer2.layers.norm2.bias_step=13.10, grad_2.0_norm_net.features.denseblock2.denselayer2.layers.conv2.weight_step=357.0, grad_2.0_norm_net.features.denseblock2.denselayer3.layers.norm1.weight_step=18.20, grad_2.0_norm_net.features.denseblock2.denselayer3.layers.norm1.bias_step=10.20, grad_2.0_norm_net.features.denseblock2.denselayer3.layers.conv1.weight_step=140.0, grad_2.0_norm_net.features.denseblock2.denselayer3.layers.norm2.weight_step=12.90, grad_2.0_norm_net.features.denseblock2.denselayer3.layers.norm2.bias_step=8.050, grad_2.0_norm_net.features.denseblock2.denselayer3.layers.conv2.weight_step=252.0, grad_2.0_norm_net.features.denseblock2.denselayer4.layers.norm1.weight_step=11.10, grad_2.0_norm_net.features.denseblock2.denselayer4.layers.norm1.bias_step=6.270, grad_2.0_norm_net.features.denseblock2.denselayer4.layers.conv1.weight_step=98.70, grad_2.0_norm_net.features.denseblock2.denselayer4.layers.norm2.weight_step=7.500, grad_2.0_norm_net.features.denseblock2.denselayer4.layers.norm2.bias_step=6.970, grad_2.0_norm_net.features.denseblock2.denselayer4.layers.conv2.weight_step=180.0, grad_2.0_norm_net.features.denseblock2.denselayer5.layers.norm1.weight_step=14.30, grad_2.0_norm_net.features.denseblock2.denselayer5.layers.norm1.bias_step=9.480, grad_2.0_norm_net.features.denseblock2.denselayer5.layers.conv1.weight_step=147.0, grad_2.0_norm_net.features.denseblock2.denselayer5.layers.norm2.weight_step=13.00, grad_2.0_norm_net.features.denseblock2.denselayer5.layers.norm2.bias_step=9.410, grad_2.0_norm_net.features.denseblock2.denselayer5.layers.conv2.weight_step=234.0, grad_2.0_norm_net.features.denseblock2.denselayer6.layers.norm1.weight_step=15.20, grad_2.0_norm_net.features.denseblock2.denselayer6.layers.norm1.bias_step=9.810, grad_2.0_norm_net.features.denseblock2.denselayer6.layers.conv1.weight_step=154.0, grad_2.0_norm_net.features.denseblock2.denselayer6.layers.norm2.weight_step=12.40, grad_2.0_norm_net.features.denseblock2.denselayer6.layers.norm2.bias_step=9.180, grad_2.0_norm_net.features.denseblock2.denselayer6.layers.conv2.weight_step=288.0, grad_2.0_norm_net.features.denseblock2.denselayer7.layers.norm1.weight_step=18.40, grad_2.0_norm_net.features.denseblock2.denselayer7.layers.norm1.bias_step=8.870, grad_2.0_norm_net.features.denseblock2.denselayer7.layers.conv1.weight_step=212.0, grad_2.0_norm_net.features.denseblock2.denselayer7.layers.norm2.weight_step=15.80, grad_2.0_norm_net.features.denseblock2.denselayer7.layers.norm2.bias_step=8.840, grad_2.0_norm_net.features.denseblock2.denselayer7.layers.conv2.weight_step=337.0, grad_2.0_norm_net.features.denseblock2.denselayer8.layers.norm1.weight_step=22.30, grad_2.0_norm_net.features.denseblock2.denselayer8.layers.norm1.bias_step=12.40, grad_2.0_norm_net.features.denseblock2.denselayer8.layers.conv1.weight_step=253.0, grad_2.0_norm_net.features.denseblock2.denselayer8.layers.norm2.weight_step=17.10, grad_2.0_norm_net.features.denseblock2.denselayer8.layers.norm2.bias_step=12.20, grad_2.0_norm_net.features.denseblock2.denselayer8.layers.conv2.weight_step=360.0, grad_2.0_norm_net.features.denseblock2.denselayer9.layers.norm1.weight_step=22.90, grad_2.0_norm_net.features.denseblock2.denselayer9.layers.norm1.bias_step=15.10, grad_2.0_norm_net.features.denseblock2.denselayer9.layers.conv1.weight_step=280.0, grad_2.0_norm_net.features.denseblock2.denselayer9.layers.norm2.weight_step=21.90, grad_2.0_norm_net.features.denseblock2.denselayer9.layers.norm2.bias_step=14.90, grad_2.0_norm_net.features.denseblock2.denselayer9.layers.conv2.weight_step=401.0, grad_2.0_norm_net.features.denseblock2.denselayer10.layers.norm1.weight_step=19.80, grad_2.0_norm_net.features.denseblock2.denselayer10.layers.norm1.bias_step=10.80, grad_2.0_norm_net.features.denseblock2.denselayer10.layers.conv1.weight_step=254.0, grad_2.0_norm_net.features.denseblock2.denselayer10.layers.norm2.weight_step=18.00, grad_2.0_norm_net.features.denseblock2.denselayer10.layers.norm2.bias_step=9.840, grad_2.0_norm_net.features.denseblock2.denselayer10.layers.conv2.weight_step=306.0, grad_2.0_norm_net.features.denseblock2.denselayer11.layers.norm1.weight_step=16.10, grad_2.0_norm_net.features.denseblock2.denselayer11.layers.norm1.bias_step=8.110, grad_2.0_norm_net.features.denseblock2.denselayer11.layers.conv1.weight_step=183.0, grad_2.0_norm_net.features.denseblock2.denselayer11.layers.norm2.weight_step=10.50, grad_2.0_norm_net.features.denseblock2.denselayer11.layers.norm2.bias_step=5.670, grad_2.0_norm_net.features.denseblock2.denselayer11.layers.conv2.weight_step=221.0, grad_2.0_norm_net.features.denseblock2.denselayer12.layers.norm1.weight_step=13.80, grad_2.0_norm_net.features.denseblock2.denselayer12.layers.norm1.bias_step=7.720, grad_2.0_norm_net.features.denseblock2.denselayer12.layers.conv1.weight_step=191.0, grad_2.0_norm_net.features.denseblock2.denselayer12.layers.norm2.weight_step=10.30, grad_2.0_norm_net.features.denseblock2.denselayer12.layers.norm2.bias_step=6.640, grad_2.0_norm_net.features.denseblock2.denselayer12.layers.conv2.weight_step=254.0, grad_2.0_norm_net.features.transition2.norm.weight_step=51.20, grad_2.0_norm_net.features.transition2.norm.bias_step=32.40, grad_2.0_norm_net.features.transition2.conv.weight_step=702.0,
...
grad_2.0_norm_net.features.denseblock3.denselayer10.layers.norm1.weight_epoch=60.60, grad_2.0_norm_net.features.denseblock3.denselayer10.layers.norm1.bias_epoch=42.40, grad_2.0_norm_net.features.denseblock3.denselayer10.layers.conv1.weight_epoch=1.05e+3, grad_2.0_norm_net.features.denseblock3.denselayer10.layers.norm2.weight_epoch=57.80, grad_2.0_norm_net.features.denseblock3.denselayer10.layers.norm2.bias_epoch=38.40, grad_2.0_norm_net.features.denseblock3.denselayer10.layers.conv2.weight_epoch=2.43e+3, grad_2.0_norm_net.features.denseblock3.denselayer11.layers.norm1.weight_epoch=57.70, grad_2.0_norm_net.features.denseblock3.denselayer11.layers.norm1.bias_epoch=40.20, grad_2.0_norm_net.features.denseblock3.denselayer11.layers.conv1.weight_epoch=980.0, grad_2.0_norm_net.features.denseblock3.denselayer11.layers.norm2.weight_epoch=56.00, grad_2.0_norm_net.features.denseblock3.denselayer11.layers.norm2.bias_epoch=41.80, grad_2.0_norm_net.features.denseblock3.denselayer11.layers.conv2.weight_epoch=2.32e+3, grad_2.0_norm_net.features.denseblock3.denselayer12.layers.norm1.weight_epoch=59.30, grad_2.0_norm_net.features.denseblock3.denselayer12.layers.norm1.bias_epoch=40.80, grad_2.0_norm_net.features.denseblock3.denselayer12.layers.conv1.weight_epoch=1.03e+3, grad_2.0_norm_net.features.denseblock3.denselayer12.layers.norm2.weight_epoch=60.60, grad_2.0_norm_net.features.denseblock3.denselayer12.layers.norm2.bias_epoch=39.70, grad_2.0_norm_net.features.denseblock3.denselayer12.layers.conv2.weight_epoch=2.41e+3, grad_2.0_norm_net.features.denseblock3.denselayer13.layers.norm1.weight_epoch=58.60, grad_2.0_norm_net.features.denseblock3.denselayer13.layers.norm1.bias_epoch=41.40, grad_2.0_norm_net.features.denseblock3.denselayer13.layers.conv1.weight_epoch=1.05e+3, grad_2.0_norm_net.features.denseblock3.denselayer13.layers.norm2.weight_epoch=54.30, grad_2.0_norm_net.features.denseblock3.denselayer13.layers.norm2.bias_epoch=41.40, grad_2.0_norm_net.features.denseblock3.denselayer13.layers.conv2.weight_epoch=2.24e+3, grad_2.0_norm_net.features.denseblock3.denselayer14.layers.norm1.weight_epoch=53.40, grad_2.0_norm_net.features.denseblock3.denselayer14.layers.norm1.bias_epoch=37.80, grad_2.0_norm_net.features.denseblock3.denselayer14.layers.conv1.weight_epoch=972.0, grad_2.0_norm_net.features.denseblock3.denselayer14.layers.norm2.weight_epoch=53.00, grad_2.0_norm_net.features.denseblock3.denselayer14.layers.norm2.bias_epoch=37.50, grad_2.0_norm_net.features.denseblock3.denselayer14.layers.conv2.weight_epoch=2.13e+3, grad_2.0_norm_net.features.denseblock3.denselayer15.layers.norm1.weight_epoch=52.60, grad_2.0_norm_net.features.denseblock3.denselayer15.layers.norm1.bias_epoch=38.40, grad_2.0_norm_net.features.denseblock3.denselayer15.layers.conv1.weight_epoch=991.0, grad_2.0_norm_net.features.denseblock3.denselayer15.layers.norm2.weight_epoch=47.50, grad_2.0_norm_net.features.denseblock3.denselayer15.layers.norm2.bias_epoch=39.70, grad_2.0_norm_net.features.denseblock3.denselayer15.layers.conv2.weight_epoch=2.1e+3, grad_2.0_norm_net.features.denseblock3.denselayer16.layers.norm1.weight_epoch=49.30, grad_2.0_norm_net.features.denseblock3.denselayer16.layers.norm1.bias_epoch=34.30, grad_2.0_norm_net.features.denseblock3.denselayer16.layers.conv1.weight_epoch=941.0, grad_2.0_norm_net.features.denseblock3.denselayer16.layers.norm2.weight_epoch=54.20, grad_2.0_norm_net.features.denseblock3.denselayer16.layers.norm2.bias_epoch=37.20, grad_2.0_norm_net.features.denseblock3.denselayer16.layers.conv2.weight_epoch=1.94e+3, grad_2.0_norm_net.features.denseblock3.denselayer17.layers.norm1.weight_epoch=53.50, grad_2.0_norm_net.features.denseblock3.denselayer17.layers.norm1.bias_epoch=36.70, grad_2.0_norm_net.features.denseblock3.denselayer17.layers.conv1.weight_epoch=1.04e+3, grad_2.0_norm_net.features.denseblock3.denselayer17.layers.norm2.weight_epoch=53.90, grad_2.0_norm_net.features.denseblock3.denselayer17.layers.norm2.bias_epoch=38.50, grad_2.0_norm_net.features.denseblock3.denselayer17.layers.conv2.weight_epoch=2.21e+3, grad_2.0_norm_net.features.denseblock3.denselayer18.layers.norm1.weight_epoch=52.00, grad_2.0_norm_net.features.denseblock3.denselayer18.layers.norm1.bias_epoch=36.90, grad_2.0_norm_net.features.denseblock3.denselayer18.layers.conv1.weight_epoch=1.03e+3, grad_2.0_norm_net.features.denseblock3.denselayer18.layers.norm2.weight_epoch=49.50, grad_2.0_norm_net.features.denseblock3.denselayer18.layers.norm2.bias_epoch=40.60, grad_2.0_norm_net.features.denseblock3.denselayer18.layers.conv2.weight_epoch=2.04e+3, grad_2.0_norm_net.features.denseblock3.denselayer19.layers.norm1.weight_epoch=50.90, grad_2.0_norm_net.features.denseblock3.denselayer19.layers.norm1.bias_epoch=34.10, grad_2.0_norm_net.features.denseblock3.denselayer19.layers.conv1.weight_epoch=1.02e+3, grad_2.0_norm_net.features.denseblock3.denselayer19.layers.norm2.weight_epoch=46.10, grad_2.0_norm_net.features.denseblock3.denselayer19.layers.norm2.bias_epoch=34.30, grad_2.0_norm_net.features.denseblock3.denselayer19.layers.conv2.weight_epoch=2.08e+3, grad_2.0_norm_net.features.denseblock3.denselayer20.layers.norm1.weight_epoch=46.30, grad_2.0_norm_net.features.denseblock3.denselayer20.layers.norm1.bias_epoch=33.30, grad_2.0_norm_net.features.denseblock3.denselayer20.layers.conv1.weight_epoch=989.0, grad_2.0_norm_net.features.denseblock3.denselayer20.layers.norm2.weight_epoch=47.60, grad_2.0_norm_net.features.denseblock3.denselayer20.layers.norm2.bias_epoch=34.40, grad_2.0_norm_net.features.denseblock3.denselayer20.layers.conv2.weight_epoch=1.98e+3, grad_2.0_norm_net.features.denseblock3.denselayer21.layers.norm1.weight_epoch=46.90, grad_2.0_norm_net.features.denseblock3.denselayer21.layers.norm1.bias_epoch=33.30, grad_2.0_norm_net.features.denseblock3.denselayer21.layers.conv1.weight_epoch=980.0, grad_2.0_norm_net.features.denseblock3.denselayer21.layers.norm2.weight_epoch=48.50, grad_2.0_norm_net.features.denseblock3.denselayer21.layers.norm2.bias_epoch=36.90, grad_2.0_norm_net.features.denseblock3.denselayer21.layers.conv2.weight_epoch=1.89e+3, grad_2.0_norm_net.features.denseblock3.denselayer22.layers.norm1.weight_epoch=45.10, grad_2.0_norm_net.features.denseblock3.denselayer22.layers.norm1.bias_epoch=29.30, grad_2.0_norm_net.features.denseblock3.denselayer22.layers.conv1.weight_epoch=1e+3, grad_2.0_norm_net.features.denseblock3.denselayer22.layers.norm2.weight_epoch=46.10, grad_2.0_norm_net.features.denseblock3.denselayer22.layers.norm2.bias_epoch=30.70, grad_2.0_norm_net.features.denseblock3.denselayer22.layers.conv2.weight_epoch=1.92e+3, grad_2.0_norm_net.features.denseblock3.denselayer23.layers.norm1.weight_epoch=45.20, grad_2.0_norm_net.features.denseblock3.denselayer23.layers.norm1.bias_epoch=31.10, grad_2.0_norm_net.features.denseblock3.denselayer23.layers.conv1.weight_epoch=982.0, grad_2.0_norm_net.features.denseblock3.denselayer23.layers.norm2.weight_epoch=48.40, grad_2.0_norm_net.features.denseblock3.denselayer23.layers.norm2.bias_epoch=33.50, grad_2.0_norm_net.features.denseblock3.denselayer23.layers.conv2.weight_epoch=1.98e+3, grad_2.0_norm_net.features.denseblock3.denselayer24.layers.norm1.weight_epoch=41.70, grad_2.0_norm_net.features.denseblock3.denselayer24.layers.norm1.bias_epoch=29.60, grad_2.0_norm_net.features.denseblock3.denselayer24.layers.conv1.weight_epoch=925.0, grad_2.0_norm_net.features.denseblock3.denselayer24.layers.norm2.weight_epoch=43.40, grad_2.0_norm_net.features.denseblock3.denselayer24.layers.norm2.bias_epoch=30.30, grad_2.0_norm_net.features.denseblock3.denselayer24.layers.conv2.weight_epoch=1.74e+3, grad_2.0_norm_net.features.transition3.norm.weight_epoch=332.0, grad_2.0_norm_net.features.transition3.norm.bias_epoch=239.0, grad_2.0_norm_net.features.transition3.conv.weight_epoch=7.43e+3, grad_2.0_norm_net.features.denseblock4.denselayer1.layers.norm1.weight_epoch=72.20, grad_2.0_norm_net.features.denseblock4.denselayer1.layers.norm1.bias_epoch=60.20, grad_2.0_norm_net.features.denseblock4.denselayer1.layers.conv1.weight_epoch=1.16e+3, grad_2.0_norm_net.features.denseblock4.denselayer1.layers.norm2.weight_epoch=68.50, grad_2.0_norm_net.features.denseblock4.denselayer1.layers.norm2.bias_epoch=58.80, grad_2.0_norm_net.features.denseblock4.denselayer1.layers.conv2.weight_epoch=2.83e+3, grad_2.0_norm_net.features.denseblock4.denselayer2.layers.norm1.weight_epoch=68.10, grad_2.0_norm_net.features.denseblock4.denselayer2.layers.norm1.bias_epoch=60.10, grad_2.0_norm_net.features.denseblock4.denselayer2.layers.conv1.weight_epoch=1.12e+3, grad_2.0_norm_net.features.denseblock4.denselayer2.layers.norm2.weight_epoch=59.30, grad_2.0_norm_net.features.denseblock4.denselayer2.layers.norm2.bias_epoch=52.60, grad_2.0_norm_net.features.denseblock4.denselayer2.layers.conv2.weight_epoch=2.61e+3, grad_2.0_norm_net.features.denseblock4.denselayer3.layers.norm1.weight_epoch=67.40, grad_2.0_norm_net.features.denseblock4.denselayer3.layers.norm1.bias_epoch=61.50, grad_2.0_norm_net.features.denseblock4.denselayer3.layers.conv1.weight_epoch=1.15e+3, grad_2.0_norm_net.features.denseblock4.denselayer3.layers.norm2.weight_epoch=64.80, grad_2.0_norm_net.features.denseblock4.denselayer3.layers.norm2.bias_epoch=56.80, grad_2.0_norm_net.features.denseblock4.denselayer3.layers.conv2.weight_epoch=2.6e+3, grad_2.0_norm_net.features.denseblock4.denselayer4.layers.norm1.weight_epoch=65.30, grad_2.0_norm_net.features.denseblock4.denselayer4.layers.norm1.bias_epoch=57.10, grad_2.0_norm_net.features.denseblock4.denselayer4.layers.conv1.weight_epoch=1.12e+3, grad_2.0_norm_net.features.denseblock4.denselayer4.layers.norm2.weight_epoch=59.20, grad_2.0_norm_net.features.denseblock4.denselayer4.layers.norm2.bias_epoch=52.70, grad_2.0_norm_net.features.denseblock4.denselayer4.layers.conv2.weight_epoch=2.47e+3, grad_2.0_norm_net.features.denseblock4.denselayer5.layers.norm1.weight_epoch=66.40, grad_2.0_norm_net.features.denseblock4.denselayer5.layers.norm1.bias_epoch=56.70, grad_2.0_norm_net.features.denseblock4.denselayer5.layers.conv1.weight_epoch=1.18e+3, grad_2.0_norm_net.features.denseblock4.denselayer5.layers.norm2.weight_epoch=60.70, grad_2.0_norm_net.features.denseblock4.denselayer5.layers.norm2.bias_epoch=50.60, grad_2.0_norm_net.features.denseblock4.denselayer5.layers.conv2.weight_epoch=2.54e+3, grad_2.0_norm_net.features.denseblock4.denselayer6.layers.norm1.weight_epoch=59.90, grad_2.0_norm_net.features.denseblock4.denselayer6.layers.norm1.bias_epoch=52.60, grad_2.0_norm_net.features.denseblock4.denselayer6.layers.conv1.weight_epoch=1.1e+3, grad_2.0_norm_net.features.denseblock4.denselayer6.layers.norm2.weight_epoch=52.30, grad_2.0_norm_net.features.denseblock4.denselayer6.layers.norm2.bias_epoch=44.80, grad_2.0_norm_net.features.denseblock4.denselayer6.layers.conv2.weight_epoch=2.27e+3, grad_2.0_norm_net.features.denseblock4.denselayer7.layers.norm1.weight_epoch=60.50, grad_2.0_norm_net.features.denseblock4.denselayer7.layers.norm1.bias_epoch=53.60, grad_2.0_norm_net.features.denseblock4.denselayer7.layers.conv1.weight_epoch=1.1e+3, grad_2.0_norm_net.features.denseblock4.denselayer7.layers.norm2.weight_epoch=57.40, grad_2.0_norm_net.features.denseblock4.denselayer7.layers.norm2.bias_epoch=48.10, grad_2.0_norm_net.features.denseblock4.denselayer7.layers.conv2.weight_epoch=2.48e+3, grad_2.0_norm_net.features.denseblock4.denselayer8.layers.norm1.weight_epoch=63.20, grad_2.0_norm_net.features.denseblock4.denselayer8.layers.norm1.bias_epoch=57.80, grad_2.0_norm_net.features.denseblock4.denselayer8.layers.conv1.weight_epoch=1.17e+3, grad_2.0_norm_net.features.denseblock4.denselayer8.layers.norm2.weight_epoch=64.90, grad_2.0_norm_net.features.denseblock4.denselayer8.layers.norm2.bias_epoch=53.80, grad_2.0_norm_net.features.denseblock4.denselayer8.layers.conv2.weight_epoch=2.39e+3, grad_2.0_norm_net.features.denseblock4.denselayer9.layers.norm1.weight_epoch=56.00, grad_2.0_norm_net.features.denseblock4.denselayer9.layers.norm1.bias_epoch=49.20, grad_2.0_norm_net.features.denseblock4.denselayer9.layers.conv1.weight_epoch=1.09e+3, grad_2.0_norm_net.features.denseblock4.denselayer9.layers.norm2.weight_epoch=57.50, grad_2.0_norm_net.features.denseblock4.denselayer9.layers.norm2.bias_epoch=45.70, grad_2.0_norm_net.features.denseblock4.denselayer9.layers.conv2.weight_epoch=2.31e+3, grad_2.0_norm_net.features.denseblock4.denselayer10.layers.norm1.weight_epoch=52.20, grad_2.0_norm_net.features.denseblock4.denselayer10.layers.norm1.bias_epoch=45.60, grad_2.0_norm_net.features.denseblock4.denselayer10.layers.conv1.weight_epoch=1.06e+3, grad_2.0_norm_net.features.denseblock4.denselayer10.layers.norm2.weight_epoch=51.70, grad_2.0_norm_net.features.denseblock4.denselayer10.layers.norm2.bias_epoch=41.50, grad_2.0_norm_net.features.denseblock4.denselayer10.layers.conv2.weight_epoch=2.08e+3, grad_2.0_norm_net.features.denseblock4.denselayer11.layers.norm1.weight_epoch=45.10, grad_2.0_norm_net.features.denseblock4.denselayer11.layers.norm1.bias_epoch=40.90, grad_2.0_norm_net.features.denseblock4.denselayer11.layers.conv1.weight_epoch=925.0, grad_2.0_norm_net.features.denseblock4.denselayer11.layers.norm2.weight_epoch=42.10, grad_2.0_norm_net.features.denseblock4.denselayer11.layers.norm2.bias_epoch=36.80, grad_2.0_norm_net.features.denseblock4.denselayer11.layers.conv2.weight_epoch=1.83e+3, grad_2.0_norm_net.features.denseblock4.denselayer12.layers.norm1.weight_epoch=48.60, grad_2.0_norm_net.features.denseblock4.denselayer12.layers.norm1.bias_epoch=42.00, grad_2.0_norm_net.features.denseblock4.denselayer12.layers.conv1.weight_epoch=965.0, grad_2.0_norm_net.features.denseblock4.denselayer12.layers.norm2.weight_epoch=51.00, grad_2.0_norm_net.features.denseblock4.denselayer12.layers.norm2.bias_epoch=37.40, grad_2.0_norm_net.features.denseblock4.denselayer12.layers.conv2.weight_epoch=2.03e+3, grad_2.0_norm_net.features.denseblock4.denselayer13.layers.norm1.weight_epoch=53.80, grad_2.0_norm_net.features.denseblock4.denselayer13.layers.norm1.bias_epoch=49.80, grad_2.0_norm_net.features.denseblock4.denselayer13.layers.conv1.weight_epoch=1.15e+3, grad_2.0_norm_net.features.denseblock4.denselayer13.layers.norm2.weight_epoch=54.10, grad_2.0_norm_net.features.denseblock4.denselayer13.layers.norm2.bias_epoch=43.70, grad_2.0_norm_net.features.denseblock4.denselayer13.layers.conv2.weight_epoch=2.18e+3, grad_2.0_norm_net.features.denseblock4.denselayer14.layers.norm1.weight_epoch=54.60, grad_2.0_norm_net.features.denseblock4.denselayer14.layers.norm1.bias_epoch=48.30, grad_2.0_norm_net.features.denseblock4.denselayer14.layers.conv1.weight_epoch=1.17e+3, grad_2.0_norm_net.features.denseblock4.denselayer14.layers.norm2.weight_epoch=54.30, grad_2.0_norm_net.features.denseblock4.denselayer14.layers.norm2.bias_epoch=39.00, grad_2.0_norm_net.features.denseblock4.denselayer14.layers.conv2.weight_epoch=2.12e+3, grad_2.0_norm_net.features.denseblock4.denselayer15.layers.norm1.weight_epoch=51.40, grad_2.0_norm_net.features.denseblock4.denselayer15.layers.norm1.bias_epoch=47.40, grad_2.0_norm_net.features.denseblock4.denselayer15.layers.conv1.weight_epoch=1.14e+3, grad_2.0_norm_net.features.denseblock4.denselayer15.layers.norm2.weight_epoch=51.90, grad_2.0_norm_net.features.denseblock4.denselayer15.layers.norm2.bias_epoch=38.30, grad_2.0_norm_net.features.denseblock4.denselayer15.layers.conv2.weight_epoch=2.07e+3, grad_2.0_norm_net.features.denseblock4.denselayer16.layers.norm1.weight_epoch=45.70, grad_2.0_norm_net.features.denseblock4.denselayer16.layers.norm1.bias_epoch=41.90, grad_2.0_norm_net.features.denseblock4.denselayer16.layers.conv1.weight_epoch=1.03e+3, grad_2.0_norm_net.features.denseblock4.denselayer16.layers.norm2.weight_epoch=49.90, grad_2.0_norm_net.features.denseblock4.denselayer16.layers.norm2.bias_epoch=36.50, grad_2.0_norm_net.features.denseblock4.denselayer16.layers.conv2.weight_epoch=2e+3, grad_2.0_norm_net.features.norm5.weight_epoch=2.11e+3, grad_2.0_norm_net.features.norm5.bias_epoch=3.64e+3, grad_2.0_norm_net.class_layers.out.weight_epoch=8.85e+4, grad_2.0_norm_net.class_layers.out.bias_epoch=8.91e+3, grad_2.0_norm_total_epoch=9.96e+4]
How do I actually know if there is vanishing gradient? For examples, gradient at the end of the network in conv2 layers seem pretty low. Also, is there any way that I can get a nicer output of this? Thanks.