Hi all, I’m trying to use lighting together with Ray to perform hyperparameter search.
I already had a fully functioning pipeline where I tested a couple of models, and it worked fine / as intended.
However, when using Ray, it throws some errors about missing files, which I’m being unable to debug, as the files do exist, and they are being loaded by the Dataloaders themselves.
Does anyone know if it’s possible to use LightningModule and LightningDataModule along with ray, and if so, what error am I doing?
Here are the dataloaders
Here is the Ray code, adapted from the examples they provide
And here is the trace, where the image is now being loaded inside Ray:
(pid=3520770) 2021-11-04 18:20:11,285 ERROR function_runner.py:268 -- Runner Thread raised error.
(pid=3520770) Traceback (most recent call last):
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/ray/tune/function_runner.py", line 262, in run
(pid=3520770) self._entrypoint()
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/ray/tune/function_runner.py", line 330, in entrypoint
(pid=3520770) return self._trainable_func(self.config, self._status_reporter,
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/ray/util/tracing/tracing_helper.py", line 451, in _resume_span
(pid=3520770) return method(self, *_args, **_kwargs)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/ray/tune/function_runner.py", line 599, in _trainable_func
(pid=3520770) output = fn()
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/ray/tune/utils/trainable.py", line 344, in inner
(pid=3520770) trainable(config, **fn_kwargs)
(pid=3520770) File "/tmp/ipykernel_3514805/1116110716.py", line 45, in train_prostatex_tune_checkpoint
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 552, in fit
(pid=3520770) self._run(model)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 922, in _run
(pid=3520770) self._dispatch()
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _dispatch
(pid=3520770) self.accelerator.start_training(self)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
(pid=3520770) self.training_type_plugin.start_training(trainer)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
(pid=3520770) self._results = trainer.run_stage()
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1000, in run_stage
(pid=3520770) return self._run_train()
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run_train
(pid=3520770) self._run_sanity_check(self.lightning_module)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1122, in _run_sanity_check
(pid=3520770) self._evaluation_loop.run()
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
(pid=3520770) self.advance(*args, **kwargs)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance
(pid=3520770) dl_outputs = self.epoch_loop.run(
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
(pid=3520770) self.advance(*args, **kwargs)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 94, in advance
(pid=3520770) batch_idx, batch = next(dataloader_iter)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
(pid=3520770) data = self._next_data()
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
(pid=3520770) return self._process_data(data)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
(pid=3520770) data.reraise()
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
(pid=3520770) raise exception
(pid=3520770) FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
(pid=3520770) Original Traceback (most recent call last):
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/torchio/data/io.py", line 25, in read_image
(pid=3520770) result = _read_sitk(path)
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/torchio/data/io.py", line 56, in _read_sitk
(pid=3520770) image = sitk.ReadImage(str(path))
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/SimpleITK/extra.py", line 346, in ReadImage
(pid=3520770) return reader.Execute()
(pid=3520770) File "/home/aime/miniconda3/envs/nuno/lib/python3.8/site-packages/SimpleITK/SimpleITK.py", line 8015, in Execute
(pid=3520770) return _SimpleITK.ImageFileReader_Execute(self)
(pid=3520770) RuntimeError: Exception thrown in SimpleITK ImageFileReader_Execute: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:97:
(pid=3520770) sitk::ERROR: The file "Images/Original/Volume0133.nii.gz" does not exist.
Greatly appreciate all help.