Noisy file detection method and its design

[pythonista-blitz]

Hi, I want to detect noisy files after learning and get their paths and save them.
I was originally planning to implement this during test_step()/on_test_end(). As follows:

    def on_test_epoch_start(self) -> None:
        super().on_test_epoch_start()
        self.test_step_outputs = []
        return

    def test_step(self, batch: Any, batch_idx):
        inputs, targets = batch
        preds = ...
        self.test_step_outputs += [{"preds_probs": preds, "targets": targets}]

    def on_test_end(self) -> None:
        preds_probs = (
            torch.cat([x["preds_probs"] for x in self.test_step_outputs], dim=0).cpu().numpy()
        )
        targets = torch.cat([x["targets"] for x in self.test_step_outputs], dim=0).cpu().numpy()
        noisy_labels = find_label_issues(targets, preds_probs, filter_by="confident_learning")
        noisy_indices = [str(i) for i, x in enumerate(noisy_labels) if x]
        # I want to get noisy filenames here.
        # And save .txt files

within the datamodule I am using Pytorch's ImageFolder class to generate the batch. However, I don't know how to access the datamodule from within the LightningModule class and how to specify the run folder generated by sensorboard. Also, I don't think the datamodule class and lightningmodule class should be tightly coupled in the first place.

1. Is there any way to access the filename in the datamodule from lightningmodule?
2. Where should such a mechanism be implemented in the design in the first place?
3. Is there a way to specify the folder path generated by tensorboard?

We apologize for asking so many different questions at once, but we welcome any suggestions as long as they can be answered.

[ashleve]

1. After training or testing starts you can access self.datamodule inside LightningModule.
2. I think saving files from on_test_end hook is fine
3. You can access loggers through self.logger (if there's one) or self.loggers list (if you use many at once). Take a look at docs to see what tensorboard properties are accessible.

The cleanest approach would probably be to write a callback dedicated to saving those images (you can access both datamodule and lightningmodule inside the callback hooks). This way you could decouple datamodule-related behavior from ligtningmodule.