Visualization During Training

[FabianSchuetze]

Thanks for the wonderful work, it is a pleasure to use mmsegmentation.

Can I somehow visualize the training and validation predictions during the training process? I would like to dump the predicted images periodically to the hard-disk, (or even better to wandb). Should I maybe use hooks from mmgeneration for that? Could they be linked with wandb? If this is not possible currently, would you welcome a PR for that?

[xiexinch]

Hi @FabianSchuetze,
You might try to implement a visualization hook. #1710

[ayulockin]

Hey @FabianSchuetze you can try to use MMSegWandbHook. You can find more info in the PR that added this feature.

[xiexinch]

Hey @FabianSchuetze you can try to use MMSegWandbHook. You can find more info in the PR that added this feature.

Hi @FabianSchuetze,
Sorry I almost forgot about this feature, WandbHook should meet all your needs.

[FabianSchuetze]

Thanks for the kind and informative replies!

Indeed, I missed that wandb already logs the predictions and some images, but I am not sure the results are very informative (at least with my settings). I use the following settings:

cfg.log_config.hooks.append(
        dict(type='MMSegWandbHook',
         init_kwargs={'project': 'MMDetection-tutorial'},
         log_checkpoint=False,
         num_eval_images=1,
         interval=eval_interval))

With these settings, two problems arise for me:

• During the evaluation, only the predicted mask is saved, but not the image
• The mask is not normalized so that the visualization in the browser is all-black for me (I have two classes, and the outputs are zero and one).

I think this behaviour stems from the following functions: _log_predictions seems to be periodically called, but only the predicted mask is added, not the ground truth mask and neither the input image. Furthermore, pred_mask is copied as outputed by the model, and no normalization (color palette) is applied.

The function _add_ground_truth which logs the input images is only called once in my settings, at the very beginning of a training process.

Should I somehow change my settings or do you have any other suggestions for me, @ayulockin? Thanks again for your help!

[ayulockin]

Hey @FabianSchuetze, apologies for the delayed reply.

Mechanics of logging

The function _add_ground_truth which logs the input images is only called once in my settings, at the very beginning of a training process.

This is true; it logs the ground truth image + associated masks. We need to call this once as the evaluation samples need not change (ideally).

I think this behaviour stems from the following functions: _log_predictions seems to be periodically called, but only the predicted mask is added, not the ground truth mask and neither the input image.

We call _log_predictions periodically to log the predictions of the model. The ground truth is if you notice is logged as well but by using referencing to the table logged when _add_ground_truth is called. The self.data_table_ref is how we are getting reference to the self.data_table logged at before_run.

Furthermore, pred_mask is copied as outputed by the model, and no normalization (color palette) is applied.

Yes we are not normalizing the mask. The output of model is unnormalized mask always? Can we dynamically figure out that the mask is normalized or not?

I hope your concern around During the evaluation, only the predicted mask is saved, but not the image is resolved. Note that the image once logged during _add_ground_truth is referenced when calling _log_predictions.

Mask issue

The mask is not normalized so that the visualization in the browser is all-black for me (I have two classes, and the outputs are zero and one).

If you have two classes with zero and one, I think it should ideally render the mask properly. It might be a bug in MMDetWandbHook or the W&B UI. In either case, will you be able to share the W&B run page where you are logging the evaluation so that I can take a look?