Description
Hi,
We are training a 3d_fullres model. There are six chunks for validation. Looking at the dice score, five of them return a good value (>.95), but one returns 0 and the overall loss is also larger. This makes the progress look like overfitting but the predicted label for all six validation chunks looks close to the ground truth.
Two themes of questions:
1.) What is it about this data chunk that is throwing off the dice score for one but not the others?
- A trend we noticed is that this chunk has a lot more "ignore" annotations, where entire frames of the 3d stack are supposed to be excluded from the training. Perhaps someone is familiar with common errors people make in ignoring data, like maybe it gets thrown off when entire frames are missing?
- chunk size is 590 x 433 x 512, slices 422-505 is entirely assigned to class "ignore". There is background, ignore, and two other pixel classes used in the training. Some frames are also entirely "background".
- Is the dice score calculated frame by frame within a chunk and then somehow averaged? I wonder if one frame has a dice score of zero if this would somehow make the "average" zero (say if it was a geometric mean).
2.) Even though our model overall looks good, is it possible these zero dice scores are hindering the training?
We are training five different folds for this model, so this test data will be part of the training in the other folds. Perhaps fixing this error would further improve our final conglomerate model.
Attached are the dice scores. Search for "1887_2398" to find where it reports zero.
Any help much appreciated. Thanks for making nnUNet available to everyone
All best,
Steve