I need help! (Dataset Quality)

[u/KneeOpening]

Hi there!

Need help from some Computer Vision expert, with knowledge on dataset quality.

My company works on computer vision applied to digital pathology. In this case, we are developing an object detection model to detect tumour cells in HER2-stained breast tissue and classify them into four classes: 0, 1+, 2+, and 3+. Our dataset grows as we incorporate cases from new hospitals or scanners, and we call each new batch a “wave”.

Example annotations

Our latest model achieves around 0.73 macro F1 on the combined test set, only slightly above our internal acceptance threshold of 0.7. Performance also varies substantially across waves and classes. We already have approximately 4,500 image patches with almost one million cell annotations. The overall class distribution is approximately 28.5% class 0, 48.3% class 1+, 12.3% class 2+, and 10.9% class 3+.

We suspect that inconsistent annotation criteria may be contributing to this. Our current annotation guidelines are very brief—around 200 words—and provide neither image examples nor detailed guidance on uncertain cases. The boundary between 1+ and 2+ is particularly difficult, and we are concerned that different pathologists may interpret it differently. In our latest confusion-matrix analysis, 21.9% of cells annotated as 2+ are predicted as 1+. Different pathologists have contributed over time, but changes in annotators are mixed with changes in hospitals, staining and scanners, making the causes difficult to distinguish.

I would really appreciate advice on two questions:

1. What minimum experiment would you recommend to investigate inconsistent annotation criteria versus domain shift or model errors, before involving the pathologists in a new review? We have images, original annotations and predictions from two models evaluated on the same cohort.

2. What would you consider essential in an annotation guideline and a quality check for each new wave? In particular, we would like to establish agreed examples of class boundaries, clear rules for uncertain or unannotated cells, and a small inter-annotator agreement assessment before adding new data to training.

Any references, suggestions or critical feedback would be greatly appreciated.

Thanks in advance!