I need help! (Dataset Quality)

[u/KneeOpening]

Hi there!

Need help from some Computer Vision expert, with knowledge on dataset quality.

My company works on computer vision applied to digital pathology. In this case, we are developing an object detection model to detect tumour cells in HER2-stained breast tissue and classify them into four classes: 0, 1+, 2+, and 3+. Our dataset grows as we incorporate cases from new hospitals or scanners, and we call each new batch a “wave”.

Example annotations

Our latest model achieves around 0.73 macro F1 on the combined test set, only slightly above our internal acceptance threshold of 0.7. Performance also varies substantially across waves and classes. We already have approximately 4,500 image patches with almost one million cell annotations. The overall class distribution is approximately 28.5% class 0, 48.3% class 1+, 12.3% class 2+, and 10.9% class 3+.

We suspect that inconsistent annotation criteria may be contributing to this. Our current annotation guidelines are very brief—around 200 words—and provide neither image examples nor detailed guidance on uncertain cases. The boundary between 1+ and 2+ is particularly difficult, and we are concerned that different pathologists may interpret it differently. In our latest confusion-matrix analysis, 21.9% of cells annotated as 2+ are predicted as 1+. Different pathologists have contributed over time, but changes in annotators are mixed with changes in hospitals, staining and scanners, making the causes difficult to distinguish.

I would really appreciate advice on two questions:

1. What minimum experiment would you recommend to investigate inconsistent annotation criteria versus domain shift or model errors, before involving the pathologists in a new review? We have images, original annotations and predictions from two models evaluated on the same cohort.

2. What would you consider essential in an annotation guideline and a quality check for each new wave? In particular, we would like to establish agreed examples of class boundaries, clear rules for uncertain or unannotated cells, and a small inter-annotator agreement assessment before adding new data to training.

Any references, suggestions or critical feedback would be greatly appreciated.

Thanks in advance!

[u/Exciting-Hearing-794]

With that many annotations and a confusion matrix already, you got more than enough to find where the trouble is without bothering the pathologists yet. The 21.9% of 2+ predicted as 1+ is screaming annotation inconsistency, not model error, but domain shift can still mess things up in specific waves. I'd pull all the 2+ ground truth cases that the model confidently called 1+, then check if those misclassified cells cluster by wave or by annotator era. If it's by wave, probably a staining or scanner problem, if it's by annotator era, then your guidelines are definitely too vague

For the guidelines, 200 words is nothing for a 4-class problem where the boundaries are this subjective. You need a reference sheet with at least 3-4 example images per class, including the tricky borderline cases with arrows pointing to why it's one class and not the other. Before each new wave, have two pathologists annotate the same small subset of like 50-100 cells and calculate Cohen's kappa, if it drops below 0.8 you know you got a problem before the data even hits training

[u/KneeOpening → u/Exciting-Hearing-794]

Nice, thank you, I will work on that evaluation you suggest and come back with insights. Sadly, waves are not separated from annotator eras.

We have to improve our annotation procedure, for sure, so some follow up questions:

• Could you provide some reference of annotation guidelines / reference sheet? So that I can help the lead pathologist understand
  
• I really like the idea of assessing if 'everyone it on the same page' before starting a wave. Maybe that helps to iterate on the guidelines or do something about poor performance from someone. But I have a couple of questions: kappa is for classification, so any idea for doing something similar for object detection? And would you tell the pathologists or do this without telling them?

[u/Dry-Snow5154]

Regarding annotation quality, can you just do a random sampling from each wave and reannotate it yourself? And then check correctness of the original annotation. You can calculate the exact sample size needed to be statistically significant, but for a wave of 500 images, I'd say you can check like random 20.

Generally, models have enough capacity to fit several domains, if detecting the domain is simpler than the task itself. So when you add domain 2 to training, metric usually at least doesn't go down (significantly) in domain 1 (should improve, really). But when annotations in domain 2 are contradicting, metric will likely go down. So if you have historical model scores you can check Model N on Eval N vs Model N+1 on Eval N (last one you probably do not have and will have to run). If there is a big drop, then wave N+1 likely has inconsistent annotations. No retraining needed.

I don't have an advice regarding guidelines. From my experience annotator's quality is responsible for 90% of the outcome, rather than any detailed guides, review strategies and agreements.

[u/KneeOpening → u/Dry-Snow5154]

Thanks for your reply! We compared the wave-2 and wave-3 models on the same test cohort. Macro F1 improved on every wave:

Test subset	Test patches	Wave-2 model	Wave-3 model
Wave 1	181	0.581	0.625
Wave 2	36	0.480	0.582
Wave 3	249	0.510	0.662

Given this trend, would you expect further improvement as we add more data, even if some annotation inconsistencies remain?

And regarding the reannotation you mention, it would require pathologists, since we cannot reliably judge the HER2 classes ourselves, but a small review is feasible. Our dataset contains 1,677 patches from wave 1, 327 from wave 2, and 2,438 from wave 3 — 4,442 overall. Would you start with 20 random patches per wave, spread across different slides, or stratify the sample by class and/or model disagreement?

[u/Dry-Snow5154 → u/KneeOpening]

Per-wave eval indicates your task is just hard. Doesn't look like big annotation contradiction. Improvements are consistent across waves, slightly more on Wave 3 eval, but this is expected. Most likely you simply need more data to improve further.

One other thing you can check is metrics isolated per class in the same setup. Like, does confusion between 1-2 go up for Wave-1/2/3 model on Wave-1 eval? Maybe improvement on classes 3/4 is obscuring regression on classes 1/2. I doubt tho.

If you can't annotate data yourself, then sampling check is tricky. Unless you have some gold standard pathologist that you can 100% rely on. You can ask several pathologists and cross-review their annotations, but this is getting expensive fast. It also appears I was way off with 20 sample, you need more like 100 for 90% confidence for population of 2000. Best stratified by class too.

[u/19pomoron]

I wonder if it's possible to train iteratively (model 1 - wave 1, model 2 - wave 1+2, model 3 - wave 1+2+3...) . This enables plotting the change of F1 (or other metrics) as the waves progresses, and then plot the progression of F1 for each category.

This would be able to figure out which wave of new data causes the problem, regardless of whether it's the model's or annotator's fault.

Then further investigation can begin. A possible culprit is that there are new appearances of class 1 and 2+ instances that causes confusion previously not observed, or maybe the background/image conditions that make models more prone to cause confusion.

Also if a certain wave of prediction causes the most drop-off, OP can ignore this wave, and train models of class 1 and 2+ using data not in this wave. Then maybe find overlaps and further investigate into non-matching samples.

Coz with these dense detections, despite good guidelines, I fear that there can be a chance that the annotator would have a lapse of moment, make errors and snowballs as labelling progresses.

[u/KneeOpening → u/19pomoron]

Thanks for your answer. If you check my answer to Dry-Snow5154 , I give the metrics you mention. Performance seems to keep scaling with dataset growth (despite confusion between class 1+ and 2+). No wave seems to cause a drop-off in performance.

About the confusion you talk about: maybe it does disappear with dataset size?

[u/bob_why_]

Out of interest have you tried working on the preprocessing side, this can also be used as a guide to cluster annotation reasonings.

[u/sam9510]

Are your labels intended to describe individual-cell membrane staining characteristics, or are they adaptations of the clinical HER2 IHC scoring categories?

I ask because HER2 scoring criteria are based on staining intensity, membrane completeness, and the proportion of invasive tumor cells showing particular staining patterns. These criteria may not necessarily translate directly into an independent four-class decision for every 'cell'. (this is my limited knowledge - i could be totally wrong here).

If you're adapting those categories to cell-level annotations, perhaps your pathologist could define the cell-level decision rules explicitly - and map out how scores translate to those rules (or vice versa).

If you are concerned about annotation quality - pull random samples and have your experts re-annotate it blindly (they dont see prior annotations or model outputs) - and calculate how much they differ.

https://pmc.ncbi.nlm.nih.gov/articles/PMC13431868/