My hope with this post is that I will save at least one person some time - and that will be enough for me. I spent the last couple of weeks building an auto-labelling pipeline on SAM 3 and figured the gotchas were worth writing down, because most of what I got wrong had nothing to do with the model.
Quick context if you haven't used it: SAM 3 does what Meta calls Promptable Concept Segmentation. You give it a short noun phrase - forklift, person in hi-vis vest - and it segments every instance of that concept. No seed clicks, no fixed class list, no fine-tuning. That's the bit that makes unattended labelling possible; with SAM 2 you still needed something to tell it where to look.
The minimal version is genuinely this short:
from transformers import Sam3Model, Sam3Processor
model = Sam3Model.from_pretrained("facebook/sam3").to("cuda").eval()
processor = Sam3Processor.from_pretrained("facebook/sam3")
inputs = processor(images=image, text="forklift", return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model(**inputs)
results = processor.post_process_instance_segmentation(
outputs, threshold=0.5, mask_threshold=0.5,
target_sizes=inputs["original_sizes"].tolist(),
)[0]
# results["masks"] / ["boxes"] / ["scores"]
That works. Everything below is what I learned scaling it past one image.
1. Reuse the vision embedding across prompts
Naive multi-class loop encodes the image once per class. 3 classes × 40k images = 120k passes through an 848M-param backbone, 80k of which recompute something you already had. SAM 3 lets you split it:
vision_embeds = model.get_vision_features(pixel_values=inputs.pixel_values)
for prompt in prompts:
text_inputs = processor(text=prompt, return_tensors="pt").to(model.device)
outputs = model(vision_embeds=vision_embeds, **text_inputs)
Backbone runs once, only the text conditioning and mask decode repeat. Close to an N-fold speedup on multi-class jobs. There's a mirror version (get_text_features) for one prompt across many images.
2. Resolution is tricky
SAM 3 runs at 1008px native. Two failure modes:
- Upscaling small images to 1008 gives you confidently mushy boundaries. It adds no information.
- Downscaling big images destroys small objects. A 40px defect in a 4000px frame becomes a 10px smudge at 1008. If your targets are tiny, tile into overlapping 1008px crops and merge masks back with the offset. Don't resize.
Also: run ImageOps.exif_transpose() before anything else, or phone photos come back with masks correct for the stored orientation and wrong for the one you see.
3. Prompt phrasing does more than threshold tuning
Short concrete noun phrases. Singular. One concept per prompt.
- forklift ✅ / find all the forklifts ❌
- person in hi-vis vest ✅ / PPE compliant worker ❌ (trained on how things look, not your industry's vocabulary)
- car or truck ❌ - that's two prompts
Biggest thing: test each prompt against images you know contain none of that class. A prompt that quietly fires on empty frames poisons the whole dataset. And if a prompt over-fires, add an adjective before you touch the threshold - white bicycle vs bicycle returns genuinely different sets.
4. You can sweep thresholds without re-running inference
The detection threshold is just a filter over stored confidence scores. So label a 50-image dev slice once at threshold=0.15, keep every score, and sweep offline.
Look for the false-positive cliff and stop just above it. If med area% collapses as you lower the threshold, the extra detections are specks - raise a minimum-area filter instead. If empty stays high at every threshold, your prompt is wrong and no threshold will save it. (The mask threshold can't be swept this way - it changes pixels, not scores.)
5. Small export things that cost me an hour each
- pycocotools.mask.encode() needs np.asfortranarray(). Pass a C-ordered array and you get a silently transposed mask. No error.
- The RLE counts field is bytes; json.dumps refuses it. Decode to ASCII.
- For YOLO, write an empty .txt for images with no detections. Missing file = missing data; empty file = confirmed negative, which is how the model learns not to hallucinate.
6. Look at the labels
Auto-labelling fails quietly - no exceptions, no bad metrics, just a pallet prompt that's been segmenting the wooden floor for 12,000 images. Render a contact sheet of overlays sorted lowest confidence first and actually look at it. Ten seconds catches what an aggregate metric won't.
That's it. Hopefully I saved you guys some time and feel free to ask questions!
UPDATE: since I got a couple of similar questions about the auto-labelling pipeline in my DMs, I posted a full write up of it here . If you are curious about how to get the best results when auto-labelling - feel free to check it out.
