r/computervision • u/Substantial_Border88 • May 21 '26
Help: Theory Did SAM3 changed the Image Annotation game completely?
Recently auto-annotation has been commoditised, which means, due to the advancements in Foundation models like SAM3, Dino family and also VLMs like Gemini 3.0 Flash, T Rex + Models from IDEA Research ; it has become much easier to generate bounding boxes and use them to train domain specific models. Review and QA of AI generated annotation surely becomes a bottleneck as no model is 100% accurate in whatever it sees.
I have annotated hundreds of images manually a couple of years ago and it feels much easier than before to use AI to annotate, but the ChatGPT moment still seems really far.
The importance of the following question will be felt by everyone in this sub and everyone who trains specialised models professionally or for hobby.
Like LLMs have a huge scope of fine tuning and pre training specialised models for specific use cases, do vision models still have similar scope where people will keep training Object Detection models for their use cases? Or there will be a time where some AI lab will launch an efficient enough model which will detect anything without any pretraining or finetuning.?
Consider this an open discussions, suggest techniques or simply act on your insecurities of gradually becoming obsolete( hehe)
16
u/0bi_nx May 21 '26
There might come a time where a super model is launched that understands images like humans do, but as of now we are far away from it. Imo these models will also not be trained on images alone, but videos, 3D scans and probably be multimodal.
For now, SAM3 is decent for general domain stuff, but still fails on niche domains. I think for most use cases you still want to train your own custom model on top of a good backbone.
LLMs are much simpler to train on all sorts of tasks, because everything can be formulated as text. For vision, you need different formats and there is no common language.
3
u/fixitchris May 21 '26
The 'no common language' point is real, though patch embeddings are getting close to a lingua franca for vision. Domain-specific stuff wrecks that fast though, SAM3 sees something its training never touched and guesses confidently wrong. Still running fine-tuned models on a SAM backbone for anything production-critical.
-1
u/0bi_nx May 22 '26
Imo world models will allow much better performance, but their training takes immense efforts. I guess we will see more models like that in the near future tho.
1
u/fixitchris May 22 '26
The training cost is real, but the bigger bottleneck I've seen is data curation. Getting clean, consistent annotation at the scale world models need takes longer than the compute time in most shops I've worked with. Once you actually have one, though, the frame-to-frame consistency improvements in video tasks are a pretty significant jump.
2
u/Loud_Ninja2362 May 21 '26
SAM3 fails hard on out of domain or specific targeted objects. If your just trying to get general stuff in an image it's great. But once you get outside of that it requires a lot of fine tuning or full on training runs.
2
u/corevizAI May 22 '26
Humans don’t necessarily “understand images” the same way either, for example, your average joe wouldn’t be able to annotate breast cancer on an x-ray whereas a trained oncologist would. That’s why we believe that a Visual AI harness that can orchestrate and choose between different vision models is what’s needed today, and that’s exactly why we built coreviz :)
2
u/0bi_nx May 22 '26
I like the approach. Is your AI just trained to predict the image domain? It probably suffers from the same problems tho: what if your AI encounters a domain it hasnt seen in training yet?
2
u/corevizAI May 22 '26
It essentially has access to 50k+ public models that folks have already trained / fine tuned for various use cases, that on top of VLMs and general models like SAM3 and Florence. If that’s not sufficient (rarely the case), you can train/fine tune your own detection/classification/segmentation/embedding model and simply use the coreviz harness to bulk apply it to your image collection.
3
u/0bi_nx May 22 '26
Are these community models then? How do you validate their performance/ robustness?
2
u/corevizAI May 22 '26
That’s correct, we’re just starting so community models are a great start but we’re intending to fine tune our own “expert” models and we’ll be publishing their accuracy and robustness as we go. But the underlying model orchestration will remain the same regardless.
1
u/0bi_nx May 22 '26
Very interesting approach. Basically, a mixture of experts model in a way. Do you have any papers or something on how performance compares?
7
u/c0mbatduckzz May 21 '26
Ontology is a bitch. Are you looking at your phone? The screen? The pixels? Etc. it gets real muddy real fast, especially when you take into account fictional stuff or domain specific knowledge.
5
u/indieGoatRocket May 21 '26
I used sam3 for pre annotation on crowded scenes in train carriages. Needed a lot of manual post labelling, due to occlusions and bad cctv image quality. But it worked better then Qwen 3 vl :)
1
u/Substantial_Border88 May 21 '26
Did you use any technique for post labelling review or correcting the annotations?
2
2
u/HistoricalMistake681 May 21 '26
In some ways, it has helped speed up annotation pipelines but a lot of important applications have specific niche annotation requirements and I’ve seen SAM understandably fail in those cases.
2
u/Aryan_Chougule May 22 '26
I have been using sam3 since the launch and honestly its the best autoannotations tool if you get the perfect prompt for your object it works fantastically but there is one problem I think that is that it requires GPU with vram of 9 GB to load, it can't be loaded on a 8 GB non GPU system.
2
u/ResponsibilityNo7189 May 22 '26
I think it is moving farther than that. We recently found out that Nano Banana Pro can generate both images and semantic segmentation mask, in a single prompt. We reach a point where you can start training network with zero real data, with large models acting as prompt-based training data factory. Check our paper here.
1
u/Any_Frame9721 May 22 '26
We made a deployment tutorial for SAM3. Crosss-posting here:
One script (https://docs.embedl.com/embedl-deploy/latest/auto_tutorials/sam3.html) that only requires a Python package with the only dependency being PyTorch. The script takes you from a HuggingFace checkpoint to running TensorRT engine export, fusions, quantization, compilation.
Use a smaller image size for Nano.
The performance:
NVIDIA Jetson AGX Orin Image size Latency
224×224 → 40.4ms / 24.7 FPS (real-time)
448×448 → 118.5ms INT8, 10% faster than FP16
672×672 → 187.6ms INT8, 27% faster than FP16
NVIDIA Jetson Orin Nano
224×224 → 89.6ms / 11.2 FPS
448×448 → 262.6ms INT8, 20% faster than FP16
1
u/Many_Apple_4481 May 24 '26
Not drastically. For simple objects, yes sure. But for complex new concepts, you still need a human to do it. It can still facilitate the human labeling process even for new concepts, but didn't really revolutionize the game.
0
u/mongoOzzy May 22 '26
We might always need task-specific distilled models for efficiency. The large transformers take seconds for inference, not fast enough for real time vision.
But having an unsupervised training loop to automatically create the distilled model would be awesome - like type I, type II thinking.
I have done this recently with Qwen and it wasn't good enough, still needed human review of annotations.
1
u/Substantial_Border88 May 22 '26
Review is the real bottleneck it seems.
Have you used any advanced techniques for review or just manual human review?
-1
u/Acrobatic_Limit9108 May 21 '26
How do you automate annotating 1000 of images let’s say 6-8k images? One thing I noticed is you can use LLM’s to do a few hundred but that large data count is still far away.
1
u/Substantial_Border88 May 22 '26
You can use foundation vision models like Grounding Dino or OwlV2. OwlV2 basically takes an image as a crop and detects similar objects. It works flawlessly here - Imflow.xyz
40
u/GFrings May 21 '26
Frankly Sam2 did, whereas sam1 was a bit rough around the edges