Stuck on the problem since 2 months

[u/Senior-Ad1636]

I am new to computer vision and I am trying to make a pipeline which can predict if a hospital beds rail is up or down , The bed looks somewhat like the image attached , What I currently have
- A live stream from my camera
- A 3d simulation enviornment mimicing the bed closely
- A finetuned Model which gives me bed segment mask
- A finetuned model giving me bbox of rails
considering the wide variety of positions of camera and camera angles and occlusion
Is this a 2d spatial problem which is almost unsolvable with current landscape ?

[u/a_i_g]

You have a 3d model of the bed? Is it always the exact same bad type? If so you can use render&compare method

[u/Senior-Ad1636 → u/a_i_g]

I have the static 3d model in which I cant change position of rail up or down, forcing it via astra the depth is always off when compared to real examples

[u/a_i_g → u/Senior-Ad1636]

You can then render many random (few hundreds) of hypotheses. Feed these and the image to a backbone (ResNet, ViT) and find the cosine similarity between the image and the hypotheses. Then get the best one. Do the same thing again with hypothesis that are around that closest hypothesis to refine the prediction.

[u/a_i_g → u/a_i_g]

A backbone trained on Image net should be okay. Something more special could be better. You need something trained for object identification. If you think this approach is good I can give you clues for that.

[u/DiddlyDinq]

Can it not be implicitly derived by tracking whether a person is in the bed and their bounding box and skeletal movements? For example, their spine angle or feet elevation over time

[u/Senior-Ad1636 → u/DiddlyDinq]

So without a person on the bed theres no way to do it ?

[u/DiddlyDinq → u/Senior-Ad1636]

It's just one way. I dont really know the conditions of your real world use case or how much of a difference the real world bed is from your model. Whether the sides will be covered in blankets, whether it will be occluded etc. Are you aiming for this should work in any hospital from any angle as long as that bed is in the scene, occupied or not

[u/Senior-Ad1636 → u/DiddlyDinq]

ya thats my aim , atleast lets say i fix camera towards the footend of the wall someplace not side walls

[u/UmutIsRemix]

What view do you exactly have of the bed though? You could "track" certain parts of the bed itself. You could slap a marker on it and track that marker? Since you already have bboxes of rails you can annotate those for classification and have a simple classifier predict up or down? You could perhaps do image matching since you get bed segments anyway? I might not be experienced enough but you also gotta tell us what you tried I guess?

[u/Senior-Ad1636 → u/UmutIsRemix]

thats the issue camera is usually placed on foot end but in some rooms we might have to place it on the side , but 80 % on foot-end walls but camera can rotate and markers would have no idea of space until we get depth cameras or something

[u/Chemical_Side_4135]

that sounds rough, have u tried breaking the pipeline into smaller chunks to see exactly where its failing

[u/Senior-Ad1636 → u/Chemical_Side_4135]

some pipelines work well when bed is flat and stop working when its inclined / declined or camera goes to complete side view