I haven't looked at cogvlm, but if you mean doing bounding boxes w/ classification, I'd love to support models like that (like detectron2) in the future.
"what are the coordinates of the bounding box for the rubber duck in the image [img]" >>> "[10,50,200,300]"