Multimodal contrastive models are widely used for bidirectional image-text retrieval, but their rankings remain vulnerable to attraction bias, where certain candidates become hubs for many unrelated queries. Existing training-free normalization methods suppress such hubs, but may also penalize candidates that are genuinely relevant. We propose GLOBE, a training-free plug-in module that combines global-local bias calibration with graph-based semantic evidence to suppress unsupported hubs while preserving true matches. Across MS-COCO and Flickr30K, GLOBE improves diverse frozen retrieval backbones, achieving up to 7.9% absolute Recall@1 gain over the original model and up to 5.2% over strong training-free baselines. We further construct IRB-COCO-500, a mixed real-and-synthetic benchmark with AI-generated near-positive distractors, and show that GLOBE improves retrieval robustness in this challenging setting.