Misogyny and hate content have become a serious issue in the current digital era. Such content predominantly manifests as short video clips, making automated detection mechanisms imperative for social media platforms. To effectively classify such videos, a multimodal setup is required, as video data contains image frames, audio clips, and text captions. The multimodal misogyny video classification task aims to combine all audio, visual, and extracted text information. Conventional methods typically aim to process this extracted information in parallel to reach a conclusion. Our proposed framework utilizes the pre-trained models DINOv2, wav2vec 2.0, and MuRiL-large to embed visual, audio, and text features, respectively. To mitigate overfitting on datasets of different sizes, we propose a dual-architecture approach: we utilize a regularized Gated Early-Fusion Network (EF-Net) for the highly constrained Tamil dataset, and a deeper Cross-Modal Temporal Attention Network (TA-Net) with an ensemble strategy for the Malayalam dataset. Ultimately, our framework achieved a 68.8% macro-F1 score for the Malayalam dataset and an 80.7% macro-F1 score for the Tamil dataset, securing Rank 1 in misogyny video classification for both languages.