Hateful meme detection is not only fundamental in governing a healthy online environment, but it is also an important research topic that requires both visual and linguistic modeling skills and advanced knowledge of effectively combining multimodal representations. In this paper, we utilized data from Meta’s Hateful Meme Detection Challenge, which contains examples of memes that can be successfully classified only when their text and images are considered coherently. We built three models, a baseline, a VisualBERT, and a VisualBERT with external feature extraction (Model 3). By leveraging large pre-trained models and fine-tuning, our best model, Model 3, achieves a 62.4% accuracy. We presented our findings, analyzed and compared the results both quantitatively and qualitatively, and discussed potential future steps.
Please see Poster.pdf for final poster, and final_report.pdf for final report