​​​​​In the 70th session of Multimodal Weekly, we had two exciting presentations on multilingual multimodal model and text-video retrieval.
​​​​​​​​✅ Nahid Alam, Karthik Reddy Kanjula, and Surya Guthikonda will present their work Maya, a new multimodal multilingual vision-language model. It is completely open source, open weight and open dataset, designed to handle 8 languages, cultural diversity, and nuanced real-world contexts in vision-language models.
- Connect with Nahid: www.linkedin.com/in/nahidalam/
- Connect with Karthik: www.linkedin.com/in/karthik-reddy-kanjula/
- Connect with Surya: www.linkedin.com/in/surya-guthikonda/
- Maya: github.com/nahidalam/maya
​​​​​​​​​​​​​​​​​​​​​​​​✅ Jiamian Wang will present his work called "Diffusion-Inspired Truncated Sampler for Text-Video Retrieval" - in which he devised a diffusion-inspired iterative alignment process to solve for the multimodal modality gap and achieves encouraging performance.
- Connect with Jiamian: jiamian-wang.github.io/
- DITS: openreview.net/forum?id=SrQua0ATRZ&referrer=%5Bthe%20profile%20of%20Pichao%20WANG%5D(%2Fprofile%3Fid%3D~Pichao_WANG3)
​​​​​​​​
Timestamps:
00:15 Introduction
03:20 Nahid starts
04:35 The challenge
06:05 Maya's solution
07:00 Karthik talks about dataset preparation
09:40 Multilingual pre-train preamble preparation
11:00 Preamble evaluation
11:35 Multilingual pre-train dataset translation
13:25 Surya talks about dataset toxicity analysis
14:05 Text toxicity analysis
15:07 Image toxicity analysis framework
16:05 Toxicity in LLaVA pretrain dataset
17:00 Quality assurance - token repetitions
17:42 Quality assurance - back translate
17:55 Translation verification analysis
18:35 Model training
18:50 Model architecture
20:35 Pre-training LLaVA-1.5 on 8 languages
21:23 Instruction tuning
23:00 Maya demo
24:40 Team Maya
26:00 Q&A with Maya team
35:15 Jiamian starts
35:30 Background - text-video data
36:25 Background - loss function
36:55 Background - evaluation metrics
37:33 Motivation - bridging the modality gap
39:43 Motivation - diffusion model for retrieval task
41:33 Method - DITS (Diffusion-inspired truncated Sampler)
43:10 Experiments - Performance
44:00 Experiments - Model Discussion
47:47 Q&A with Jiamian
Join the Multimodal Minds community to receive an invite for future webinars: discord.gg/CzeUNYr5Bt