Multimodal Models

Source papers 31

The Arxiv papers behind this sub-topic. Links may take you off-site to arxiv.org.

• Aether Weaver: Multimodal Affective Narrative Co-Generation with Dynamic Scene Graphs
  We introduce Aether Weaver, a novel, integrated framework for multimodal narrative cogeneration that overcomes limitations of sequential text-to-visual pipelines. Our system concurrently synthesizes t…
• Beyond Language Modeling: An Exploration of Multimodal Pretraining
  The foundation model era has been defined largely by the success of language pretraining. By scaling autoregressive models on trillions of text tokens, we have created systems with remarkable reasonin…
• Continual Instruction Tuning for Large Multimodal Models
  Instruction tuning is now a widely adopted approach to aligning large multimodal models (LMMs) to follow human intent. It unifies the data format of vision-language tasks, enabling multi-task joint tr…
• Deep Interest Network for Click-Through Rate Prediction
  “Click-through rate prediction is an essential task in industrial applications, such as online advertising. Recently deep learning based models have been proposed, which follow a similar Embedding& ML…
• Do Language Models Understand Time?
  Large language models (LLMs) have revolutionized video-based computer vision applications, including action recognition, anomaly detection, and video summarization. Videos inherently pose unique chall…
• DocLLM: A layout-aware generative language model for multimodal document understanding
  Enterprise documents such as forms, invoices, receipts, reports, contracts, and other similar records, often carry rich semantics at the intersection of textual and spatial modalities. The visual cues…
• Emerging Properties in Unified Multimodal Pretraining
  Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. In this work, we introduce BAGEL, an open-source foundational model that nativel…
• Explainable Multimodal Emotion Reasoning
  Multimodal emotion recognition is an active research topic in the field of artificial intelligence. It aims to integrate multimodal clues (including acoustic, visual, and lexical clues) and recognize …
• Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
  We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space. We leverage the multimodal capabi…
• How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  Multimodal Large Language Models (MLLMs) have demonstrated strong performance across a wide range of vision-language tasks, yet their internal processing dynamics remain underexplored. In this work, w…
• LLaMA-Omni: Seamless Speech Interaction with Large Language Models
  Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there…
• Large Multimodal Agents: A Survey
  Large language models (LLMs) have achieved superior performance in powering text-based AI agents, endowing them with decision-making and reasoning abilities akin to humans. Concurrently, there is an e…
• Lumiere: A Space-Time Diffusion Model for Video Generation
  We introduce Lumiere – a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion – a pivotal challenge in video synthesis. To this end, we in…
• MAPS: A Multi-Agent Framework Based on Big Seven Personality and Socratic Guidance for Multimodal Scientific Problem Solving
  Multimodal scientific problems (MSPs) involve complex issues that require the integration of multiple modalities, such as texts and diagrams, presenting a significant challenge in artificial intellige…
• MIO: A Foundation Model on Multimodal Tokens
  In this paper, we introduce MIO, a novel foundation model built on multimodal tokens, capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. …
• MLLM-CBench: A Comprehensive Benchmark for Continual Instruction Tuning of Multimodal LLMs with Chain-of-Thought Reasoning Analysis
  Multimodal Large Language Models (MLLMs) rely on continual instruction tuning to adapt to the evolving demands of real-world applications. However, progress in this area is hindered by the lack of rig…
• MM-LLMs: Recent Advances in MultiModal Large Language Models
  In the past year, MultiModal Large Language Models (MM-LLMs) have undergone substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or outputs via cost-effective training strategi…
• Mindstorms in Natural Language-Based Societies of Mind
  Both Minsky’s “society of mind” and Schmidhuber’s “learning to think” inspire diverse societies of large multimodal neural networks (NNs) that solve problems by interviewing each other in a “mindstorm…
• Navigating the State of Cognitive Flow: Context-Aware AI Interventions for Effective Reasoning Support
  Flow Theory describes an optimal cognitive state where individuals experience deep focus and intrinsic motivation when a task’s difficulty aligns with their skill level. In AI-augmented reasoning, int…
• Nested Attention: Semantic-aware Attention Values for Concept Personalization
  Personalizing text-to-image models to generate images of specific subjects across diverse scenes and styles is a rapidly advancing field. Current approaches often face challenges in maintaining a bala…
• No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
  Web-crawled pretraining datasets underlie the impressive “zero-shot” evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation. H…
• Pixel-Level Reasoning Segmentation via Multi-turn Conversations
  Existing visual perception systems focus on region-level segmentation in single-turn dialogues, relying on complex and explicit query instructions. Such systems cannot reason at the pixel level and co…
• Pixels, Patterns, but No Poetry: To See The World like Humans
  Achieving human-like perception and reasoning in Multimodal Large Language Models (MLLMs) remains a central challenge in artificial intelligence. While recent research has primarily focused on enhanci…
• ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answering
  In this paper, we propose a new dataset, ReasonVQA, for the Visual Question Answering (VQA) task. Our dataset is automatically integrated with structured encyclopedic knowledge and constructed using a…
• Self-Rewarding Vision-Language Model via Reasoning Decomposition
  Vision-Language Models (VLMs) often suffer from visual hallucinations – saying things that aren’t actually in the image – and language shortcuts, where they skip the visual part and just rely on text …
• Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models
  Note: This is not an official technical report from OpenAI. Sora is a text-to-video generative AI model, released by OpenAI in February 2024. The model is trained to generate videos of realistic or im…
• Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs
  Humans spontaneously use increasingly efficient language as interactions progress, by adapting and forming ad-hoc conventions. This phenomenon has been studied extensively using reference games, showi…
• The Curse Of Recursion: Training On Generated Data Makes Models Forget
  Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT introduced such langua…
• The Demon is in Ambiguity: Revisiting Situation Recognition with Single Positive Multi-Label Learning
  Abstract—Context recognition (SR) is a fundamental task in computer vision that aims to extract structured semantic summaries from images by identifying key events and their associated entities. Speci…
• The Evolution of Multimodal Model Architectures
  This work uniquely identifies and characterizes four prevalent multimodal model architectural patterns in the contemporary multimodal landscape. Systematically categorizing models by architecture type…
• VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
  Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction. However, …