A Specialized UI Multimodal Model

MLLM by Motiff leverages a classic mixture-of-experts approach, linking pre-trained vision encoders with a large language model (LLM) through connectors. The workflow is as follows:

- Visual Processing: Images are processed by a vision encoder and transformed into visual tokens by the vision-language connector. - Text Generation: The visual tokens are combined with text tokens, allowing the LLM to generate comprehensive text responses, enhancing UI design interaction.

Due to the scarcity of high-quality UI domain data, we employed the following methods for data collection:

- UI Screenshot Descriptions: Detailed modular descriptions of UI screenshots, covering layouts, components, and functionalities. - Structured UI Descriptions: Focus on high-quality, knowledge-dense data, precisely identifying and describing UI components. - UI Task Tuning Data: Constructed a comprehensive set of UI-related tasks, including descriptions, Q&A, pixel-level positioning, and interaction guides.