Yameng Li
2024
SciMind: A Multimodal Mixture-of-Experts Model for Advancing Pharmaceutical Sciences
Zhaoping Xiong
|
Xintao Fang
|
Haotian Chu
|
Xiaozhe Wan
|
Liwei Liu
|
Yameng Li
|
Wenkai Xiang
|
Mingyue Zheng
Proceedings of the 1st Workshop on Language + Molecules (L+M 2024)
Large language models (LLMs) have made substantial strides, but their use in reliably tackling issues within specialized domains, particularly in interdisciplinary areas like pharmaceutical sciences, is hindered by data heterogeneity, knowledge complexity, unique objectives, and a spectrum of constraint conditions. In this area, diverse modalities such as nucleic acids, proteins, molecular structures, and natural language are often involved. We designed a specialized token set and introduced a new Mixture-of-Experts (MoEs) pre-training and fine-tuning strategy to unify these modalities in one model. With this strategy, we’ve created a multi-modal mixture-of-experts foundational model for pharmaceutical sciences, named SciMind. This model has undergone extensive pre-training on publicly accessible datasets including nucleic acid sequences, protein sequences, molecular structure strings, and biomedical texts, and delivers good performance on biomedical text comprehension, promoter prediction, protein function prediction, molecular description, and molecular generation.
2023
Word-level Prefix/Suffix Sense Detection: A Case Study on Negation Sense with Few-shot Learning
Yameng Li
|
Zicheng Li
|
Ying Chen
|
Shoushan Li
Findings of the Association for Computational Linguistics: ACL 2023
Morphological analysis is an important research issue in the field of natural language processing. In this study, we propose a context-free morphological analysis task, namely word-level prefix/suffix sense detection, which deals with the ambiguity of sense expressed by prefix/suffix. To research this novel task, we first annotate a corpus with prefixes/suffixes expressing negation (e.g., il-, un-, -less) and then propose a novel few-shot learning approach that applies an input-augmentation prompt to a token-replaced detection pre-training model. Empirical studies demonstrate the effectiveness of the proposed approach to word-level prefix/suffix negation sense detection.
Search
Co-authors
- Zicheng Li 1
- Ying Chen 1
- Shoushan Li 1
- Zhaoping Xiong 1
- Xintao Fang 1
- show all...