I am a PhD student in Computer Science at University of California, Berkeley, advised by Prof. Alvin Cheung. I was also advised by Prof. Dawn Song at UC Berkeley. Before that, I received my Bachelor's degree in Computer Science from Peking University, China, advised by Prof. Liwei Wang and Prof. Di He.
I specialize in Artificial Intelligence (AI) with a focus on Large Language Models (LLMs). My research includes pretraining, prompting, and evaluation methodologies for a variety of language models, including BERT, T5, and GPT-like LLMs. My recent research delves into leveraging LLMs for code generation, infilling, transpilation, and understanding, pushing the boundaries of how AI interacts with programming languages.
We introduce Syntax-Aware Fill-In-the-Middle (SAFIM), a new benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task.
We introduce AST-T5, a novel pretraining paradigm that leverages the Abstract Syntax Tree (AST) for enhanced code generation, transpilation, and understanding.
This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5.
📄Linyuan Gong, Jiayi Wang, Alvin Cheung (2023). ADELT: Transpilation Between Deep Learning Frameworks. In IJCAI 2024.
📄Jianhao Shen, Chenguang Wang, Linyuan Gong, Dawn Song (2022). Joint Language Semantic and Structure Embedding for Knowledge Graph Completion. In COLING 2022.
📄Xinyun Chen, Linyuan Gong, Alvin Cheung, Dawn Song (2021). PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context. In ACL 2021.
📄Yilun Xu, Yang Song, Sahaj Garg, Linyuan Gong, Rui Shu, Aditya Grover, Stefano Ermon (2021). Anytime Sampling for Autoregressive Models via Ordered Autoencoding. In ICLR 2021.
📄Juyong Kim, Linyuan Gong, Justin Khim, Jeremy C. Weiss, Pradeep Ravikumar (2020). Improved Clinical Abbreviation Expansion via Non-Sense-Based Approaches. In ML4H (NeurIPS Workshop) 2020.
📄Zhenhui Xu, Linyuan Gong, Guolin Ke, Di He, Shuxin Zheng, Liwei Wang, Jiang Bian, Tie-Yan Liu (2020). MC-BERT: Efficient Language Pre-Training via a Meta Controller.
📄Yingce Xia, Xu Tan, Fei Tian, Fei Gao, Weicheng Chen, Yang Fan, Linyuan Gong, Yichong Leng, Renqian Luo, Yiren Wang, Lijun Wu, Jinhua Zhu, Tao Qin, Tie-Yan Liu (2019). Microsoft Research Asia's Systems for WMT19. In WMT19 (ACL 2019 Workshop).
📄Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, Tieyan Liu (2019). Efficient training of BERT by progressively stacking. In ICML 2019.