OMRON SINIC X to Present Latest Research Findings at NAACL 2025, Top-tier Conference in the Field of Natural Language Processing
- April 16, 2025
OMRON SINIC X Corporation (HQ: Bunkyo-ku, Tokyo; President and CEO: Masaki Suwa, hereinafter "OSX") will present the latest research findings at 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (hereinafter " NAACL 2025").
NAACL 2025 is one of the top-tier international conferences in the field of natural language processing. The conference will be held from April 29 to May 4, 2025, in New Mexico, United States (local time).
The research papers to be presented by OSX are as follows:
笆Where is the answer? An empirical study of positional bias for parametric knowledge extraction in language model
| Authors | Kuniaki Saito (OSX), Chen-yu Lee (Google Cloud AI), Kihyuk Sohn (Google Research), Yoshitaka Ushiku (OSX) |
| Outlines | This paper investigates how language models memorize information and extract it in the form of questions. The language model performs self-supervised learning using unlabeled text data, during which it learns knowledge about various facts. This knowledge is required to be "extracted" from the model's parameters to answer specific questions. However, traditional methods face the issue known as the "perplexity curse1)." This refers to the phenomenon where, even if the loss is minimized for the training data documents and the model memorizes them accurately, it is difficult to extract correct answers to actual questions. This study investigates this problem and demonstrates that a "positional bias" in the training data is one of the causes of this issue. Specifically, it was found that the model struggles to answer information presented in the middle or end of sentences in the training data. The study shows that this problem is caused by autoregressive learning (the method where the next token is predicted based on previous tokens). These findings are considered to be key to effectively extracting knowledge from the language model's parameters. 1) Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig,Xi Victoria Lin, Wen-tau Yih, and Srinivasan Iyer. Instruction-tuned language models are better knowledge learners. arXiv preprint arXiv:2402.12847, 2024. |
| Related Page | https://arxiv.org/abs/2402.12170 |
*Author information is current as of the date of writing or submission. Please be advised that the information may become outdated after that point.
縲
About OMRON SINIC X Corporation
OMRON SINIC X Corporation is a strategic subsidiary seeking to realize the "near-future design" that OMRON forecasts. It is comprised of researchers with cutting-edge knowledge and experience across a wide range of technological domains, including AI, Robotics, IoT, and Sensing. With the aim of addressing social issues, OSX integrates innovative technologies with business models and strategies in technology and IP to create near-future design. Additionally, the company accelerates the creation of these designs through collaborative research with universities and external research institutions.
笳Website: https://www.omron.com/sinicx/en/
笳Activities: https://www.omron.com/sinicx/en/activity/
For any inquiries about OSX, please contact us here.
Adobe Acrobat Reader is free software that lets you view and print Adobe Portable Document Format (PDF) files.