loading page

Unsupervised Pretraining Approach for Open Relation Extraction
  • Israel Fianyi,
  • James Montgomery,
  • Soonja Yeom
Israel Fianyi
University of Tasmania - Launceston Campus

Corresponding Author:[email protected]

Author Profile
James Montgomery
University of Tasmania - Hobart Campus
Author Profile
Soonja Yeom
University of Tasmania - Hobart Campus
Author Profile

Abstract

In the realm of traditional representation learning for Natural Language Processing (NLP) tasks, the prevalent approach involves leveraging large language models like BERT, GPT-2, XLNet, and others, which demonstrate their prowess when trained on extensive datasets. However, the quest for effective representation learning becomes a formidable challenge when data is scarce or costly to acquire, such as in specialised domains and low-resource languages. This challenge stems from the scarcity of suitable algorithms and the inherent difficulties in learning representations from limited data. In this study, we introduce a novel and groundbreaking technique, the Attention-sequence Bidirectional Long Short-Term Memory (ABiLSTM), to tackle this challenge head-on. This method amalgamates the strengths of attention mechanisms and bidirectional Long Short-Term Memory (LSTM) networks, bolstering the model’s ability to capture intricate patterns and dependencies within sequential data, particularly in scenarios involving small or constrained datasets. Through empirical analysis, we delve into the intricate dynamics of neural network architectures, specifically examining the impact of varying numbers of hidden layers and the occurrence of training errors. These aspects are scrutinised in the context of unsupervised pretraining methods, with a focus on their effectiveness in generating robust and informative representations for open relation extraction tasks. Furthermore, we propose the use of knowledge graph techniques to tackle the challenge of imbalanced data during information extraction processes. Our study demonstrates that the use of ABiLSTM yields superior results in open relation extraction tasks, even when dealing with small or limited datasets. This advancement represents a significant contribution to the fields of natural language understanding and information extraction.
07 May 2024Submitted to Expert Systems
08 May 2024Submission Checks Completed
08 May 2024Assigned to Editor
15 May 2024Reviewer(s) Assigned
05 Jun 2024Review(s) Completed, Editorial Evaluation Pending
02 Jul 20241st Revision Received
04 Jul 2024Submission Checks Completed
04 Jul 2024Assigned to Editor
04 Jul 2024Reviewer(s) Assigned
15 Sep 2024Review(s) Completed, Editorial Evaluation Pending
20 Sep 2024Editorial Decision: Revise Major
17 Oct 20242nd Revision Received