In the realm of traditional representation learning for Natural Language Processing (NLP) tasks, the prevalent approach involves leveraging large language models like BERT, GPT-2, XLNet, and others, which demonstrate their prowess when trained on extensive datasets. However, the quest for effective representation learning becomes a formidable challenge when data is scarce or costly to acquire, such as in specialised domains and low-resource languages. This challenge stems from the scarcity of suitable algorithms and the inherent difficulties in learning representations from limited data. In this study, we introduce a novel and groundbreaking technique, the Attention-sequence Bidirectional Long Short-Term Memory (ABiLSTM), to tackle this challenge head-on. This method amalgamates the strengths of attention mechanisms and bidirectional Long Short-Term Memory (LSTM) networks, bolstering the model’s ability to capture intricate patterns and dependencies within sequential data, particularly in scenarios involving small or constrained datasets. Through empirical analysis, we delve into the intricate dynamics of neural network architectures, specifically examining the impact of varying numbers of hidden layers and the occurrence of training errors. These aspects are scrutinised in the context of unsupervised pretraining methods, with a focus on their effectiveness in generating robust and informative representations for open relation extraction tasks. Furthermore, we propose the use of knowledge graph techniques to tackle the challenge of imbalanced data during information extraction processes. Our study demonstrates that the use of ABiLSTM yields superior results in open relation extraction tasks, even when dealing with small or limited datasets. This advancement represents a significant contribution to the fields of natural language understanding and information extraction.