Page 52 - The-5th-MCAIT2021-eProceeding

An Adjusted BERT Architecture for The Automatic Essay
Scoring Task

b
a
c
Ridha Hussein Chassab , Lailatul Qadri Zakaria *, Sabrina Tiun
a,b,c The Asean Natural Language Processing (ASLAN), Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia
43600 Bangi, Selangor Darul Ehsan, Malaysia.
* Email: lailatul.qadri@ukm.edu.my

Abstract

Automatic Essay Scoring (AES) is the process of identifying an automatic score for an essay answer. The state-of-the-art in
AES task relies on word embedding techniques. One of the advanced embedding architecture that seems promising is the
Bidirectional Encoder Representations from Transformers (BERT). Yet, such an architecture suffers from ‘catastrophic
forgetting’ problem. This problem occurs because the gradients of fine-tuning BERT quickly forget significant information.
In order to overcome such a limitation and to adapt the BERT architecture to be fit for the AES task, it is imperative to
address an adequate adjustment on the learning rate. Therefore, this paper aims at proposing an adjusted BERT architecture
based on unfreezing fine-tune mechanism in which the BERT architecture can adequately adopted for the AES task.

Keywords: Automatic Essay Scoring; Automatic Essay Grading; Bidirectional Encoder Representations from Transformers.

1. Introduction

Assessment is considered as the main component in the educational domain where student’s abilities are
being evaluated (Valenti, Neri, & Cucchiarelli, 2003). The last two decades showed a great investment in E-
learning by utilizing the latest technologies. The dawn of 21 century has witnessed a great progress in terms of
computerized assessment systems (Valenti et al. 2000). The majority of such systems were concentrating on
multiple choice questions where the assessment would take the form of storing correct answers in a relational
database and accommodating a simple matching mechanism to identify whether the student’s answer is correct
or not. Since the multiple-choice questions have a factual and exact answers, it was to implement an automatic
assessment system to evaluate answers. Yet, another challenge has been arisen toward the computerized exams,
such challenge is represented by the assessment of questions that require subjective answers. Automatic Essay
Scoring (AES) is the process of identifying an automatic score for an essay answer. Subjective or essay answers
require much more effort in order to articulate an automatic assessment for it.
The state-of-the-art in AES task relies Word2Vec and pretrained Glove architectures (Chen & Zhou, 2019;
Hendre, Mukherjee, Preet, & Godse, 2020; Li, Chen, & Nie, 2020; Li et al., 2018; Liu et al., 2019; Wang, Liu,
& Dong, 2018; Zhang & Litman, 2019). The main limitation behind such architectures lies in its inability to
handle sentence-level embedding and they suffer from ‘out-of-vocabulary’ problem. This problem occurs when
a term would have no embedding vector within the training (i.e., unseen). A solution for the aforementioned
problems depicted by the Bidirectional Encoder Representations from Transformers (BERT) architecture where
it has a fixed vocabulary size and a mechanism of rooting the terms. Although BERT showed remarkable
performance for tasks like question-answering yet, it showed incompetency when tested for the AES task
(Mayfield & Black, 2020; Rodriguez, Jafari, & Ormerod, 2019). The reason behind such miscarriage is due to
a well-known limitation behind the BERT architecture which is ‘catastrophic forgetting’ (Rodriguez et al.,
2019). This problem occurs because the gradients of fine-tuning BERT quickly forget significant information.

E- Proceedings of The 5th International Multi-Conference on Artificial Intelligence Technology (MCAIT 2021) [40]
Artificial Intelligence in the 4th Industrial Revolution