Adapting Transformers

to Wording-Based
Item Difficulty Prediction

Jan Netík

Czech Academy of Sciences & Charles University

Introduction

• Knowing the difficulty of items is vital for test construction
  (minimize measurement error at targeted ability levels)

• Pretesting is oftentimes not feasible at all
  (target population not available, item leakage, cost etc.)

• Expert judgment is inconsistent (AlKhuzaey et al., 2023)

What other options do we have?

The approaches

1. Hand-crafted features → separate difficulty modeling (Freedle & Kostin, 1993)
2. Hand-crafted features + word embeddings → separate modeling (Štěpánek et al., 2023)
3. Hand-crafted features + last hidden states of pre-trained transformers → separate modeling (Sharpnack et al., 2024; Yancey et al., 2024)
4. raw item wording → transformer fine-tuning (Netík et al., 2024)

Transformers 101

• Originally illustrated on machine translation (Vaswani et al., 2017) – encoder and decoder parts with cross attention in between
• “Attention is all you need” (Vaswani et al., 2017)
• State-of-the-art Large Language Models (LLMs) use only the decoder part to “autoregressively” generate next token (during pre-training, the model cannot attend future tokens)
• Encoder-only models are used to represent the input sequence (bi-directional self-attention during training)

Figure 1: The architecture of a transformer (Vaswani et al., 2017)

Figure 2: Encoder part of a transformer (Vaswani et al., 2017)

Figure 3: Multi-head self-attention (Vaswani et al., 2017)

Figure 4: Scaled dot-product attention (Vaswani et al., 2017)

Previous effort (Netík et al., 2024)

• Fine-tuned DistillBERT (Sanh et al., 2020) predicting item difficulty from the wording alone, zero preprocessing

  • Classification head placed on top of the last encoder layer (the last hidden state for the [CLS] token):
    \hat{\mathbf{y}} = \mathrm{softmax}(\mathbf{W} \mathbf{h}_{\text{[CLS]}} + \mathbf{b}),\quad \mathbf{W} \in \mathbb{R}^{5 \times 768},\ \mathbf{b} \in \mathbb{R}^5,\ \mathbf{h}_{\text{[CLS]}} \in \mathbb{R}^{768}

  • Trained on 720 multiple-choice RACE¹ items labeled with Rasch difficulty, after MLM for domain adaptation on full RACE

  • Results only on par with the traditional approaches with hand-crafted features

1. Large-scale ReAding Comprehension dataset from Examinations (RACE, Lai et al., 2017).

Sequence classification with an encoder-only transformer (from Tunstall et al., 2022)

Figure 5: A sample item from the RACE dataset (Lai et al., 2017)

Takeaways from post-hoc evaluation

• Local optimum achieved?
• The model “focuses” primarily on the article¹, failed to learn item structure and relationships between the parts
• Limited diversity of the training set – labelled subset only has a single question and option set per stem
• Unreliable Rasch difficulty estimates – large standard errors
• Small dataset

1. The first and longest part of the wording, usually.

Current directions

• Overcome the diversity/sample size issues by model and/or training procedure modification
  1. Do not use [CLS]’s hidden state, but pool over the other tokens in the last layer (and/or deeper layers)
  2. Encode each wording part separately and concatenate
  3. Multi-task learning (Caruana, 1997) – add item wording part tagging alongside the difficulty prediction
  4. Use examples labelled by SOTA LLMs (using few-shot learning) for intermediate training

Preliminary experimentation results

• RACE dataset with middle and high school items + additional college-level items – school type as a difficulty proxy (NB: slightly different task)
• Balanced sample of approx. 3K items (70% train, 15% validation, 15% test)
• Predictive accuracy improved from 0.69 (previous [CLS]-based approach on the same dataset) to 0.86 (mean-pooled last hidden states)

Thank you!

Jan Netík
netik@cs.cas.cz

Institute of Computer Science, The Czech Academy of Sciences
Faculty of Education, Charles University

References

AlKhuzaey, S., Grasso, F., Payne, T. R., & Tamma, V. (2023). Text-based question difficulty prediction: A systematic review of automatic approaches. International Journal of Artificial Intelligence in Education, 34, 862–914. https://doi.org/10.1007/s40593-023-00362-1

Caruana, R. (1997). Multitask Learning. Machine Learning, 28(1), 41–75. https://doi.org/10.1023/A:1007379606734

Freedle, R., & Kostin, I. (1993). The prediction of TOEFL reading item difficulty: Implications for construct validity. Language Testing, 10(2), 133–170. https://doi.org/10.1177/026553229301000203

Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017). RACE: Large-scale ReAding Comprehension Dataset From Examinations. https://doi.org/10.48550/arXiv.1704.04683

Netík, J., Martinek, F., & Martinková, P. (2024, July 16–19). Fine-tuning language models to predict item difficulty from wording. International Meeting of Psychometric Society 2024, International Meeting of Psychometric Society 2024, Prague, Czech Republic.

Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2020). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. https://doi.org/10.48550/arXiv.1910.01108

Sharpnack, J., Hao, K., Mulcaire, P., Bicknell, K., LaFlair, G., Yancey, K., & von Davier, A. A. (2024). BanditCAT and AutoIRT: Machine Learning Approaches to Computerized Adaptive Testing and Item Calibration. Proceedings of Machine Learning Research, 264, 121–135. https://proceedings.mlr.press/v264/sharpnack25a.html

Štěpánek, L., Dlouhá, J., & Martinková, P. (2023). Item Difficulty Prediction Using Item Text Features: Comparison of Predictive Performance across Machine-Learning Algorithms. Mathematics, 11(19, 19), 4104. https://doi.org/10.3390/math11194104

Tunstall, L., Werra, L. von, & Wolf, T. (2022). Natural language processing with transformers: Building language applications with Hugging Face (First edition). O’Reilly Media.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. https://doi.org/10.48550/arXiv.1706.03762

Yancey, K. P., Runge, A., LaFlair, G., & Mulcaire, P. (2024). BERT-IRT: Accelerating Item Piloting with BERT Embeddings and Explainable IRT Models (E. Kochmar, M. Bexte, J. Burstein, A. Horbach, R. Laarmann-Quante, A. Tack, V. Yaneva, & Z. Yuan, Eds.; pp. 428–438). Association for Computational Linguistics. https://aclanthology.org/2024.bea-1.35/