Skip to content
Jan Netík

Research

Psychometrics in education and psychology: what makes test and questionnaire items behave as they do, the models and open-source tools that make sense of them, and the evaluation of programs that rely on them.

Item difficulty modeling with transformers

Current focus

Modeling what makes a test item hard from the words it is made of, before anyone has answered it.

Calibrating test items traditionally requires administering them to hundreds of respondents. Response-free item difficulty modeling asks instead how much of an item's difficulty can be predicted from its wording alone. I fine-tune transformer encoders end-to-end on item text, including the passage, question, and answer options of reading comprehension multiple-choice items, and study extensions that inject psychometric structure into the model: component-wise encoding of item parts, and multi-task learning with an auxiliary question-answering objective. My current work goes a step further and models the simulated response behavior of an "item reader" model, deriving difficulty from predicted response distributions rather than regressing on it directly.

How item difficulty is modeled from item wordingThe item's wording is split into tokens, each token becomes a vector of numbers, a transformer encoder mixes information between tokens with attention and transforms each token with feed-forward layers, the result is pooled into one vector, and a regression head maps it to a single difficulty number. During fine-tuning the gap between the predicted and the known difficulty is traced back through every layer and nudges the weights until the predictions match.item wordingpassage · question · optionstokensThephotosynthesisof…embeddingsnumbers per tokentransformer encoderattentionfeed-forward× 12 layerspooledregression headdifficulty−0.40.30.6known: 0.6fine-tuningthe gap to the known value flows back and nudges the weights of every layerHow item difficulty is modeled from item wordingThe item's wording is split into tokens, each token becomes a vector of numbers, a transformer encoder mixes information between tokens with attention and transforms each token with feed-forward layers, the result is pooled into one vector, and a regression head maps it to a single difficulty number. During fine-tuning the gap between the predicted and the known difficulty is traced back through every layer and nudges the weights until the predictions match.item wordingpassage · question · optionstokensThephotosynthesisof…embeddings · numbers per tokentransformer encoderattentionfeed-forward× 12 layerspooledregression headdifficulty−0.40.30.6known: 0.6fine-tuningthe gap to the known value flows backand nudges the weights of every layer
The wording is split into tokens, each token becomes a vector of numbers, and the encoder's layers repeatedly mix information between tokens (attention) and transform each token (feed-forward) until the whole item is summarized in one vector, which a small regression head maps to a single difficulty number. Fine-tuning is what makes this work: for thousands of items with known difficulty, the gap between the predicted and the known value flows back through every layer and nudges the model's weights, so the encoder learns which features of wording matter for difficulty.
Too technical? The same idea in plain words.Seven short steps with small animations, no formulas.

How it works

The same idea in plain words, one step at a time.

This line of work is funded by my Charles University Grant Agency project Wording-Based Difficulty Prediction of Educational Test Items with Large Language Models (GA UK 332525, 2025–2027), where I am the principal investigator, and by the EduCoDe project (GAČR 25-16951S).

Interactive psychometrics: ShinyItemAnalysis and its modules

Open-source item analysis, and an architecture others can build on.

Together with Patrícia Martinková and Adéla Hladká I co-develop {ShinyItemAnalysis}, an open-source application for test and item analysis, and its modular architecture, which I designed. The {SIAtools} developer toolkit lets anyone create, test, and distribute their own modules as R packages. The framework and sample modules are described in our 2026 Psychometrika paper. In the EduTest project I also built standalone apps that let schools analyze their own Maturita and entrance-exam data from CZVV.

Item response theory methods

Parametrizations of the nominal response model, built for interpretability.

Bock's nominal response model gives every response category its own slope and intercept, which suits multiple-choice items but leaves the model hard to identify, slow to converge, and awkward to interpret. With Patrícia Martinková I proposed a baseline-category logit parametrization whose constraints are anchored on the correct response category, in an intercept–slope form (BLIS) and in the traditional IRT form (BLIRT). Anchoring on the correct answer keeps its traceline monotonically increasing, and in the BLIRT form each distractor's threshold reads directly as the ability needed to move from that distractor to the correct answer. Against the parametrization of Thissen, Cai, and Bock it recovered parameters with much lower error, avoided the sign-flipped slopes that can leave estimated and true abilities negatively correlated, and converged in fewer EM iterations. Both forms are available in {ShinyItemAnalysis}.

Applied research and program evaluation

Methodology, measurement, and implementation fidelity, in education and the social sector.

At Schola Empirica I design and analyze program evaluations. A good part of that is psychometric, adapting instruments into Czech, checking that the translated versions hold up, and establishing norms. With Adam Klocek and Lenka Kollerová I took part in validating the Florence Bullying-Victimization Scales, evaluating the KiVa anti-bullying program in a cluster randomized trial, and modeling risk and protective factors as a longitudinal network.

Alongside that come commissioned evaluations for other organizations across education and the social sector: quasi-experimental studies with comparison schools and longitudinal cohorts, secondary analysis of existing screening and testing data, and survey work backed by interviews and focus groups. The largest ran for five years and analyzed close to 50,000 responses from nine sources and four groups of actors in education. I am also a member of the evaluation unit for Skoala, Nadace České spořitelny's financial-literacy platform. The analyses run on reproducible R workflows built around {reschola}, to which I added the LimeSurvey API client.

Projects and grants