## Abstract

## Keywords

## 1. Introduction

*j*'th student (${Y}_{j}$) is the result of his/her proficiency (${\theta}_{j}$) with a random measurement error (${\epsilon}_{j}$). The error term (${\epsilon}_{j}$) has the expected value of zero, is assumed normally distributed and unrelated to the proficiency: $E\left({\epsilon}_{j}\right)=\phantom{\rule{0.25em}{0ex}}0$, ${\epsilon}_{j}\sim N(0,{\sigma}_{\epsilon}^{2})$, and ${\rho}_{\epsilon \theta}=0$. Thus, the expected value of ${Y}_{j}$, $E\left({Y}_{j}\right)$, is ${\theta}_{j}$. As a result, the average score is normally distributed around ${\theta}_{j}$ with variance ${\sigma}_{\epsilon}^{2}/n$ with

*n*being the number of observations. Hence, the more observations, the closer the average score is in general to the proficiency.

probability (${\pi}_{ij}$) of the correct response of student

*j*to the item

*i*is described by a logistic function of the difference between the student's proficiency parameter (${\theta}_{j}$) and the item difficulty parameter (${\delta}_{i}$). To fit the Rasch model, a marginal maximum likelihood procedure is often used to estimate the item difficulty parameters, assuming that the students are a random sample from population where the student proficiencies are normally distributed with ${\theta}_{j}\sim N(0,{\sigma}_{\theta}^{2})$, while the items have fixed difficulty. Individual student parameters can be estimated afterwards using empirical Bayes procedures.

## 2. Model

where ${u}_{1j}\sim N(0,{\sigma}_{u1}^{2})$ and ${u}_{2i}\sim N(0,{\sigma}_{u2}^{2})$.

*j*from the overall proficiency. The second residual term shows the deviation of the difficulty of item

*i*from the overall difficulty, in the sense that the larger the residual, the easier the item.

### 2.1 Extension one

${b}_{0}$ equals the estimated logit of probability of the correct response of an average student on an average item in weekly summative assessment; $attemp{t}_{ij}$ is 0, 1, 2, 3 or 4 and means the first, the second, the third, the fourth, or the fifth or higher attempt respectively; ${b}_{10}$ is overall effect of attempt, while ${b}_{1j}\sim N(0,{\sigma}_{b1}^{2})$ is a deviation of the attempt effect for student

*j*from the overall effect; and ${u}_{1j}\sim N(0,{\sigma}_{u1}^{2})$ and ${u}_{2i}\sim N(0,{\sigma}_{u2}^{2})$.

### 2.2 Extension two

where ${b}_{1i}\sim N(0,{\sigma}_{b1}^{2})$ is a deviation of the attempt effect for item

*i*from the overall effect.

### 2.3 Extension three

where $clas{s}_{j}$ is 0, 1, 2, and 3.

*b*

_{3}is negative. We also expect a negative main effect of the class, in the sense that students with a higher maximum number of attempts are less proficient. For instance, it could be students, interested in formal achievements in assessment rather than in mastering the course.

### 2.4 Extension four

where $formative.assessment.performanc{e}_{j}$ is the index that refers to the student's performance in the previous formative assessments of the module; it equals the number of correct attempts, divided by the sum of the number of student's correct and wrong attempts, that may take values from 0 to 1; $lecture.activit{y}_{j}$ is the index that refers to student's activity with video lectures in the module; it takes values from 0, which means that the student did not watch video lectures at all, to 2, which means that the student watched all video lectures in the module. Therefore, ${b}_{4}$ and ${b}_{5}$ are overall effects of performance in formative assessments and activity with video lectures respectively. The coefficients of the formative assessment performance and the lecture activity do not vary from student to student because these variables were measured only once, before the summative assessment starts, and therefore only one value of the variable corresponds to each single student. In this extension we expect these both effects to be positive.

## 3. Methods

Higher School of Economics. (n.d.). Economics for Non-economists. Retrieved from Coursera: https://www.coursera.org/learn/ekonomika-dlya-neekonomistov.

Higher School of Economics. (n.d.). Game Theory. Retrieved from Coursera: https://www.coursera.org/learn/game-theory.

Higher School of Economics. (n.d.). Introduction to Neuroeconomics: How the Brain Makes Decisions. Retrieved from Coursera: https://www.coursera.org/learn/neuroeconomics.

Course 1 | Course 2 | Course 3 | ||
---|---|---|---|---|

Module 1 | Students | 1609 | 3069 | 4806 |

Items | 10 | 10 | 15 | |

Responses | 51550 | 88210 | 141735 | |

Module 2 | Students | 986 | 2050 | 2609 |

Items | 10 | 10 | 15 | |

Responses | 30430 | 55100 | 83940 | |

Module 3 | Students | 700 | 1465 | 1735 |

Items | 10 | 10 | 15 | |

Responses | 13810 | 36940 | 42750 |

*Note:*In the table, Course 1 is “Economics for Non-Economists” (

Higher School of Economics. (n.d.). Economics for Non-economists. Retrieved from Coursera: https://www.coursera.org/learn/ekonomika-dlya-neekonomistov.

Higher School of Economics. (n.d.). Game Theory. Retrieved from Coursera: https://www.coursera.org/learn/game-theory.

Higher School of Economics. (n.d.). Introduction to Neuroeconomics: How the Brain Makes Decisions. Retrieved from Coursera: https://www.coursera.org/learn/neuroeconomics.

where $TP$ is true positives; $TN$ is true negatives; $P$ is all positives; $N$ is all negatives.

## 4. Results

Basic Model | Extension 1 | Extension 2 | Extension 3 | Extension 4 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|

Fixed | Intercept | 0.63 (0.22) ** | 0.27 (0.25) | 0.29 (0.27) | 0.65 (0.27) * | -0.75 (0.30) * | ||||||

Attempt | 0.92 (0.03) *** | 0.90 (0.06) *** | 1.34 (0.07) *** | 1.34 (0.07) *** | ||||||||

Class | -0.48 (0.03) *** | -0.42 (0.03) *** | ||||||||||

Attempt * Class | -0.33 (0.03) *** | -0.33 (0.03) *** | ||||||||||

Lect. Act. | 0.32 (0.07) *** | |||||||||||

Prac. Perf. | 1.16 (0.09) *** | |||||||||||

Var | SD | Var | SD | Var | SD | Var | SD | Var | SD | |||

Random | Intercept | Stud. | 0.61 | 0.78 | 0.84 | 0.92 | 0.92 | 0.96 | 0.74 | 0.86 | 0.59 | 0.77 |

Item | 1.02 | 1.01 | 1.25 | 1.12 | 1.49 | 1.22 | 1.49 | 1.22 | 1.49 | 1.22 | ||

Attempt | Stud. | 0.36 | 0.60 | 0.35 | 0.59 | 0.26 | 0.52 | 0.26 | 0.52 | |||

Item | 0.04 | 0.21 | 0.04 | 0.21 | 0.04 | 0.21 | ||||||

AIC | 58779 | 55401 | 54929 | 54487 | 54260 |

*Note:****:

*p*< .001; **:

*p*< .01; *:

*p*< .05.

Effect of Attempt | Maximum Number of Used Attempts | Watched All Video Lectures and Was Productive with Formative Assessments | Attempt | ||||
---|---|---|---|---|---|---|---|

Student-Specific | Item-Specific | 1 | 2 | 3 | |||

Basic Model | .65 | ||||||

Extension 1 | Average | .57 | .77 | .89 | |||

+1 SD | .57 | .86 | .96 | ||||

−1 SD | .57 | .64 | .71 | ||||

Extension 2 | Average | Average | .57 | .77 | .89 | ||

+1 SD | .57 | .80 | .92 | ||||

−1 SD | .57 | .73 | .84 | ||||

+1 SD | Average | .57 | .86 | .96 | |||

+1SD | .57 | .88 | .98 | ||||

−1 SD | .57 | .83 | .95 | ||||

−1 SD | Average | .57 | .65 | .71 | |||

+1 SD | .57 | .69 | .79 | ||||

−1 SD | .57 | .60 | .62 | ||||

Extension 3 | Average | Average | ≤2 | .66 | .88 | .97 | |

>6 | .31 | .39 | .48 | ||||

Extension 4 | Average | Average | Yes | .74 | .92 | .98 | |

No | .32 | .64 | .87 |

*Note:*In the table, for the basic model and extensions 1 and 2 the student's proficiency is considered as average.

Overall | Course 1 | Course 2 | Course 3 | |||||
---|---|---|---|---|---|---|---|---|

M | SD | M | SD | M | SD | M | SD | |

Rasch (1^{st} att.) | .710 | .07 | .695 | .03 | .638 | .02 | .797 | .02 |

Basic Model | .739 | .06 | .719 | .03 | .688 | .01 | .811 | .02 |

Extension 1 | .768 | .04 | .751 | .02 | .734 | .02 | .819 | .02 |

Extension 2 | .770 | .04 | .753 | .02 | .737 | .02 | .820 | .02 |

Extension 3 | .770 | .04 | .753 | .02 | .737 | .02 | .821 | .02 |

Extension 4 | .771 | .04 | .754 | .02 | .738 | .02 | .821 | .02 |

*Note:*In the table, Course 1 is “Economics for Non-Economists” (

Higher School of Economics. (n.d.). Economics for Non-economists. Retrieved from Coursera: https://www.coursera.org/learn/ekonomika-dlya-neekonomistov.

Higher School of Economics. (n.d.). Game Theory. Retrieved from Coursera: https://www.coursera.org/learn/game-theory.

Higher School of Economics. (n.d.). Introduction to Neuroeconomics: How the Brain Makes Decisions. Retrieved from Coursera: https://www.coursera.org/learn/neuroeconomics.

## 5. Discussion & conclusion

## Declarations

### Author contribution statement

### Funding statement

### Competing interest statement

### Additional information

## References

- A new look at the statistical model identification.
*IEEE Trans. Automat. Contr.*1974; 19: 716-723 - Fitting linear mixed-effects models using lme4.
*J. Stat. Software.*2015; 67: 1-48 - Measuring the Mind: Conceptual Issues in Contemporary Psychometrics.Cambridge University Press, Cambridge2005
- Explanatory Item Response Models: a Generalized Linear and Nonlinear Approach.Springer, New York, NY2004
- The estimation of item response models with the lmer function from the lme4 package in R.
*J. Stat. Software.*2011; 39: 1-28 - T-SKIRT: online estimation of student proficiency in an adaptive learning system.in: 31st International Conference on Machine Learning. Lille. 2015
- Comparison of classical test theory and item response theory and their applications to test development.
*Educ. Meas.*1993; 12: 38-47 - Test length and decision quality in personnel selection: when is short too short?.
*Int. J. Test.*2012; 12: 321-344 - Many-facet Rasch Measurement.MESA Press, Chicago, IL1992
- Statistical Theories of Mental Test Scores.Addison Wesley Publishing Company, Reading, MA1968
- R: a Language and Environment for Statistical Computing.R Foundation for Statistical Computing, 2013 (Retrieved from)
- Probabilistic Models for Some Intelligence and Attainment Tests.Danish Institute for Educational Research, Copenhagen1960
- Coursera's 2017: Year in Review.2017 (Retrieved from Class Central:)
- A Product at Every price: a Review of MOOC Stats and Trends in 2017.2018 (Retrieved from Class Central:)
- Cross-classification multilevel logistic models in psychometrics.
*J. Educ. Behav. Stat.*2003; 28: 369-386

## Article Info

### Publication History

### Identification

### Copyright

### User License

Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) |## Permitted

### For non-commercial purposes:

- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works

## Not Permitted

- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article

Elsevier's open access license policy

### ScienceDirect

Access this article on ScienceDirect## Related Articles

## Comments

#### Cell Press commenting guidelines

To submit a comment for a journal article, please use the space above and note the following:

- We will review submitted comments within 2 business days.
- This forum is intended for constructive dialog. Comments that are commercial or promotional in nature, pertain to specific medical cases, are not relevant to the article for which they have been submitted, or are otherwise inappropriate will not be posted.
- We recommend that commenters identify themselves with full names and affiliations.
- Comments must be in compliance with our Terms & Conditions.
- Comments will not be peer-reviewed.