Page 362 - Invited Paper Session (IPS) - Volume 2

IPS279 Rense Lange

Quality control in large-scale onine knowledge
integration using OBJECTIVE
Rense Lange
1,2
1 Global Psytech, Selangor, Malaysia (rense.lange@globalpsytech.com)
2 Laboratory for Statistics and Computation, ISLA, Vila Nova de Gaia, Portugal

Abstract
As is useful in essay-grading, product-evaluation, and some crowd-sourcing
tasks, Multi-Faceted Rasch Scaling (MFRS) models provide simultaneous
diagnostic information concerning products’ (essays) perceived quality,
buyers’ (test-takers) overall preferences or trait-levels, raters’ leniency /
severity, as well as rating scale usage. Unfortunately, the traditional
approaches to fitting MFRS models are suitable for batch processing only, thus
greatly limiting their real-time (online) potential usage. However, it proved
possible to structure the PAIRS parameter estimation method such that most
of the estimation computations are already performed during data entry.
Simulations indicate that the resulting system is highly efficient and for
samples with up to simulated 2,000,000 cases in an offline context, parameter
updates took less than 0.001 seconds. The findings were sufficiently
encouraging to form the basis for the OBJECTIVE system that can update
model parameters nearly instantaneously, thereby creating actionable quality-
control information for use in actual practice.

Keywords
OBJECTIVE; Multi-Faceted Rasch Scaling; Grading; Rater severity; Rater bias

1. Introduction
Modern psychometrics provides powerful practical tools for test
construction, quality control – including identifying and removing item and
test biases [see, e.g., 1] – and test equating (i.e., deriving equivalent scores
using different items). Psychometrics aims to assess respondents’ (test-takers)
traits or abilities based on their answers to a series of questions (items). Its
methods are widely applied in education and psychology and usage is
increasing in other online areas like product evaluation [2].
Here I focus on Multi-Faceted Rasch Scaling (MFRS), as was pioneered by
Linacre [3]. MFRS can include factors beyond test-takers and items – the
classical application being essay grading where MFRS is used to quantify the
role played by the essay graders. In addition to providing parameter estimates,
it also serves a quality control function by providing extensive model fit
information. The central feature of essay grading (i.e., evaluation by proxy) are
shared by applications such as crowd-sourcing, product rating, and online
349 | I S I W S C 2 0 1 9