Page 56 - Special Topic Session (STS) - Volume 4

STS560 James Houran et al.
the bottom, “hard” items occur near the top), together with the scale score
(inside circle). The green area indicates the areas that are already mastered by
this respondent, the red area marks the topics yet to be mastered (none here),
while the yellow area marks the primary training goals. Most importantly, the
right-hand side of Figure 3 shows the model deviations of this respondent’s
test answers in terms of their zij (see, Eq. 6 above). Note that the answers to
the easiest questions (bottom part of figure) do not reach any of the vertical
dotted lines, where the first line away from 0 (solid black line) represents zij =
1.65 and the second zij = 2.0. Thus, while the answer residuals are not exactly
0, these deviations can be seen as random. However, this is not the case for
“Tracks and documents all mistakes” which is highly negative (z < 2). In other
words, while most of the responses of this test taker are only to be expected
given his/her Tj, this item stands out significantly (p < .05). From a practical
perspective this means that this issue may need further attention. The
identification of items with particular low or high zij is exploited to formulate
or guide the most pertinent and useful questions to pose the test-taker (i.e.,
job candidate or incumbent) during subsequent job-interviews. For instance,
in the situation depicted in Figure 3 it is suggested that the interviewer might
ask use the follow-up question “Do you think it is important that complete
records should be kept of employees’ mistakes?” Note that for each question
this approach presumes the availability of a library of follow-up questions
corresponding to low vs. high residuals.

8. Discussion
The 20|20Skills™ system outlined above has been in commercial use for
over 15 years. Over 100,000 applicants or incumbents in the service-hospitality
industry have been tested globally across a wide range of roles (consumer
non-facing and facing) up-and-down the organizational chart – and it has
been translated into several languages other than English. Conceptually, this
paper demonstrated a specific and proven method of combining (i) tailor-
made individual feedback obtained via psychometrics with (ii) structured (or
situational)-based interviews by human interviewers to create an integrated
process of pre-employment screening and selection. The 20|20Skills™ served
as proof-of-concept for the development of a customized “line-level”
assessment system that combines artificial intelligence and machine-learning
with the response residual approach. The new system is optimized for a
particular company culture to quickly and accurately pinpoint test-takers with
attitudes and behaviours characteristic of markedly-high levels of service
orientation and contextual performance (Houran, Tracey, & Lange, 2017). In
combination with the use of “big-data” in HR (daily attendance information,
service latencies, customer evaluations, … etc.), we anticipate that residuals
also can play an important role in performance evaluations. In addition, we
expect that response residuals proves useful equally in engagement surveys,

45 | I S I W S C 2 0 1 9