Page 318 - Contributed Paper Session (CPS) - Volume 2

CPS1873 Ferenc M. et al.
questions, random choice from response categories, substantial subjective
reduction of scale granularity was particularly common among these
interviewees.
The other important result on which our presentation focuses addresses
the question of usability of the presented interpretive cognitive questionnaire
testing methodology when the goal is the regular testing of complex and
extensive survey questionnaires. Besides all the advantages of the method, it
is highly time and resource consuming. The entire duration of the full project
was a whole year, of which the cognitive testing itself took six months’ work
of three researchers from the design phase till the completion of the Final
Report. Apart from the demanding methodology, it was also due to the
characteristics of the tested questionnaire. Even a simple field interview with
the HBLS questionnaire imposes huge burden on the interviewer as well as the
respondent owing to its extreme length and the sensitive nature of the
majority of the questions. In contrast, the time to be spent on a single question
in a CI is multiple times more than simply asking it, and this additional burden,
mutatis mutandis, demands the same scale of additional work in the analysis,
too. It follows from the experiences of this pilot project that implementing the
interpretive cognitive testing methodology in the Office requires further
consideration.

4. Discussion and Conclusion
Although the aim of the presentation is to show the experiences of a particular
questionnaire testing project, the results are relevant for any statistical office
or NSI with respect to methodological protocols, design, and organisation
practices of cognitive test projects on extensive and internationally
harmonised questionnaires. Based on our experiences with the pilot project
on the HBLS questionnaire, the following suggestions and possible solutions
can be outlined with respect to the issues of cognitive testing of extensive and
complex questionnaires at statistical offices/NSIs:
1. establishing a permanent, dedicated researcher team of
approximately 10-15 researchers (‘test laboratory’) capable of
successful execution of simultaneous cognitive questionnaire tests;
2. outsourcing specific tasks that do not require expert knowledge (e.g.
typing the interview transcriptions);
3. less interviews per a testing programme (not recommended, only if
nothing else can be done);
4. abandoning the practice of the complete testing of such extensive
questionnaires, and designing individual test projects focusing on
particular question blocks or selected (key) questions;
5. after devising a simpler testing protocol yielding valid and controlled
results, elaborating a ‘mixed’ model: performing the majority of the

307 | I S I W S C 2 0 1 9