See "through the front line": reliability of questionnaires in wartime
Abstract
Diagnosis and screening of individuals, both civilian and military, is typically done through a clinical interview and\or self-report questionnaires. This information is used to adjust treatment strategies, psychological and psychotherapeutic interventions, and to predict the patient's condition. In modern science, methods that have been shown to be effective and have satisfactory test validity and reliability are considered the gold standard for testing. Adhering to this standard ensures that the tool being used is performing its intended functions to a certain degree of quality providing decent results. Many Ukrainian studies are based on methods that were adapted or created during the era of the Soviet Union and have not been reviewed since. On the other hand, we have methods that were translated from another languages without going through the proper adaptation process. These questionnaires are used in a clinical practice too, by psychologists, psychiatrists and psychotherapists. Moreover, there is no known or open data about the performance of these tests amid the war, both in original and in translated versions.
The aim of this article was to explore the reliability of the most frequently used screening tools. The study took place in one of the rehabilitation center in Kyiv, Ukraine. Overall, 28 participants joined, all of whom were soldiers who are currently enrolled in the Armed forces of Ukraine. Measurements were taken 3 times with a one week gaps. GAD-7 and STAI (modified by Hanin) were used to screen anxiety, PCL-5 for PTSD screening and ISI helped us to determine sleep problems. The GAD-7, ISI, and PCL-5 questionnaires, in the versions we tested, demonstrated more than satisfactory levels of reliability and are recommended by the authors for use in clinical conditions and for screening soldiers in the armed forces. The STAI, on the other hand, showed unsatisfactory indicators and is not recommended for use in screening procedures or for research purposes. In conclusion, it is recommended that Ukrainian researchers adopt international methods properly, conducting validation studies adhering to so-called “golden standards”.
References
Bobyreva, L., & Shpetny, O. (2020). Otsinka rivnia tryvozhnosti u khvorykh na tsukrovyi diabet v umovakh pandemii COVID 19. [Assessment of anxiety levels in diabetic patients in the context of the COVID 19 pandemic.] http://repository.pdmu.edu.ua/bitstream/123456789/14478/1/kyivmedcenter.pdf[in Ukrainian]
Press service of the regional state administration. (2022, september 14). V Ukraini prezentuvaly rezultaty pershoho z pochatku povnomasshtabnoi viiny doslidzhennia psykholohichnoho stanu naselennia—Cherkaska oblasna derzhavna administratsiia. [The results of the first study of the psychological state of the population since the beginning of a full-scale war were presented in Ukraine—Cherkasy regional state administration].. https://ck-oda.gov.ua/novyny-cherkaskoyi-oblasti/v-ukrayini-prezentuvali-rezultati-pershogo-z-pochatku-povnomasshtabnoyi-vijni-doslidzhennya-psixologichnogo-stanu-naselennya/[in Ukrainian]
Shkala tryvohy Spilberhera [Spielberg's anxiety scale] (STAI). (2022). https://mozok.ua/depressiya/testy/item/2703-shkala-trivogi-splbergera-STAI [in Ukrainian]
Anima. (2022). Neuroscientific Anxiety Test. https://anima.help/
Blevins, C. A., Weathers, F. W., Davis, M. T., Witte, T. K., & Domino, J. L. (2015). The Posttraumatic Stress Disorder Checklist for DSM-5 (PCL-5): Development and Initial Psychometric Evaluation: Posttraumatic Stress Disorder Checklist for DSM-5. Journal of Traumatic Stress, 28(6), 489–498. https://doi.org/10.1002/jts.22059
Brantly, A. (2022). Narrative Battles: The Impact Open-Source Intelligence on the Framing of Russia’s War on Ukraine. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4256772
Cai, W., Dong, W., Pan, Y., Wei, C., Zhang, S., Tian, B., Yan, J., & Deng, G. (2018). Reliability, validation and norms of the Chinese version of Anxiety Sensitivity Index 3 in a sample of military personnel. PLOS ONE, 13(8), e0201778. https://doi.org/10.1371/journal.pone.0201778
Cardoso, J. R., Pereira, L. M., Iversen, M. D., & Ramos, A. L. (2014). What is gold standard and what is ground truth? Dental Press Journal of Orthodontics, 19(5), 27–30. https://doi.org/10.1590/2176-9451.19.5.027-030.ebo
Fernandez-Mendoza, J., Rodriguez-Muñoz, A., Vela-Bueno, A., Olavarrieta-Bernardino, S., Calhoun, S. L., Bixler, E. O., & Vgontzas, A. N. (2012). The Spanish version of the Insomnia Severity Index: A confirmatory factor analysis. Sleep Medicine, 13(2), 207–210. https://doi.org/10.1016/j.sleep.2011.06.019
Gold standard (test). (2022). Psychology Wiki. https://psychology.fandom.com/wiki/Gold_standard_(test)
Hancock, G. R., Stapleton, L. M., & Mueller, R. O. (Ред.). (2019). The reviewer’s guide to quantitative methods in the social sciences (Second Edition). Routledge, Taylor & Francis Group.
Kalcza-Janosi, K., Kotta, I., Marschalko, E. E., & Szabó, K. (2022). The Fear of War Scale (FOWARS): Development and initial validation [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/sfz2v
Morin, C. M., Belleville, G., Bélanger, L., & Ivers, H. (2011). The Insomnia Severity Index: Psychometric Indicators to Detect Insomnia Cases and Evaluate Treatment Response. Sleep, 34(5), 601–608. https://doi.org/10.1093/sleep/34.5.601
Muñiz, J., Elosua, P., Hambleton, R. K., & International Test Commission. (2013). [International Test Commission Guidelines for test translation and adaptation: Second edition]. Psicothema, 25(2), 151–157. https://doi.org/10.7334/psicothema2013.24
Murphy, B. A., & Lilienfeld, S. O. (2019). Are self-report cognitive empathy ratings valid proxies for cognitive empathy ability? Negligible meta-analytic relations with behavioral task performance. Psychological Assessment, 31(8), 1062–1072. https://doi.org/10.1037/pas0000732
Netflix Ukraine (Режисер). (2022, Грудень 26). НаступнийгістьДевідаЛеттермана: ВолодимирЗеленський | Повнийепізод | Netflix. https://www.youtube.com/watch?v=96Op72mMSJw
Osokina, O., Silwal, S., Bohdanova, T., Hodes, M., Sourander, A., & Skokauskas, N. (2022). Impact of the Russian Invasion on Mental Health of Adolescents in Ukraine. Journal of the American Academy of Child & Adolescent Psychiatry, S0890856722018913. https://doi.org/10.1016/j.jaac.2022.07.845
Otte, J. L., Bakoyannis, G., Rand, K. L., Ensrud, K. E., Guthrie, K. A., Joffe, H., McCurry, S. M., Newton, K. M., & Carpenter, J. S. (2019). Confirmatory factor analysis of the Insomnia Severity Index (ISI) and invariance across race: A pooled analysis of MsFLASH data. Menopause, 26(8), 850–855. https://doi.org/10.1097/GME.0000000000001343
R Core Team. (2022). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. https://www.R-project.org/
Revelle, W. (2022). psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University. https://CRAN.R-project.org/package=psych
Selker, R., Love, J., Dropmann, D., & Moreno, V. (2022). jmv: The «jamovi» Analyses. https://CRAN.R-project.org/package=jmv
Sheather, J. (2022). As Russian troops cross into Ukraine, we need to remind ourselves of the impact of war on health. BMJ, o499. https://doi.org/10.1136/bmj.o499
Shevlin, M., Hyland, P., Karatzias, T., Makhashvili, N., Javakhishvili, J., & Roberts, B. (2022). The Ukraine crisis: Mental health resources for clinicians and researchers. Journal of Traumatic Stress, 35(3), 775–777. https://doi.org/10.1002/jts.22837
Spielberger, C. D. (2012). State-Trait Anxiety Inventory for Adults [Data set]. American Psychological Association. https://doi.org/10.1037/t06496-000
Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006). A Brief Measure for Assessing Generalized Anxiety Disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092. https://doi.org/10.1001/archinte.166.10.1092
Taber, K. S. (2018). The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
Thombs, B. D., Kwakkenbos, L., Levis, A. W., & Benedetti, A. (2018). Addressing overestimation of the prevalence of depression based on self-report screening questionnaires. Canadian Medical Association Journal, 190(2), E44–E49. https://doi.org/10.1503/cmaj.170691
Van Doorn, E. A., Heerdink, M. W., & Van Kleef, G. A. (2012). Emotion and the construal of social situations: Inferences of cooperation versus competition from expressions of anger, happiness, and disappointment. Cognition & Emotion, 26(3), 442–461. https://doi.org/10.1080/02699931.2011.648174
Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. https://ggplot2.tidyverse.org
Wickham, H., François, R., Henry, L., & Müller, K. (2022). dplyr: A Grammar of Data Manipulation. https://CRAN.R-project.org/package=dplyr
Wouters, B., Groen, J., Janssen Steenberg, J., & Kling, S. (2020). Online Influence: Boost your results with proven behavioral science. Verlag nicht ermittelbar.
Wubben, M. J. J., Cremer, D. D., & Dijk, E. van. (2009). How emotion communication guides reciprocity: Establishing cooperation through disappointment and anger. Journal of Experimental Social Psychology, 45(4), 987–990. https://doi.org/10.1016/j.jesp.2009.04.010
Yuryeva, L., Vyshnichenko, S., & Shornikov, A. (2022). Analysis of anxiety and depression phenomena in the first weeks of the war: Gender and age aspects. Psychosomatic Medicine and General Practice, 7(1), Art. 1. https://doi.org/10.26766/pmgp.v7i1.351
Zijlmans, E. A. O., Tijmstra, J., van der Ark, L. A., & Sijtsma, K. (2018). Item-Score Reliability in Empirical-Data Sets and Its Relationship With Other Item Indices. Educational and Psychological Measurement, 78(6), 998–1020. https://doi.org/10.1177/0013164417728358
This work is licensed under a Creative Commons Attribution 4.0 International License.
