Keywords
content validation
delphi technique
mobile application
Abstract
Background: Anemia remains a critical public health issue among adolescents, particularly in developing countries such as Indonesia. Poor nutritional knowledge and limited awareness of anemia-related symptoms, etiology, and prevention exacerbate this condition. Mobile health (mHealth) technologies have the potential to address these gaps through accessible, engaging, and scalable education tools. This study aimed to validate the content of AneMia_Prev®, a mobile application designed as an educational tool to enhance adolescent knowledge on the prevention of anemia.
Methods: A Delphi technique was employed involving two rounds of expert panel review. Twelve experts with clinical and academic backgrounds in nursing and public health evaluated the content of AneMia_Prev® based on relevance, clarity, layout, illustrations, language, and motivational features. The Content Validity Index (CVI) and modified kappa statistics were used to assess inter-rater agreement and content adequacy. Data were collected through an online survey using a 17-item validated questionnaire.
Results: In the first round, all 17 items achieved excellent content validity with I-CVI values ranging from 0.87 to 1.00 and kappa values above 0.87. Following minor expert recommendations, a revised version of the application was re-evaluated, resulting in unanimous ratings of excellence (I-CVI = 1.00; kappa > 0.92 for all items). Experts emphasized the application’s innovation, relevance, and potential to promote anemia awareness among adolescents.
Conclusion: AneMia_Prev® demonstrated excellent content validity and is considered suitable for educational interventions targeting anemia prevention among adolescents. Future research is recommended to assess semantic validation, cognitive impact, and learning outcomes among adolescent users to further refine the tool and evaluate its effectiveness in real-world settings.
References
Alaunyte, I., Stojceska, V., & Plunkett, A. (2015). Iron and the female athlete: a review of dietary treatment methods for improving iron status and exercise performance. Journal of the International Society of Sports Nutrition, 12(1), 38. https://doi.org/10.1186/s12970-015-0099-2
Ali, E. E., Chan, S. S. L., Leow, J. L., Chew, L., & Yap, K. Y.-L. (2019). User acceptance of an app-based adherence intervention: perspectives from patients taking oral anticancer medications. Journal of Oncology Pharmacy Practice, 25(2), 390–397. https://doi.org/10.1177/1078155218778106
Amor-García, M. Á., Collado-Borrell, R., Escudero-Vilaplana, V., Melgarejo-Ortuño, A., Herranz-Alonso, A., Arija, J. Á. A., & Sanjurjo-Sáez, M. (2020). Assessing apps for patients with genitourinary tumors using the mobile application rating scale (MARS): systematic search in app stores and content analysis. JMIR MHealth and UHealth, 8(7), e17609. https://doi.org/10.2196/17609
Angadi, N., & Ranjitha, A. (2016). Knowledge, attitude, and practice about anemia among adolescent girls in urban slums of Davangere City, Karnataka. International Journal of Medical Science and Public Health, 5(3), 416–419.
Asch, S. M. (1998). Developing a clinical performance measure. American Journal of Preventive Medicine, 14(3), 14–21. https://doi.org/10.1016/S0749-3797(97)00032-9
Berglund, M., Andersson, S., & Kjellsdotter, A. (2024). Reflection tools–support for patient learning in group education. Reflective Practice, 25(5), 634–649. https://doi.org/10.1080/14623943.2024.2384119
Collado-Borrell, R., Escudero-Vilaplana, V., Villanueva-Bueno, C., Herranz-Alonso, A., & Sanjurjo-Saez, M. (2020). Features and functionalities of smartphone apps related to COVID-19: systematic search in app stores and content analysis. Journal of Medical Internet Research, 22(8), e20334. https://doi.org/10.2196/20334
Dantas, L. O., Carvalho, C., Prando, B. C., McAlindon, T. E., & da Silva Serrao, P. R. M. (2021). Mobile health technologies for the management of rheumatic diseases: a systematic review of online stores in Brazil. Clinical Rheumatology, 40(7), 2601–2609. https://doi.org/10.1007/s10067-020-05561-y
de Souza Menezes, F., Leite, H. P., & Nogueira, P. C. K. (2012). Malnutrition as an independent predictor of clinical outcome in critically ill children. Nutrition, 28(3), 267–270. https://doi.org/10.1016/j.nut.2011.05.015
Dodt, R. C. M., Ximenes, L. B., Almeida, P. C., Oriá, M. O. B., & Dennis, C. L. (2012). Psychometric and maternal sociodemographic assessment of the breastfeeding self-efficacy scale-short form in a brazilian sample. J Nurs Educ Pract, 2(3), 66–73. http://dx.doi.org/10.5430/jnep.v2n3p66
Fogg, B. J. (2002). Persuasive technology: using computers to change what we think and do. Ubiquity, 2002(December), 2. https://doi.org/10.1145/764008.763957
Fogg, B. J. (2007). Mobile persuasion: 20 perspectives on the future of behavior change. Mobile Persuasion.
Fogg, B. J., & Eckles, D. (2007). Mobile persuasion: 20 perspectives on the future of behavior change. (No Title).
Ghosal, S., Jana, S., Mani, A., Mitra, S., & Roy, S. (2020). Sex workers, Stigma and Self-Image: Evidence from Kolkata Brothels (CSAE Working Paper Series, Issues 2020–03). Centre for the Study of African Economies, University of Oxford. https://doi.org/10.1162/rest_a_01013
Johnson, C. O., Nguyen, M., Roth, G. A., Nichols, E., Alam, T., Abate, D., Abd-Allah, F., Abdelalim, A., Abraha, H. N., Abu-Rmeileh, N. M. E., & others. (2019). Global, regional, and national burden of stroke, 1990--2016: a systematic analysis for the Global Burden of Disease Study 2016. The Lancet Neurology, 18(5), 439–458. https://doi.org/10.1016/S1474-4422(19)30034-1
Jongerius, C., Russo, S., Mazzocco, K., & Pravettoni, G. (2019). tested mobile apps for breast cancer care: systematic review. JMIR MHealth and UHealth, 7(2), e10930. https://doi.org/10.2196/10930
Kalhori, S. R. N., Hemmat, M., Noori, T., Heydarian, S., & Katigari, M. R. (2021). Quality evaluation of English mobile applications for gestational diabetes: app review using Mobile Application Rating Scale (MARS). Current Diabetes Reviews, 17(2), 161–168. https://doi.org/10.2174/1573399816666200703181438
Kebede, M. M., & Pischke, C. R. (2019). Popular diabetes apps and the impact of diabetes app use on self-care behaviour: a survey among the digital community of persons with diabetes on social media. Frontiers in Endocrinology, 10, 135. https://doi.org/10.3389/fendo.2019.00135
Keeney, S., McKenna, H. A., & Hasson, F. (2011). The Delphi technique in nursing and health research. John Wiley & Sons.
Kwan, Y. H., Ong, W. J., Xiong, M., Leung, Y. Y., Phang, J. K., Wang, C. T. M., & Fong, W. (2019). Evaluation of mobile apps targeted at patients with spondyloarthritis for disease monitoring: systematic app search. JMIR MHealth and UHealth, 7(10), e14753. https://doi.org/10.2196/14753
Mulianingsih, M., Suriah, S., Hidayanty, H., Amiruddin, R., Hadju, V., Salmah, A. U., & Yusron, M. A. (2024). Nutritional Deficiency Anemia Status among Adolescent Girls in North Lombok District, West Nusa Tenggara, Indonesia. The Open Public Health Journal, 17(1). http://dx.doi.org/10.2174/0118749445353728241130164045
Narrillos-Moraza, Á., Gómez-Martínez-Sagrera, P., Amor-García, M. Á., Escudero-Vilaplana, V., Collado-Borrell, R., Villanueva-Bueno, C., Gómez-Centurión, I., Herranz-Alonso, A., & Sanjurjo-Sáez, M. (2022). Mobile Apps for Hematological Conditions: Review and Content Analysis Using the Mobile App Rating Scale. JMIR MHealth and UHealth, 10(2), e32826. https://doi.org/10.2196/32826
Oliveira, M. S. de, Fernandes, A. F. C., & Sawada, N. O. (2008). Educational handbook for self care in women with mastectomies: a validation study. Texto & Contexto-Enfermagem, 17, 115–123. https://doi.org/10.1590/S0104-07072008000100013
Polit, D. F., & Beck, C. T. (2010). Essentials of nursing research: Appraising evidence for nursing practice. Lippincott Williams & Wilkins.
Polit, D. F., Beck, T., & Owen, S. V. (2007). Focus on research methods is the CVI an acceptable indicator of content validity. Res Nurs Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199
Preti, O. (2010). Produção de material didático impresso: orientações técnicas e pedagógicas. Cuiabá: UAB/UFMT, 175.
Reberte, L. M., Hoga, L. A. K., & Gomes, A. L. Z. (2012). Process of construction of an educational booklet for health promotion of pregnant women. Revista Latino-Americana de Enfermagem, 20, 101–108. https://doi.org/10.1590/S0104-11692012000100014
Rubio, D. M., Berg-Weger, M., Tebb, S. S., Lee, E. S., & Rauch, S. (2003). Objectifying content validity: Conducting a content validity study in social work research. Social Work Research, 27(2), 94–104. https://doi.org/10.1093/swr/27.2.94
Salehinejad, S., Niakan Kalhori, S. R., Hajesmaeel Gohari, S., Bahaadinbeigy, K., & Fatehi, F. (2021). A review and content analysis of national apps for COVID-19 management using Mobile Application Rating Scale (MARS). Informatics for Health and Social Care, 46(1), 42–55. https://doi.org/10.1080/17538157.2020.1837838
Saraswati, P. W. (2021). How National Public Policies in Indonesia Address Maternal Mortality from The Perspective of Health Equity and Intersectionality. https://doi.org/10.26656/fr.2017.8(4).339
Singh, J. L. (n.d.). H., Couch, D., & Yap, K.(2020). Mobile health apps that help with COVID-19 management: Scoping review. JMIR Nursing, 3(1), e20596. https://doi.org/10.2196/20596
Sousa, C. S., & Turrini, R. N. T. (2012). Construct validation of educational technology for patients through the application of the Delphi technique. Acta Paulista de Enfermagem, 25, 990–996. https://doi.org/10.1590/S0103-21002012000600026
Statistik, B. P. (2020). Info geografi dan kependudukan.
Teixeira, E., & Mota, V. (2011). Tecnologias educacionais em foco. São Paulo: Difusão Editora, 72. https://doi.org/10.1590/S0034-71672011000200024
Terhorst, Y., Rathner, E. M., Baumeister, H., & Sander, L. (2018). Help from the App Store?’: a systematic review of depression apps in German app stores. Verhaltenstherapie, 28(2), 101–112.
The Ministry of Health of Republic of Indonesia. (2018). The Report of National Health Research 2018.
Vosnacos, E., & Pinchon, D. J. (2015). Survey of women’s perceptions of information provided in the prevention or treatment of iron deficiency anaemia in an Australian tertiary obstetric hospital. Women and Birth, 28(2), 166–172. https://doi.org/10.1016/j.wombi.2014.11.004
Walker, H. C. (2013). Establishing content validity of an evaluation rubric for mobile technology applications utilizing the Delphi method. Johns Hopkins University. https://doi.org/10.5205/reuol.4761-42160-1-ED.0712201311
World Health Organization (WHO). (2023). Adolecents.
Wynd, C. A., Schmidt, B., & Schaefer, M. A. (2003). Two quantitative approaches for estimating content validity. Western Journal of Nursing Research, 25(5), 508–518. https://doi.org/10.1177/0193945903252998
Xelegati, R., & Évora, Y. D. M. (2011). Development of a virtual learning environment addressing adverse events in nursing. Revista Latino-Americana de Enfermagem, 19, 1181–1187. https://doi.org/10.1590/S0104-11692011000500016
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Copyright (c) 2025 Sri Rahayu, Mohamed Saifulaman Mohamed Said, Tukimin Bin Sansuwito, Sigit Mulyono