DOI : https://doi.org/10.18517/ijaseit.14.4.16976

Evaluation Criteria for Developing a Quality Management System for EdTech Tools

Min Young Kim (1), Ja Mee Kim (2), Dae Hyun Lee (3), Ho Sung Woo (4), Yong Kim (5)
(1) Intube Inc., 330, Neungdong-ro, Gwangjin-gu, Seoul, Republic of Korea
(2) Major of Computer Science Education, Graduate School of Education, Korea University, Seoul, Republic of Korea
(3) Intube Inc., 330, Neungdong-ro, Gwangjin-gu, Seoul, Republic of Korea
(4) Department of Edutech., Graduate School, Korea National Open Univ., 86 Daehak-ro, Jongno-gu, Seoul, Republic of Korea
(5) Department of Edutech., Graduate School, Korea National Open Univ., 86 Daehak-ro, Jongno-gu, Seoul, Republic of Korea
Fulltext View | Download
How to cite (IJASEIT) :
Kim, Min Young, et al. “Evaluation Criteria for Developing a Quality Management System for EdTech Tools ”. International Journal on Advanced Science, Engineering and Information Technology, vol. 14, no. 4, Aug. 2024, pp. 1199-08, doi:10.18517/ijaseit.14.4.16976.
Citation Format :
To support various changes in the education environment and teaching methods, a type of EdTech is emerging which utilizes new information technology and cloud computing-based platforms for education services and ensures more stable operation. As various types of EdTech tools are expected to become more widely adopted in school education in the future, there is a need for accurate information about these tools to be made available, as well as concrete strategies for expanding EdTech-based choice and effective utilization. This study aims to develop optimal evidence-based EdTech evaluation and certification indicators. The process involved five stages: 1) literature review, 2) gathering of expert opinions, 3) deriving improvements, 4) gathering opinions from school environments, and 5) finalizing a framework through analysis of school environment opinions and expert/researcher feedback. The proposed evidence-based EdTech evaluation and certification system includes five evaluation criteria in the technical domain (purpose, convenience, sustainability, integrity/stability, accessibility), with 13 evaluation items, and seven evaluation criteria in the usability domain (purpose, usability, functionality, compatibility, sustainability, integrity/security, ethics), with 20 evaluation items. The results of this study will enhance choice in school environments and contribute to the development of high-quality EdTech tools. The introduction of an evidence-based evaluation and certification system, based on objective data from EdTech stakeholders rather than relying on evaluation by a small group of experts, is expected to enhance trust in EdTech and improve its overall quality.

A. K. Goel, “AI Education for the World,” AI Magazine, vol. 38, no. 2, pp. 3–4, Jun. 2017, doi: 10.1609/aimag.v38i2.2740.

A. Eldow, M. Shakir, M. Y. Abdullah, and S. I. Malik, “Using Information Communication Technology as a Teaching tool in Sudanese Governmental Universities of Khartoum State,” International Journal on Informatics Visualization, vol. 1, no. 4, pp. 150–156, Jan. 1970, doi: 10.30630/joiv.1.4.55.

M. Ibraigheeth and S. A. Fadzli, “Core Factors for Software Projects Success,” International Journal on Informatics Visualization, vol. 3, no. 1, pp. 69–74, Jan. 2019, doi: 10.30630/joiv.3.1.217.

B. S Chai, D. W Min, H. J Lee, "Research to Improve the Quality of Non-Contact Education Systems", The Society of Convergence Knowledge Transactions, vol.11(1), 89-100, 2023.

The Global Goals, “4. Quality Education”, Online, Sep 2023. https://www.globalgoals.org/goals/4-quality-education/

D. Lee and E. Lee, “An Analysis of Educational Needs on Teacher Competencies for Education using AI,” Korean Association for Educational Information and Media, vol. 28, no. 3, pp. 821–842, Sep. 2022, doi: 10.15833/kafeiam.28.3.821.

INNOPOLIS, “Global Market Trends Report: Smart Learning Market”. Korea Innovation Foundation, Daejeon, Republic of Korea, Apr 2021.

H. Y Lee, “Current Status and Implications of the EdTech Market; Korea International Trade Association”, IIT Trade Focus: Seoul, Republic of Korea, May 2020.

KERIS, "2022 White Paper on ICT in Education in Korea", Korea Education and Research Information Service, Daegu, Republic of Korea, Apr 2023.

류진선, Heepil Kim, MoonDaeYoung, and 이경순, “Development of an Examination System for a e-Learning Quality Manager’s Certificate,” Journal of Engineering Education Research, vol. 16, no. 1, pp. 35–44, Jan. 2013, doi: 10.18108/jeer.2013.16.1.35.

KERIS, "2023 Remote Education Training Content Review Operation Manual (ver 5.9)", Korea Education and Research Information Service, Daegu, Republic of Korea, Jan 2023.

M. Cukurova, R. Luckin, and A. Clark‐Wilson, “Creating the golden triangle of evidence‐informed education technology with EDUCATE,” British Journal of Educational Technology, vol. 50, no. 2, pp. 490–504, Dec. 2018, doi: 10.1111/bjet.12727.

T. C. Wei, M. H. Ahmad Hijazi, S. Alias, A. A. A. Ibrahim, and M. F. I. Othman, “Intelligent Course Recommender Chatbot Using Natural Language Processing,” International Journal on Advanced Science, Engineering and Information Technology, vol. 12, no. 5, p. 1915, Sep. 2022, doi: 10.18517/ijaseit.12.5.14798.

EdTech Evidence Exchange, “The EdTech Genome Project Report”, EdTech Evidence Exchange, Virginia, United States of America, July 2021.

Pellini, A., Nicolai, S., Magee, A., Sharp, S., Wilson, S., A political economy analysis framework for EdTech evidence uptake. EdTech Hub. Feb 2021. doi:10.5281/zenodo.4540204.

L. V. O. Hietajärvi and E. Maksniemi, "How to Design Engaging Educational Solutions?", Education Alliance Finland, Helsinki, Finland, Dec 2017.

Education Alliance Finland, “How we evaluate educational products”, Online, Sep 2023. https://educationalliancefinland.com/standard

Department of Education, "Realising the potential of technology in education: A strategy for education providers and the technology industry", Department of Education, DFE-00072-2019, London, The United Kingdom, Apr 2019.

R. Batty, et al. "EdTech Testbeds: Models for improving evidence", NESTA, London, The United Kingdom, May 2019.

KERIS, “What is EduTech Softlab”, Online, Sep 2023. https://www.keris.or.kr/main/cm/cntnts/cntntsViewPop.do?cntntsId=1681.

Kyonggi EdTech Soft Lab, “Process for Demonstrating Edtech Solutions”, Online, Sep 2023. https://www.EdTechlab.kr:45776/?p=14

J. W Jung et al., “Development of EdTech Soft Lab Demonstration Model”, RM 2021-18, Korea Education and Research Information Service, Daegu, Republic of Korea, Dec 2021.

S. W Kim et al., “EdTech Soft Lab Demonstration Guide”, RM 2022-06, Korea Education and Research Information Service, Daegu, Republic of Korea, May 2022.

Daegu EdTech Soft Lab, “Demonstration support”, Online, Sep 2023. https://dgEdTechlab.or.kr/?page_id=66.

Korea Digital Education Frontiers Association, "Announcement of recruiting companies for the 19th EdTech Excellent Enterprise Contest", Online, Sep 2023. https://www.bizinfo.go.kr/web/lay1/bbs/S1T122C128/AS/74/view.do?pblancId=PBLN_000000000088836.

Chungcheongnamdo office of education, “Expert-Teacher Empirical Evaluation Report Based on Clasting AI Insure”, Chungcheongnamdo office of education, Chungcheongnamdo, Republic of Korea, Sep 2022.

S. Yoon and Y. E. Ozturk, “Development and validation of Edtech evaluation tool to promote elementary and secondary teaching-learning utilizing HCI principles,” Korean Association for Educational Information and Media, vol. 28, no. 3, pp. 543–566, Sep. 2022, doi:10.15833/kafeiam.28.3.543.

K. Kim and Y. Joeng, “Effectiveness of Education using Edutech for disadvantaged students,” Korea University Institute of Educational Research, vol. 85, pp. 169–186, Nov. 2022, doi:10.24299/kier.2022.354.169.

A.-R. H. Mastul, C. T. De Vera, and C. B. Jayme, “Understanding the Use of EduTech in Schools in the Philippines: Recommendations for Effectiveness,” Engineering Science Letter, vol. 2, no. 02, pp. 37–40, Apr. 2023, doi: 10.56741/esl.v2i02.307.

Y. J Hur and C. H Ye, “An Improvement Method of Engineering Education Quality using Kano’s Dualistic Quality Model and Timko’s Satisfaction Coefficient,” Journal of Engineering Education Research, vol. 21, no. 3, pp. 31–37, May 2018, doi: 10.18108/jeer.2018.21.3.31.

N. Kano, N. Seraku, F. Takahashi, and S. Tsuji, “Attractive Quality and Must-Be Quality,” Journal of The Japanese Society for Quality Control, vol. 14, no. 2, pp. 0–0, 1984, [Online]. Available: https://sid.ir/paper/600594/en.

S. M Song and S. H Jang, "Quality Improvement Priorities for Cosmetic Store Service Using Kano Model and Potential Customer Satisfaction Improvement Index", International Journal of Contents, vol 20(6), pp 342-353, 2020.

D. Y Kim and H. C Shin, "A Study on Customer Satisfaction for Smart Trunk using the Kano Model", The Korea Academia-Industrial cooperation Society, vol 22(4), pp 115-123, 2021.

D. H Byun and J. H Yang, "An Exploratory study on derivation and Improvement of Kano Quality Attributes in Untact Classes", The Society of service Science, vol. 12(2), pp 65-79, 2022.

Prabowo, Y. Yuniarty, and R. Bramulya Ikhsan, “Student Engagement Mechanism of Online Learning: The Effect of Service Quality on Learning Management System,” International Journal on Informatics Visualization, vol. 6, no. 3, p. 681, Sep. 2022, doi:10.30630/joiv.6.3.1263.

H. H Seo and J. H Park, "A Study on The Perception of Pre-service Teachers on The Use of Edu Tech in School: Centering on Focus Group Interview" The Journal of Learner-Centered Curriculum and Instruction, vol 21(23), pp 253-273, 2021.

Y. H Park and D. W Kim, "EdTech: Topic modeling of newspaper" The Journal of Korea Open Association for Early Childhood Education, vol 28(4), pp 139-158, 2023.

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors who publish with this journal agree to the following terms:

    1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
    2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
    3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).