DOI : https://doi.org/10.18517/ijaseit.14.5.11677

Deep Learning-based Method in Multimodal Data for Diabetic Retinopathy Detection

Kartina Diah Kesuma Wardhani (1), Shahreen Kasim (2), Aldo Erianda (3), Rohayanti Hassan (4)
(1) Soft Computing and Data Mining Centre (SMC), Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia, Parit Raja, Johor, Malaysia
(2) Soft Computing and Data Mining Centre (SMC), Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia, Parit Raja, Johor, Malaysia
(3) Department of Information Technology, Politeknik Negeri Padang, Padang, Indonesia
(4) Department of Software Engineering, Faculty of Computing, Universiti Teknologi Malaysia, Johor Bahru, Malaysia
Fulltext View | Download
How to cite (IJASEIT) :
Wardhani , Kartina Diah Kesuma, et al. “Deep Learning-Based Method in Multimodal Data for Diabetic Retinopathy Detection”. International Journal on Advanced Science, Engineering and Information Technology, vol. 14, no. 5, Oct. 2024, pp. 1602-8, doi:10.18517/ijaseit.14.5.11677.
Citation Format :
Diabetic retinopathy (DR) is a complex condition, and incorporating information from multiple sources, such as patient history, laboratory results, or genetic data, can enhance understanding. An ophthalmologist or an automated system can identify DR through manual examination. The automatic detection of diabetic retinopathy has become a preferred choice for patients and healthcare providers due to its cost-effectiveness and time efficiency. The novelty of this research lies in developing a model for predicting diabetic retinopathy using multimodal data fusion, incorporating fundus retinal images, optical coherence tomography (OCT), and electronic health records (EHR) through an early fusion technique implemented in a Long Short-Term Memory (LSTM) network. Our model, which utilizes an early fusion of multimodal data with Local Binary Pattern (LBP), has demonstrated the best performance, achieving an AUC value of 0.99. This high accuracy indicates that integrating information from various data sources can significantly improve the capability of the model in detecting both positive and negative cases of diabetic retinopathy, instilling confidence in the reliability of our findings.

N. Marchesi, F. Fahmideh, F. Boschi, A. Pascale, and A. Barbieri, “Ocular Neurodegenerative Diseases: Interconnection between Retina and Cortical Areas,” Cells, vol. 10, no. 9, p. 2394, Sep. 2021, doi:10.3390/cells10092394.

W. L. Alyoubi, W. M. Shalash, and M. F. Abulkhair, “Diabetic retinopathy detection through deep learning techniques: A review,” Informatics in Medicine Unlocked, vol. 20, p. 100377, 2020, doi:10.1016/j.imu.2020.100377.

I. Qureshi, J. Ma, and Q. Abbas, “Diabetic retinopathy detection and stage classification in eye fundus images using active deep learning,” Multimedia Tools and Applications, vol. 80, no. 8, pp. 11691–11721, Jan. 2021, doi: 10.1007/s11042-020-10238-4.

N. Mukherjee and S. Sengupta, “Application of deep learning approaches for classification of diabetic retinopathy stages from fundus retinal images: a survey,” Multimedia Tools and Applications, vol. 83, no. 14, pp. 43115–43175, Oct. 2023, doi: 10.1007/s11042-023-17254-0.

W.-C. Lin, J. S. Chen, M. F. Chiang, and M. R. Hribar, “Applications of Artificial Intelligence to Electronic Health Record Data in Ophthalmology,” Translational Vision Science & Technology, vol. 9, no. 2, p. 13, Feb. 2020, doi: 10.1167/tvst.9.2.13.

D. A. Hasan, S. R. M. Zeebaree, M. A. M. Sadeeq, H. M. Shukur, R. R. Zebari, and A. H. Alkhayyat, “Machine Learning-based Diabetic Retinopathy Early Detection and Classification Systems- A Survey,” 2021 1st Babylon International Conference on Information Technology and Science (BICITS), vol. 4, pp. 16–21, Apr. 2021, doi:10.1109/bicits51482.2021.9509920.

A. Sungheetha and R. Sharma R, “Design an Early Detection and Classification for Diabetic Retinopathy by Deep Feature Extraction based Convolution Neural Network,” Journal of Trends in Computer Science and Smart Technology, vol. 3, no. 2, pp. 81–94, Jul. 2021, doi:10.36548/jtcsst.2021.2.002.

A. Skouta, A. Elmoufidi, S. Jai-Andaloussi, and O. Ouchetto, “Deep learning for diabetic retinopathy assessments: a literature review,” Multimedia Tools and Applications, vol. 82, no. 27, pp. 41701–41766, Apr. 2023, doi: 10.1007/s11042-023-15110-9.

K. Parthiban and M. Kamarasan, “Diabetic retinopathy detection and grading of retinal fundus images using coyote optimization algorithm with deep learning,” Multimedia Tools and Applications, vol. 82, no. 12, pp. 18947–18966, Nov. 2022, doi: 10.1007/s11042-022-14234-8.

S. Kannoju, “Diabetic Retinopathy Detection Using Retinal Images,” Master's thesis, California State University, Sacramento, 2021.

A. M. Dayana and W. R. S. Emmanuel, “A Comprehensive Review of Diabetic Retinopathy Detection and Grading Based on Deep Learning and Metaheuristic Optimization Techniques,” Archives of Computational Methods in Engineering, vol. 30, no. 7, pp. 4565–4599, Jun. 2023, doi: 10.1007/s11831-023-09946-5.

N. K. Waheed et al., “Optical coherence tomography angiography in diabetic retinopathy,” Progress in Retinal and Eye Research, vol. 97, p. 101206, Nov. 2023, doi: 10.1016/j.preteyeres.2023.101206.

M. M. Khansari et al., “Automated Deformation-Based Analysis of 3D Optical Coherence Tomography in Diabetic Retinopathy,” IEEE Transactions on Medical Imaging, vol. 39, no. 1, pp. 236–245, Jan. 2020, doi: 10.1109/tmi.2019.2924452.

D. Gildea, “The diagnostic value of optical coherence tomography angiography in diabetic retinopathy: a systematic review,” International Ophthalmology, vol. 39, no. 10, pp. 2413–2433, Oct. 2018, doi: 10.1007/s10792-018-1034-8.

J. Chua et al., “Optical Coherence Tomography Angiography in Diabetes and Diabetic Retinopathy,” Journal of Clinical Medicine, vol. 9, no. 6, p. 1723, Jun. 2020, doi: 10.3390/jcm9061723.

A. M. R. Godla, D. Kavitha, K. Ashok, M. S. A. Ansari, N. Satheesh, and R. V. K. Reddy, “An Ensemble Learning Approach for Multi-Modal Medical Image Fusion using Deep Convolutional Neural Networks,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 8, 2023, doi: 10.14569/ijacsa.2023.0140884.

P. Thanapol, K. Lavangnananda, P. Bouvry, F. Pinel, and F. Leprevost, “Reducing Overfitting and Improving Generalization in Training Convolutional Neural Network (CNN) under Limited Sample Sizes in Image Recognition,” 2020 - 5th International Conference on Information Technology (InCIT), Oct. 2020, doi:10.1109/incit50588.2020.9310787.

V. Vaidyaa and L.K. Vishwamitra, "Diabetes Detection using Convolutional Neural Network through Feature Sequencing", Turkish Journal of Computer and Mathematics Education, vol. 12, no. 10, pp. 2783-2789, 2021.

H. Zhang, H. Huang, and H. Han, “Attention-Based Convolution Skip Bidirectional Long Short-Term Memory Network for Speech Emotion Recognition,” IEEE Access, vol. 9, pp. 5332–5342, 2021, doi:10.1109/access.2020.3047395.

G. Van Houdt, C. Mosquera, and G. Nápoles, “A review on the long short-term memory model,” Artificial Intelligence Review, vol. 53, no. 8, pp. 5929–5955, May 2020, doi: 10.1007/s10462-020-09838-1.

C. Schröder and A. Niekler, "A survey of active learning for text classification using deep neural networks", arXiv:2008.07267, 2020.

G. Mena, K. Coussement, K. W. De Bock, A. De Caigny, and S. Lessmann, “Exploiting time-varying RFM measures for customer churn prediction with deep neural networks,” Annals of Operations Research, vol. 339, no. 1–2, pp. 765–787, Mar. 2023, doi:10.1007/s10479-023-05259-9.

P. Gholami, P. Roy, and V. Lakshminarayanan, ‘Diabetic Retinopathy Retinal OCT Images’. Borealis, 2018.

P. Darabi, ‘Diagnosis of Diabetic Retinopathy’. 15-Jul-2024, doi: 10.13140/RG.2.2.13037.19688.

X. Zhuang et al., “Data from: Association of diabetic retinopathy and diabetic macular oedema with renal function in southern Chinese patients with type 2 diabetes mellitus: a single-centre observational study,” [Dataset]. Dryad, Sep. 2019, doi:10.5061/dryad.6kg1sd7.

M. Monemian, M. Irajpour, and H. Rabbani, “A review on texture-based methods for anomaly detection in retinal optical coherence tomography images,” Optik, vol. 288, p. 171165, Oct. 2023, doi:10.1016/j.ijleo.2023.171165.

K. Zhou et al., “Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Images,” Computer Vision – ECCV 2020, pp. 360–377, 2020, doi: 10.1007/978-3-030-58565-5_22.

M. Garg and G. Dhiman, “A novel content-based image retrieval approach for classification using GLCM features and texture fused LBP variants,” Neural Computing and Applications, vol. 33, no. 4, pp. 1311–1328, Jun. 2020, doi: 10.1007/s00521-020-05017-z.

K. Kaplan, Y. Kaya, M. Kuncan, M. R. Mi̇naz, and H. M. Ertunç, “An improved feature extraction method using texture analysis with LBP for bearing fault diagnosis,” Applied Soft Computing, vol. 87, p. 106019, Feb. 2020, doi: 10.1016/j.asoc.2019.106019.

S. Hamad Khaleefah, S. A. Mostafa, A. Mustapha, and M. Faidzul Nasrudin, “Review of local binary pattern operators in image feature extraction,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 19, no. 1, p. 23, Jul. 2020, doi:10.11591/ijeecs.v19.i1.pp23-31.

N. Kaur, N. Nazir, and Manik, “A Review of Local Binary Pattern Based texture feature extraction,” 2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), vol. 475, pp. 1–4, Sep. 2021, doi:10.1109/icrito51393.2021.9596485.

Y. Luo, J. Sa, Y. Song, H. Jiang, C. Zhang, and Z. Zhang, “Texture classification combining improved local binary pattern and threshold segmentation,” Multimedia Tools and Applications, vol. 82, no. 17, pp. 25899–25916, Mar. 2023, doi: 10.1007/s11042-023-14749-8.

M. H. Shakoor, R. Boostani, M. Sabeti, and M. Mohammadi, “Feature selection and mapping of local binary pattern for texture classification,” Multimedia Tools and Applications, vol. 82, no. 5, pp. 7639–7676, Aug. 2022, doi: 10.1007/s11042-022-13470-2.

R. Tekin, Ö. F. Ertuğrul, and Y. Kaya, “New local binary pattern approaches based on color channels in texture classification,” Multimedia Tools and Applications, vol. 79, no. 43–44, pp. 32541–32561, Aug. 2020, doi: 10.1007/s11042-020-09698-5.

A. El Idrissi, Y. El merabet, and Y. Ruichek, “Palmprint recognition using state‐of‐the‐art local texture descriptors: a comparative study,”IET Biometrics, vol. 9, no. 4, pp. 143–153, May 2020, doi:10.1049/iet-bmt.2019.0103.

T. Chen et al., “A novel face recognition method based on fusion of LBP and HOG,” IET Image Processing, vol. 15, no. 14, pp. 3559–3572, Apr. 2021, doi: 10.1049/ipr2.12192.

R. Balestriero and L. Bottou, “The Effects of Regularization and Data Augmentation are Class Dependent,” Neural Information Processing Systems (NeurIPS), 2022.

E.H. Hssayni, M. Ettaouil, “Generalization Ability Augmentation and Regularization of Deep Convolutional Neural Networks Using L1/2 Pooling”, International Journal on Technical and Physical Problems of Engineering (IJTPE), Issue 48, Vol. 13, No. 3, pp. 1-6, September 2021.

C. Khosla and B. S. Saini, “Enhancing Performance of Deep Learning Models with different Data Augmentation Techniques: A Survey,” 2020 International Conference on Intelligent Engineering and Management (ICIEM), Jun. 2020, doi:10.1109/iciem48762.2020.9160048.

H. Yao et al., ‘Improving Generalization in Meta-learning via Task Augmentation’, in Proceedings of the 38th International Conference on Machine Learning, 18--24 Jul 2021, vol. 139, pp. 11887–11897.

P. Zhang, C. Li, C. Peng, and J. Tian, “Ultra-Short-Term Prediction of Wind Power Based on Error Following Forget Gate-Based Long Short-Term Memory,” Energies, vol. 13, no. 20, p. 5400, Oct. 2020, doi: 10.3390/en13205400.

J. Oruh, S. Viriri, and A. Adegun, “Long Short-Term Memory Recurrent Neural Network for Automatic Speech Recognition,” IEEE Access, vol. 10, pp. 30069–30079, 2022, doi: 10.1109/access.2022.3159339.

M. Heydarian, T. E. Doyle, and R. Samavi, “MLCM: Multi-Label Confusion Matrix,” IEEE Access, vol. 10, pp. 19083–19095, 2022, doi:10.1109/access.2022.3151048.

I. M. De Diego, A. R. Redondo, R. R. Fernández, J. Navarro, and J. M. Moguerza, “General Performance Score for classification problems,” Applied Intelligence, vol. 52, no. 10, pp. 12049–12063, Jan. 2022, doi:10.1007/s10489-021-03041-7.

Y. Li et al., “Hybrid Fusion of High-Resolution and Ultra-Widefield OCTA Acquisitions for the Automatic Diagnosis of Diabetic Retinopathy,” Diagnostics, vol. 13, no. 17, p. 2770, Aug. 2023, doi:10.3390/diagnostics13172770.

M.-Y. Hsu et al., “Deep Learning for Automated Diabetic Retinopathy Screening Fused With Heterogeneous Data From EHRs Can Lead to Earlier Referral Decisions,” Translational Vision Science & Technology, vol. 10, no. 9, p. 18, Aug. 2021, doi: 10.1167/tvst.10.9.18.

M. El Habib Daho et al., “Improved Automatic Diabetic Retinopathy Severity Classification Using Deep Multimodal Fusion of UWF-CFP and OCTA Images,” Ophthalmic Medical Image Analysis, pp. 11–20, 2023, doi: 10.1007/978-3-031-44013-7_2.

V. S. Tseng et al., “Leveraging Multimodal Deep Learning Architecture with Retina Lesion Information to Detect Diabetic Retinopathy,” Translational Vision Science & Technology, vol. 9, no. 2, p. 41, Jul. 2020, doi: 10.1167/tvst.9.2.41.

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors who publish with this journal agree to the following terms:

    1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
    2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
    3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).