Research on Eye Pupil Location and Eye Tracking by Designing a Fully Conventional
Neural Network (FCNN)



Volume 6, Issue 2
Muhammad Raza, Changyuan Wang, Pengxiang XUE, Shahzadi Bano

Published online: 29 August 2020
Article Views: 20

Abstract

Eye-tracking has become a significant tool in various areas, incorporating interaction of humans with a computer, computer vision, psychology, and medical diagnostics. Various protocols have been utilized to track the gaze. Although, some may not be accurate in the real world, while others may need explicit custom calibration, which can cause problems. Few of these techniques are associated with low image grades and varying lighting situations. The latest success and popularity of deep studying has dramatically increased the effectiveness of eye-tracking. The convenience of huge datasets additionally improves the function of deep learning-depend techniques. This technical paper introduces the latest deep studying-depend gaze assessment technology with a concentration on Fully Convolution Neural Networks (FCNN). This technical paper also gives an overview of other machine-based eye assessment methods. This research objective to enable the research population to generate significant and useful horizons that can improve the structure and growth of better, more effective deep learning depends on eye-tracking methods. This paper also gives information on different pertained models, network structures, and open origin datasets to help train deep learning models.

Reference

  1. K. Wang, S. Wang, and Q. Ji, “Deep eye fixation map learning for calibration-free eye gaze tracking,” in Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, New York, NY, 2016. doi: https://doi.org/10.1145/2857491.2857515
  2. R. A. Naqvi, M. Arsalan, G. Batchuluun, H. S. Yoon, and K. R. Park, “Deep learning-based gaze detection system for automobile drivers using a NIR camera sensor,” Sensors, vol. 18, no. 2, p. 456, 2018. doi: https://doi.org/10.3390/s18020456
  3. S. Robertson, G. Penn, and Y. Wang. (2019) Exploring spectro-temporal features in end-to-end convolutional neural networks. [Online]. Available: https://arxiv.org/pdf/1901.00072.pdf
  4. Y. Kim, Y. Jernite, D. Sontag, and A. M. Rush, “Character-aware neural language models,” in Thirtieth AAAI Conference on Artificial Intelligence, Menlo Park, CA, 2016.
  5. W. Wang and J. Shen, “Deep visual attention prediction,” IEEE Transactions on Image Processing, vol. 27, no. 5, pp. 2368–2378, 2017. doi: https://doi.org/10.1109/TIP.2017.2787612
  6. A. Tsukada, M. Shino, M. Devyver, and T. Kanade, “Illumination-free gaze estimation method for first-person vision wearable device,” in International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, 2011. doi: https://doi.org/10.1109/ICCVW.2011.6130505
  7. D. W. Hansen and Q. Ji, “In the eye of the beholder: A survey of models for eyes and gaze,” Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 3, pp. 478–500, 2009. doi: https://doi.org/10.1109/TPAMI.2009.30
  8. C. N. M. Locsin and R. J. Ferolin, “Neural networks application for water distribution demand-driven decision support system,” Journal of Advances in Technology and Engineering Research, vol. 4, no. 4, pp. 160–175, 2018. doi: https://doi.org/10.20474/jater-4.4.3
  9. H. Chennamma and X. Yuan. (2013) A survey on eye-gaze tracking techniques. [Online]. Available: https://arxiv.org/abs/1312.6410
  10. M. Quazi, “An overview of laser welding of highstrength steels for automotive application,” International Journal of Technology and Engineering Studies, vol. 6, no. 1, pp. 23–40, 2020. doi: https://doi.org/10.20469/ijtes.6.10004-1
  11. X. Zhang, Y. Sugano, M. Fritz, and A. Bulling, “Appearance-based gaze estimation in the wild,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, CA, 2015.
  12. S. F. Haider, M. Quazi, J. Bhatti, M. N. Bashir, and I. Ali, “Effect of Shielded Metal Arc Welding (SMAW) parame-ters on mechanical properties of low-carbon, mild and stainless-steel welded joints: A review,” Journal of Advances in Technology and Engineering Research, vol. 5, no. 5, pp. 191–198, 2019. doi: https://doi.org/10.20474/jater-5.5.1
  13. C. Zhang, R. Yao, and J. Cai, “Efficient eye typing with 9-direction gaze estimation,” Multimedia Tools and Applications, vol. 77, no. 15, pp. 19 679–19 696, 2018. doi: https://doi.org/10.1007/s11042-017-5426-y
  14. N. Z. T. Abdulnabi and O. Altun, “Batch size for training convolutional neural networks for sentence classification,” Journal of Advances in Technology and Engineering Studies, vol. 2, no. 5, pp. 156–163, 2016. doi: https://doi.org/10.20474/jater-2.5.3
  15. I. Ali, N. Lin, M. Quazi, M. N. Bashir, H. Sadiq, and F. Sharaf, “Investigating the wear characteristics of metal matrix composite coating deposited on AA5083 Al-alloy by laser surface engineering technique,” North American Academic Research, vol. 3, no. 1, pp. 138–146, 2019. doi: https://doi.org/10.5281/zenodo.3626448
  16. Y. Sugano, Y. Matsushita, Y. Sato, and H. Koike,“Appearance-based gaze estimation with online calibration from mouse operations,” IEEE Transactions on Human-Machine Systems, vol. 45, no. 6, pp. 750–760, 2015. doi: https://doi.org/10.1109/THMS.2015.2400434
  17. X. Zhang, Y. Sugano, M. Fritz, and A. Bulling, “It’swritten all over your face: Full-face appearance-based gaze estimation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, 2017.
  18. M. Gul, N. Zulkifli, M. Kalam, H. Masjuki, M. Mujtaba, S. Yousuf, M. N. Bashir, W. Ahmed, M. Yusoff, S. Noor et al., “RSM and artificial neural networking based production optimization of sustainable cotton bio-lubricant and evaluation of its lubricity & tribological properties,” Energy Reports, vol. 7, pp. 830–839, 2021. doi: https://doi.org/10.1016/j.egyr.2021.01.033
  19. M. M. Quazi, M. Ishak, A. Arslan, M. Nasir Bashir,and I. Ali, “Scratch adhesion and wear failure characteristics of pvd multilayer crti/crtin thin film ceramic coating deposited on AA7075-T6 aerospace alloy,” Journal of Adhesion Science and Technology, vol. 32, no. 6, pp. 625–641, 2018. doi: https://doi.org/10.1080/01694243.2017.1373988
  20. Y. Wang, T. Shen, G. Yuan, J. Bian, and X. Fu,“Appearance-based gaze estimation using deep features and random forest regression,” Knowledge-Based Systems, vol. 110, pp. 293–301, 2016. doi: https://doi.org/10.1016/j.knosys.2016.07.038
  21. M. M. Quazi, M. Ishak, A. Arslan, M. Fazal, F. Yusof, B. Sazzad, M. N. Bashir, and M. Jamshaid, “Mechanical and tribological performance of a hybrid MMC coating deposited on Al-17Si piston alloy by laser composite surfacing technique,” RSC Advances, vol. 8, no. 13, pp. 6858–6869, 2018. doi:https://doi.org/10.1039/C7RA08191J
  22. A. George and A. Routray, “Real-time eye gaze direction classification using convolutional neural network,” in International Conference on Signal Processing and Communications (SPCOM), Bangalore, India, 2016.
  23. A. Kar and P. Corcoran, “Performance evaluationstrategies for eye gaze estimation systems with quantitative metrics and visualizations,” Sensors, vol. 18, no. 9, pp. 1–35, 2018. doi: https://doi.org/10.3390/s18093151
  24. R. Yamashita, M. Nishio, R. K. G. Do, and K. Togashi, “Convolutional neural networks: An overview and application in radiology,” Insights into Imaging, vol. 9, no. 4, pp. 611–629, 2018. doi:https://doi.org/10.1007/s13244-018-0639-9
  25. T. Huff, N. Mahabadi, and P. Tadi, Neuroanatomy,Visual Cortex. Treasure Island, FL, StatPearls Publishing, 2020.
  26. S. Wakeel, S. Ahmad, S. Bingol, M. N. Bashir, T. C.Paçal, and Z. A. Khan, “Supplier selection for high temperature die attach by hybrid entropy-range of value MCDM technique: A semiconductor industry,” in 21st International Conference on Electronic Packaging Technology (ICEPT), Guangzhou, China, 2020. doi: https://doi.org/10.1109/ICEPT50128.2020.9202994
  27. B. Ramsundar and R. B. Zadeh, TensorFlow fordeep learning: From linear regression to reinforcement learning. Sebastopol, CA: O’Reilly Media, Inc, 2018.
  28. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105, 2012.
  29. S. Vora, A. Rangesh, and M. M. Trivedi, “On generalizing driver gaze zone estimation using convolutional neural networks,” in Intelligent Vehicles Symposium (IV), Los Angeles, CA, 2017. doi:https://doi.org/10.1109/IVS.2017.7995822
  30. S. Rustagi, A. Garg, P. R. Anand, R. Kumar, Y. Kumar, and R. R. Shah, “Touchless typing using head movement-based gestures,” in IEEE Sixth International Conference on Multimedia Big Data (BigMM), New Delhi, India, 2020. doi: https://doi.org/10.1109/BigMM50055.2020.00025
  31. G. Bradski, “The openCV library,” Dr. Dobb’s Journal: Software Tools for the Professional Programmer, vol. 25, no. 11, pp. 120–123, 2000.
  32. S. Wakeel, S. Bingol, S. Ahmad, M. N. Bashir, M. S. M. M. Emamat, Z. Ding, and F. Hussain, “A new hybrid LGPMBWM-PIV method for automotive material selection,” Informatica, vol. 45, no. 1, pp.105–115, 2021.
  33. Y. Lee, C. Shin, A. Plopski, Y. Itoh, T. Piumsomboon, A. Dey, G. Lee, S. Kim, and M. Billinghurst, “Estimating gaze depth using multi-layer perceptron,” in International Symposium on Ubiquitous Virtual Reality (ISUVR), Nara, Japan, 2017. doi: https://doi.org/10.1109/ISUVR.2017.13
  34. L. L. Di Stasi, R. Renner, A. Catena, J. J. Cañas, B. M. Velichkovsky, and S. Pannasch, “Towards a driver fatigue test based on the saccadic main sequence: A partial validation by subjective report data,” Transportation Research Part C: Emerging Technologies, vol. 21, no. 1, pp. 122–133, 2012.
  35. S. D. Goldinger and M. H. Papesh, “Pupil dilationreflects the creation and retrieval of memories,” Current Directions in Psychological Science, vol. 21, no. 2, pp. 90–95, 2012. doi: https://doi.org/10.1177/0963721412436811
  36. S. Vora, A. Rangesh, and M. M. Trivedi, “Drivergaze zone estimation using convolutional neural networks: A general framework and ablative analysis,”IEEE Transactions on Intelligent Vehicles, vol. 3, no. 3, pp. 254–265, 2018. doi: https://doi.org/10.1109/TIV.2018.2843120
  37. C. Shin, G. Lee, Y. Kim, J. Hong, S.-H. Hong, H. Kang, and Y. Lee, “Evaluation of gaze depth estimation using a wearable binocular eye tracker and machine learning,” Journal of The Korea Computer Graphics Society, vol. 24, no. 1, pp. 19–26, 2018.
  38. M. Trokielewicz, A. Czajka, and P. Maciejewicz,“Post-mortem iris recognition with deep-learning-based image segmentation,” Image and Vision Computing, vol. 94, pp. 1–34, 2020. doi: https://doi.org/10.1016/j.imavis.2019.103866
  39. S. Jia, A. Seccia, P. Antonenko, R. Lamb, A. Keil,M. Schneps, M. Pomplun et al., “Biometric recognition through eye movements using a recurrent neural network,” in International Conference on Big Knowledge (ICBK), Singapore, 2018. doi: https://doi.org/10.1109/ICBK.2018.00016
  40. Z. Liang, F. Tan, and Z. Chi, “Video-based biometric identification using eye tracking technique,” in International Conference on Signal Processing, Communication and Computing (ICSPCC 2012), Hong Kong, China, 2012. doi: https://doi.org/10.1109/ICSPCC.2012.6335584
  41. O. Ferhat and F. Vilariño, “Low cost eye tracking: The current panorama,” Computational Intelligence and Neuroscience, vol. 2016, pp. 1–14, 2016. doi:https://doi.org/10.1155/2016/8680541
  42. W. Zhu and H. Deng, “Monocular free-head 3dgaze tracking with deep learning and geometry constraints,” in Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 2017.
  43. Y. Sugano, Y. Matsushita, and Y. Sato, “Learning-by-synthesis for appearance-based 3d gaze estimation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, 2014, pp. 1821–1828.
  44. D. Masko, “Calibration in eye tracking using transfer learning,” KTH Royal Institute of Technology, Stockholm, Sweden, M.S. thesis, 2017.
  45. C. Palmero, J. Selva, M. A. Bagheri, and S. Escalera. (2018) Recurrent cnn for 3d gaze estimation using appearance and shape cues. [Online]. Available: https://arxiv.org/abs/1805.03064
  46. J. Niemann and C. Fussenecker, “Analysis of eye tracking usage in different domains and possible applications in the engineering environment,” in International Conference on Quality and Innovation in Engineering and Management, Napoca, Romania, 2014.
  47. X. Zhou, H. Cai, Z. Shao, H. Yu, and H. Liu, “3deye model-based gaze estimation from a depth sensor,” in International Conference on Robotics and Biomimetics (ROBIO), Qingdao, China, 2016. doi: https://doi.org/10.1109/ROBIO.2016.7866350
  48. M. Gul, M. Kalam, M. Mujtaba, S. Alam, M. N. Bashir, I. Javed, U. Aziz, M. R. Farid, M. T. Hassan, and S. Iqbal, “Multi-objective-optimization of process parameters of industrial-gas-turbine fueled with natural gas by using Grey-Taguchi and ANN methods for better performance,” Energy Reports, vol. 6, pp. 2394–2402, 2020. doi: https://doi.org/10.1016/j.egyr.2020.08.002
  49. B. A. Smith, Q. Yin, S. K. Feiner, and S. K. Nayar, “Gaze locking: passive eye contact detection for human-object interaction,” in Proceedings of the 26th annual ACM symposium on User Interface Software and Technology, New York, NY, 2013. doi: https://doi.org/10.1145/2501988.2501994
  50. I. Ali, M. Quazi, E. Zalnezhad, A. A. Sarhan, N. L. Sukiman, and M. Ishak, “Hard anodizing of aerospace AA7075-T6 aluminum alloy for improving surface properties,” Transactions of the Indian Institute of Metals, vol. 72, no. 10, pp.2773–2781, 2019. doi: https://doi.org/10.1007/s12666-019-01754-5
  51. Vision Aware. (2020) Driving with low vision. [Online]. Available: https://bit.ly/3myBojj
  52. Y. LeCun. (2013) LeNet-5, convolutional neural networks. [Online]. Available: https://bit.ly/3BwOphx
  53. C. Ma, K.-A. Choi, B.-D. Choi, and S.-J. Ko, “Robust remote gaze estimation method based on multiple geometric transforms,” Optical Engineering, vol. 54, no. 8, p. 083103, 2015. doi: https://doi.org/10.1117/1.OE.54.8.083103
  54. C.-C. Lai, S.-W. Shih, and Y.-P. Hung, “Hybrid method for 3-D gaze tracking using glint and contour features,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 1, pp.24–37, 2014.
  55. R. Newman, Y. Matsumoto, S. Rougeaux, and A. Zelinsky, “Real-time stereo tracking for headpose and gaze estimation,” in Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), Grenoble, France, 2000. doi: https://doi.org/10.1109/AFGR.2000.840622
  56. F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, 2015.
  57. P. Paysan, R. Knothe, B. Amberg, S. Romdhani,and T. Vetter, “A 3d face model for pose and illumination invariant face recognition,” in 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, Genova, Italy. Ieee, 2009, pp. 296–301.
  58. O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deepface recognition,” in Proceedings of British Machine Vision Association, Swansea, UK, 2015. doi: https://dx.doi.org/10.5244/C.29.41
  59. A. Galante and P. Menezes, “A gaze-based interaction system for people with cerebral palsy,” Procedia Technology, vol. 5, pp. 895–902, 2012. doi: https://doi.org/10.1016/j.protcy.2012.09.099
  60. P. Blignaut, “Mapping the pupil-glint vector to gaze coordinates in a simple video-based eye tracker,” Journal of Eye Movement Research, vol. 7, no. 1, pp.1–11, 2014. doi: https://doi.org/10.16910/jemr.7.1.4
  61. J. Zhu and J. Yang, “Subpixel eye gaze tracking,” in Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition, Washington, DC, WA, 2002. doi: https://doi.org/10.1109/AFGR.2002.1004144
  62. Z. Zhu, Q. Ji, and K. P. Bennett, “Nonlinear eyegaze mapping function estimation via support vector regression,” in 18th International Conference on Pattern Recognition (ICPR’06), Hong Kong, China, vol. 1, 2006. doi: https://doi.org/10.1109/ICPR.2006.864
  63. J. Kim, M. Stengel, A. Majercik, S. De Mello, D. Dunn, S. Laine, M. McGuire, and D. Luebke, “Nvgaze: An anatomically-informed dataset for low-latency, near-eye gaze estimation,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, New York, NY, 2019. doi:https://doi.org/10.1145/3290605.3300780
  64. Y.-G. Shin, K.-A. Choi, S.-T. Kim, C.-H. Yoo, and S.-J. Ko, “A novel 2-D mapping-based remote eye gaze tracking method using two IR light sources,” in IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, 2015. doi: https://doi.org/10.1109/ICCE.2015.7066375
  65. J. M. Franchak, K. S. Kretch, K. C. Soska, and K. E. Adolph, “Head-mounted eye tracking: A new method to describe infant looking,” Child Development, vol. 82, no. 6, pp. 1738–1750, 2011. doi: https://doi.org/10.1111/j.1467-8624.2011.01670.x
  66. K. Simonyan and A. Zisserman. (2014) Very deep convolutional networks for large-scale image recognition. [Online]. Available: https://arxiv.org/abs/1409.1556

To Cite this article

M. Raza, C. Wang, P. XUE, and S. Bano, “Research on eye pupil location and eye tracking by designing a Fully Conventional Neural Network (FCNN),” International Journal of Technology and Engineering Studies, vol. 6, no. 2, pp. 52 -68, 2020.