Arduino image processing Eye Pupil Tracking, how to make Haarcascade XML file

3,496

Table of Contents

Eye Pupil Tracking, Image Processing based project Description:

eye pupil tracking– A few months back I uploaded a project on “Arduino Image Processing human face recognition and Entrance control using the electronic lock, vb.net and EmguCv“. In this project, we successfully detected the human face using the frontalface XML file and controlled an electronic lock. The face recognition was possible due to the frontalface XML file, I recommend you should watch this tutorial first and then you can resume from here, as today we will be using the same project. You can create your own XML files and track almost anything you want.

In this tutorial, I will practically show you how to train an XML file for tracking the eye pupil, while the same steps can be used to train an XML file for any type of object.

In today’s episode, you will learn how to make your own XML file for eye pupil tracking. Today’s episode covers

Steps to make your own XML file.
How to use an XML file in vb.net programming.

For the complete step-by-step explanation, watch video Tutorial given at the end.

Amazon Links:

Arduino Nano USB-C Type (Recommended)

12v Adaptor:

Other Tools and Components:

ESP32 WiFi + Bluetooth Module (Recommended)

Top Arduino Sensors:

Super Starter kit for Beginners

Digital Oscilloscopes

Variable Supply

Digital Multimeter

Soldering iron kits

PCB small portable drill machines

*Please Note: These are affiliate links. I may make a commission if you buy the components through these links. I would appreciate your support in this way!

Eye Pupil Tracking XML file training:

For creating the Haarcascade XML file you will need software.

As I want to track blue eyes, so that’s why I download pictures having blue eyes. If you want to track brown eyes or black eyes then simply download images consisting of brown or black color eyes. Or you can also take pictures of your own eye…

The next step is to change the format of these images, as these images are of the type jpg, let’s convert these images to BMP format using adobe photoshop. This process is really simple, simply open the image and save as the image with a different name and select BMP as the format.

We have a total of 6 images of the type bmp.

This is the software that I will be using form making the XML file.

Open the cascades folder and delete all of these folders…

Now open the negative folder, this folder consists of the negative images. Negative images are those images that do not consist of the object you want to track. You can also add more negative images of the type jpg…currently this folder have 200 negative images… The bg and create_list these two files are not included. Double click the create_list file.

Now open the positive folder then open the raw data folder, simply delete these images

And paste the images consisting of eyes, these are the positive images.

now click on the object maker….now select the region you want to track…then simply press the space bar key on your keyboard and then press the enter key…repeat the same steps for all the images…

After you are done with all the images then Double click the samples creations file, which will make a file in the vector folder, which you can check. By Default the vector folder is empty.

Right click on the haar training and select edit with notepad…the negative images are 200…while the positive images are 6. Update the number of positive images and negative images.

Now double click on the haar training file and wait for a while, this can take some time.

Now the last step is to double click on the converted file to make the XML file…as you can see the XML file with the name myhaar is created…

Rename this file with the name eyepupil…

Now, this XML file is ready and can be used with Arduino, raspberry pi, you can use this XML file with emgucv, opencv, python, etc…

This is the same application which I designed for the image processing based entrance control system…i have already explained this project in very detailed. You can watch this tutorial “Click here link2”.

The changes I made in this application can be watched in the video given below at the end of this article.

Eye Pupil Tracking Image Processing Programming:

For software changes, watch video given below.

Imports Emgu.CV
Imports Emgu.CV.Util

Imports Emgu.CV.Structure
Imports System. Diagnostics

Imports System.IO
Imports System.IO.Ports
Imports System.Threading


Public Class Form1


    Dim facedetected As Integer

    Dim facepresent As Integer

    Dim detectTimecount As Integer

    Dim web As Capture = New Capture(0)



    Private Sub Timer1_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Timer1.Tick

        ' cars()
        faces()
    End Sub


    Sub faces()


        Dim photo As Image(Of Bgr, Byte)
        photo = web.RetrieveBgrFrame
        Dim currentCenter As New Point()

        ' Dim eyedetection As New CascadeClassifier("C:\data\haarcascades\haarcascade_eye.xml") '  ' this one is for eye detection
        Dim facedetection As New CascadeClassifier("C:\data\haarcascades\eyepupil.xml") ' this one is for the face detection

        facedetected = 0


        Try
            Dim image As Image(Of Gray, Byte) = photo.Convert(Of Gray, Byte)()

            For Each face As Rectangle In facedetection.DetectMultiScale(image, 1.15, 8, Size.Empty, Size.Empty) ' default 1.1, 8  ( while best values are 1.2 and 17 after checking)

                photo.Draw(face, New Bgr(Color.White), 4)

                currentCenter.X = CInt(CDbl(face.X + face.X + face.Width) / 2.0)
                currentCenter.Y = CInt(CDbl(face.Y + face.Y + face.Height) / 2.0)

                Label1.Text = (currentCenter.X).ToString
                Label2.Text = (currentCenter.Y).ToString

                Dim f = New MCvFont(Emgu.CV.CvEnum.FONT.CV_FONT_HERSHEY_COMPLEX, 0.5, 0.5)
                photo.Draw("Human detected and scanning....", f, currentCenter, New Bgr(0, 255, 0)) ' New Point(10, 80) 

                Dim f2 = New MCvFont(Emgu.CV.CvEnum.FONT.CV_FONT_HERSHEY_COMPLEX, 0.7, 0.7)
                photo.Draw("human detection ", f2, New Point(10, 20), New Bgr(0, 255, 255)) ' New Point(10, 80)


                Label3.Text = face.ToString

                facedetected = 1  ' we are storing 1 if faces are detected.


            Next

            PictureBox1.Image = photo.ToBitmap


        Catch ex As Exception

        End Try

        If facedetected = 1 Then
            lblfacedetect.Text = "Scanning..."
            Timer3.Enabled = True

        Else
            lblfacedetect.Text = "no Human..."
            Timer3.Enabled = False
            detectTimecount = 0
            facepresent = 0

            '  SerialPort1.Open()
            '  SerialPort1.Write("d")
            '  SerialPort1.Close()

        End If



        ' new addition end 

        If detectTimecount >= 3 Then  ' HUMAN FACE IN CENTER

            ' detectTimecount = 0
            SerialPort1.Open()
            Label4.Text = "Human detected"
            SerialPort1.Write("c") ' the human is on front
            SerialPort1.Close()

        End If

    End Sub



    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        SerialPort1.Close()
        SerialPort1.PortName = "com15"
        SerialPort1.BaudRate = "9600"
        SerialPort1.DataBits = 8
        SerialPort1.Parity = Parity.None
        SerialPort1.StopBits = StopBits.One
        SerialPort1.Handshake = Handshake.None
        SerialPort1.Encoding = System.Text.Encoding.Default
    End Sub


    Private Sub Timer3_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Timer3.Tick
        detectTimecount = detectTimecount + 1
        Label4.Text = detectTimecount
    End Sub

    Private Sub Timer2_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Timer2.Tick
        SerialPort1.Open()
        SerialPort1.Write("d") ' no human
        SerialPort1.Close()
    End Sub



    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        SerialPort1.Open()
        SerialPort1.Write("c") ' the human is on front
        SerialPort1.Close()
    End Sub


    Private Sub Button1_Click_1(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        SerialPort1.Open()
        SerialPort1.Write("c") ' the human is on front
        SerialPort1.Close()
    End Sub
End Class

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

Imports Emgu.CV

Imports Emgu.CV.Util

Imports Emgu.CV.Structure

Imports System. Diagnostics

Imports System.IO

Imports System.IO.Ports

Imports System.Threading

Public Class Form1

Dim facedetected As Integer

Dim facepresent As Integer

Dim detectTimecount As Integer

Dim web As Capture = New Capture(0)

Private Sub Timer1_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Timer1.Tick

' cars()

faces()

End Sub

Sub faces()

Dim photo As Image(Of Bgr, Byte)

photo = web.RetrieveBgrFrame

Dim currentCenter As New Point()

' Dim eyedetection As New CascadeClassifier("C:\data\haarcascades\haarcascade_eye.xml") ' ' this one is for eye detection

Dim facedetection As New CascadeClassifier("C:\data\haarcascades\eyepupil.xml") ' this one is for the face detection

facedetected = 0

Try

Dim image As Image(Of Gray, Byte) = photo.Convert(Of Gray, Byte)()

For Each face As Rectangle In facedetection.DetectMultiScale(image, 1.15, 8, Size.Empty, Size.Empty) ' default 1.1, 8 ( while best values are 1.2 and 17 after checking)

photo.Draw(face, New Bgr(Color.White), 4)

currentCenter.X = CInt(CDbl(face.X + face.X + face.Width) / 2.0)

currentCenter.Y = CInt(CDbl(face.Y + face.Y + face.Height) / 2.0)

Label1.Text = (currentCenter.X).ToString

Label2.Text = (currentCenter.Y).ToString

Dim f = New MCvFont(Emgu.CV.CvEnum.FONT.CV_FONT_HERSHEY_COMPLEX, 0.5, 0.5)

photo.Draw("Human detected and scanning....", f, currentCenter, New Bgr(0, 255, 0)) ' New Point(10, 80)

Dim f2 = New MCvFont(Emgu.CV.CvEnum.FONT.CV_FONT_HERSHEY_COMPLEX, 0.7, 0.7)

photo.Draw("human detection ", f2, New Point(10, 20), New Bgr(0, 255, 255)) ' New Point(10, 80)

Label3.Text = face.ToString

facedetected = 1 ' we are storing 1 if faces are detected.

PictureBox1.Image = photo.ToBitmap

Catch ex As Exception

End Try

If facedetected = 1 Then

lblfacedetect.Text = "Scanning..."

Timer3.Enabled = True

Else

lblfacedetect.Text = "no Human..."

Timer3.Enabled = False

detectTimecount = 0

facepresent = 0

' SerialPort1.Open()

' SerialPort1.Write("d")

' SerialPort1.Close()

End If

' new addition end

If detectTimecount >= 3 Then ' HUMAN FACE IN CENTER

' detectTimecount = 0

SerialPort1.Open()

Label4.Text = "Human detected"

SerialPort1.Write("c") ' the human is on front

SerialPort1.Close()

End If

End Sub

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

SerialPort1.Close()

SerialPort1.PortName = "com15"

SerialPort1.BaudRate = "9600"

SerialPort1.DataBits = 8

SerialPort1.Parity = Parity.None

SerialPort1.StopBits = StopBits.One

SerialPort1.Handshake = Handshake.None

SerialPort1.Encoding = System.Text.Encoding.Default

End Sub

Private Sub Timer3_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Timer3.Tick

detectTimecount = detectTimecount + 1

Label4.Text = detectTimecount

End Sub

Private Sub Timer2_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Timer2.Tick

SerialPort1.Open()

SerialPort1.Write("d") ' no human

SerialPort1.Close()

End Sub

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click

SerialPort1.Open()

SerialPort1.Write("c") ' the human is on front

SerialPort1.Close()

End Sub

Private Sub Button1_Click_1(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click

SerialPort1.Open()

SerialPort1.Write("c") ' the human is on front

SerialPort1.Close()

End Sub

End Class

Watch Video Tutorial:

Other Arduino Image Processing based Projects:

OCR Optical Character recognition using VB.net and Arduino

Arduino Image processing CCTV camera system using VB.net

Arduino Image Processing based Entrance lock Control System

2 Comments

David Ruttenberg says:

August 24, 2019 at 3:22 am

Dear Engr Fahad,
I enjoyed watching your video regarding eye pupil tracking. If you wouldn’t mind, I would like to speak with you about your interest and availability in creating a similar system for my use in an academic research environment. I have some specific objectives, but have no idea if you are at all interested or available to speak about this. At your convenience, sir, please contact me via email to advise. In advance of your reply, I thank you.
-David

1. Engr Fahad says:
  
  August 24, 2019 at 11:56 pm
  
  here is my WhatsApp number
  +923339218213