Controlling Mouse using your Eye (with Pupil)

A while back I mentioned that I was working on a an Eye-Tracking Project called Pupil with a couple of friends of mine. It’s come a long way now and we’ve have had considerable success - but that’s for another post. Pupil is a great tool for eye-tracking and getting gaze data that can be used with other applications in realtime. It utilizes two HD cameras mounted on an eye-glasses frame at 180˚ from each other. One is called ‘Eye Camera’ that records your eye movements and the other is called ‘World Camera’ which is used to process exactly where the user is looking at. This data can be streamed to other applications over TCP in real time.

We wanted to control the mouse using our eye gestures, but it heavily relied on the World Camera and defining virtual screens called surfaces within the application. This obviously resulted in greater accuracy since it moved the pointer only when you actually looked at the screen. But we didn’t have a fixed World Cam and wanted to control the mouse using raw gaze data, i.e. use the data from your current gaze at the world (instead at the screen) and move the cursor to the appropriate coordinates on the screen w.r.t that gaze. So what we did was, we made small changes to the provided script, removing any reference to the ‘surface data’ and utilized the raw norm_gaze data to move the cursor.

Required Dependencies:

You need to have PyMouse and Xlib installed to make the script work. You can install them by running these commands:

$ sudo pip install pymouse
$ sudo apt-get install python-xlib

To Control the Mouse:

1. Run Pupil
2. Adjust Focus, Zoom, ROI so that your eye is perfectly tracked
3. Do the Calibration Process in the World View (We prefer the Natural Features method, since we don’t use a fixed World Cam). Make sure to use lots of points. You can start by pressing c on your keyboard.
4. Once your gaze is also being perfectly tracked, start the TCP Streaming Server by pressing s on your keyboard
5. Make sure everything’s working fine by visiting http://0.0.0.0:5000 in your browser.
6. Now, in a new terminal, run the script: $ python mouse_control.py

The Script (mouse_control.py):

import zmq
from pymouse import PyMouse

#mouse setup
m = PyMouse()
x_dim, y_dim = m.screen_size()

#network setup
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://127.0.0.1:5000")
#filter by messages by stating string 'STRING'. '' receives all messages
socket.setsockopt(zmq.SUBSCRIBE, '')
smooth_x, smooth_y= 0.5, 0.5

while True:
    msg = socket.recv()
    items = msg.split("\n")
    msg_type = items.pop(0)
    items = dict([i.split(':') for i in items[:-1] ])
    if msg_type == 'Pupil':
        try:
            my_gaze = items['norm_gaze']

            if my_gaze != "None":
                raw_x,raw_y = map(float,my_gaze[1:-1].split(','))

                # smoothing out the gaze so the mouse has smoother movement
                smooth_x += 0.5 * (raw_x-smooth_x)
                smooth_y += 0.5 * (raw_y-smooth_y)

                x = smooth_x
                y = smooth_y

                y = 1-y # inverting y so it shows up correctly on screen
                x *= x_dim
                y *= y_dim
                # PyMouse or MacOS bugfix - can not go to extreme corners
                # because of hot corners?
                x = min(x_dim-10, max(10,x))
                y = min(y_dim-10, max(10,y))

                m.move(x,y)
        except KeyError:
            pass
    else:
        # process non gaze position events from plugins here
        pass