A while back I mentioned that I was working on a an Eye-Tracking Project called Pupil with a couple of friends of mine. It’s come a long way now and we’ve have had considerable success - but that’s for another post. Pupil is a great tool for eye-tracking and getting gaze data that can be used with other applications in realtime. It utilizes two HD cameras mounted on an eye-glasses frame at 180˚ from each other. One is called ‘Eye Camera’ that records your eye movements and the other is called ‘World Camera’ which is used to process exactly where the user is looking at. This data can be streamed to other applications over TCP in real time.
We wanted to control the mouse using our eye gestures, but it heavily relied on the World Camera and defining virtual screens called
surfaces within the application. This obviously resulted in greater accuracy since it moved the pointer only when you actually looked
at the screen. But we didn’t have a fixed World Cam and wanted to control the mouse using raw gaze data, i.e. use the data from your
current gaze at the world (instead at the screen) and move the cursor to the appropriate coordinates on the screen w.r.t that gaze.
So what we did was, we made small changes to the provided script, removing any reference
to the ‘surface data’ and utilized the raw
norm_gaze data to move the cursor.
You need to have
Xlib installed to make the script work. You can install them by running these commands:
$ sudo pip install pymouse $ sudo apt-get install python-xlib
To Control the Mouse:
- Run Pupil
- Adjust Focus, Zoom, ROI so that your eye is perfectly tracked
- Do the Calibration Process in the World View (We prefer the
Natural Featuresmethod, since we don’t use a fixed World Cam). Make sure to use lots of points. You can start by pressing
con your keyboard.
- Once your gaze is also being perfectly tracked, start the TCP Streaming Server by pressing
son your keyboard
- Make sure everything’s working fine by visiting http://0.0.0.0:5000 in your browser.
- Now, in a new terminal, run the script:
$ python mouse_control.py
The Script (
import zmq from pymouse import PyMouse #mouse setup m = PyMouse() x_dim, y_dim = m.screen_size() #network setup context = zmq.Context() socket = context.socket(zmq.SUB) socket.connect("tcp://127.0.0.1:5000") #filter by messages by stating string 'STRING'. '' receives all messages socket.setsockopt(zmq.SUBSCRIBE, '') smooth_x, smooth_y= 0.5, 0.5 while True: msg = socket.recv() items = msg.split("\n") msg_type = items.pop(0) items = dict([i.split(':') for i in items[:-1] ]) if msg_type == 'Pupil': try: my_gaze = items['norm_gaze'] if my_gaze != "None": raw_x,raw_y = map(float,my_gaze[1:-1].split(',')) # smoothing out the gaze so the mouse has smoother movement smooth_x += 0.5 * (raw_x-smooth_x) smooth_y += 0.5 * (raw_y-smooth_y) x = smooth_x y = smooth_y y = 1-y # inverting y so it shows up correctly on screen x *= x_dim y *= y_dim # PyMouse or MacOS bugfix - can not go to extreme corners # because of hot corners? x = min(x_dim-10, max(10,x)) y = min(y_dim-10, max(10,y)) m.move(x,y) except KeyError: pass else: # process non gaze position events from plugins here pass