Difference between revisions of "S15: Hand Gesture Recognition using IR Sensors"

Proximity Sensor:

This sensor by Sharp measures the distance from an obstacle by bouncing IR rays off the obstacle. This sensor can measure distances from 10 to 80 cms. The sensor returns an analog voltage corresponding to the distance from the obstacle. Depending on which sensor returns valid values, validations could be made and hand movement can be determined. The voltage returned by the sensor increases as the obstacle approaches the sensor. There is no external circuitry required for this sensor. The operating voltage recommended for this sensor is 4.5V to 5.5V.

Multiplexer:

The chip used in the project is M74HC4052 from STMicroelectronics. This is a dual 4-channel multiplexer/demultiplexer. Due to shortage of ADC pins to interface with the sensors, use of multiplexer is required. The multiplexer takes input from three sensors and enables only one of them at the output. The program logic decides which sensor’s output should be enabled at the multiplexer’s output. A and B control signals select one of the channel out of the four. The operating voltage for the multiplexer is 2 to 6V.

USB-to-UART converter:

To communicate to SJONE board over UART there is a need an USB to serial converter and a MAX232 circuit to convert the voltage levels to TTL, which the SJONE board understands. Instead it’s better to use a USB-to-UART converter to avoid the multiple conversions. This is done using CP2102 IC, which is similar to a FTDI chip.

Pin connections on SJOne board:


Figure 7: Pin connections on SJOne board

Connections between SJOne board and USB-to-UART Converter:

Figure 8: Connections between SJOne board and USB-to-UART Converter

Initialization

SJOne board has 3 ADC pins exposed on Port 0 (0.26) and Port 1 (1.30 and 1.31). To use these pins as ADC, the function should be selected in PINSEL.

The GPIO pins on Port 2 are connected to the select pins on multiplexer. These pins should be initialized as output pins. Once, initialization is completed, the function for normalizing the sensor values is called.

SJOne board uses UART 3 to communicate with the QT application. UART 3 is initialized to baud rate of 9600 with receiver buffer as 0 and transmission buffer as 32 bytes. Once initialization is completed, the function for processing sensor values is called.

Filter Algorithm

The current value of sensor is fetched by setting the corresponding values on the multiplexer select pins and reading the output of the ADC. A queue of size 5 is maintained and the fetched value is inserted at the tail of the queue. This queue is sorted using bubble sort. The median value of the queue is checked to be greater than 2000 (sensor returns a voltage corresponding to a value greater than 2000 when the hand is near enough to it) and gesture array for that sensor is set accordingly.

Gesture Recognition Algorithm

Different sets of sensors are monitored in order to recognize a valid pattern in the sensor output and thereby recognize the gesture pattern.
(We have assumed that the sensors are numbered 0 through 8 and the corresponding value for the sensor is set by the filter algorithm in gesture[] array).

Pattern 1:

Here the three sensors present at the top left corner are monitored.

• If sensor1 value is zero
  • Check the values of sensors in the second column. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Check the values of sensors in the third column. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Send “Right” to UART 3.

• If sensor3 value is zero
  • Check the values of sensors in the second row. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Check the values of sensors in the third row. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Send “Down ” to UART 3.

Pattern 2:

Here the three sensors present at the top right corner are monitored.

• If sensor1 value is zero
  • Check the values of sensors in the second column. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Check the values of sensors in the first column. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Send “Left ” to UART 3.

• If sensor5 value is zero
  • Check the values of sensors in the second row. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Check the values of sensors in the third row. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Send “Down ” to UART 3.

Pattern 3:

Here the three sensors present at the bottom left corner are monitored.

• If sensor7 value is zero
  • Check the values of sensors in the second column. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Check the values of sensors in the third column. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Send “Right ” to UART 3.

• If sensor3 value is zero
  • Check the values of sensors in the second row. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Check the values of sensors in the first row. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Send “Up ” to UART 3.

Pattern 4:

Here the three sensors present at the bottom right corner are monitored.

• If sensor7 value is zero
  • Check the values of sensors in the second column. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Check the values of sensors in the first column. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Send “Left ” to UART 3.

• If sensor5 value is zero
  • Check the values of sensors in the second row. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Check the values of sensors in the first row. If the combination of first two or last two sensors is 1, go to next step. Else, update the gesture array for these sensors and check.
    • Send “Up ” to UART 3.

Application Development
Qt Software

Qt is a cross-platform application framework that is widely used in developing application software that can be run on various software and hardware platforms with little or no change in the underlying codebase while having the speed and the power of native application.

It is mainly used to make GUI based applications but there can be applications such as consoles or command-line applications developed in Qt. Qt is preferred by many application programmers as it helps in developing GUI applications in C++ as it uses the standard C++ libraries for backend.

Platforms supported by Qt are:

• Android
• Embedded Linux
• Integrity
• iOS
• OSX
• QNX
• VxWorks
• Waylands
• Windows
• Windows CE
• Windows RT
• X11

Qt applications are highly portable from one platform to other as Qt first runs a Qmake function before compiling the source code. It is very similar to ‘cmake’ which is used for cross platform compilation of any source code. The qmake auto generates a makefile depending on the operating system and the compiler used for the project. So if a project is to be ported from windows to linux based system then the qmake auto generated a new makefile with arguments and parameters that the g++ compiler expects.

Gesture Recognition application on Qt

Once the gesture is recognized on the SJONE board, a message is sent over UART3 stating which gesture was sensed. The Qt application gets this message from the COM port and scrolls the images left and right based on the left/right gesture and it moves a vertical slider up and down which in turn changes the value on a LCD screen display.

The application opens up in a window that has 2 tabs, config and App. The config tab includes the fields required to open the COMM port and test the COMM port using a loopback connection.

This tab has the following QObjects.

The App tab includes the objects required to change images and change the value in the vertical slider and lcd number display.

This tab has the following Qobjects.

The below diagram shows the setup of the files used in the project.

The main function creates the tasks required for the application namely, SensorTask, WaitTask and ProcessSensor.

The tasks are declared in Hand_Gesture.hpp file.

This file also includes the enumerations required in the program logic.

• sensorNo_t: This enumeration assigns numbers to the sensor from 0 through 8.
• portPins_t: This enumeration assigns the shifting logic to P2_0 through P2_5 for better readability.

Hand_Gesture.cpp file contains the definitions of all the functions as listed below.

The WaitTask is used internally in ProcessSensor to wait for values of sensor to change while navigating through the above described algorithm. This task enables the system to wait for maximum of 5 seconds to receive any change of values in the sensor. If after 5 seconds, the sensor values are not as desired by the algorithm (to proceed to next stage), the sensor values are concluded to be false positives and further execution is skipped.

The constructor of SensorTask includes the initialization of pins for ADC and Port 2 pins used as select lines for the multiplexer.

UART 3 is initialized using the Singleton pattern.

The init function of UART is used to initialize UART 3 to communicate at the baud rate of 9600. Since the application will only be sending values to the Qt application and not expecting any replies, the receive buffer is set to zero and the transmission buffer is set to 32 bytes.

Includes and app initialization
This sections includes all header files required for the app and the constructor and destructor for the app. As we are displaying multiple images in the app the path of these images have to be hardcoded. We also define an enum to capture the received gesture. An instance of the serial port is also created in the constructor, and an initial welcome image (SJSU logo) is displayed on the app.

Configuration COM port
The COM port is configured in this section of the code, the name of the COM port and the baud rate is picked up from the lineEdits present on the config tab and passed to populate the structure of the serial port. If there is an error opening the COM port then an error message is displayed. The COM port is closed once the disconnect button is pressed.

Serial receive and loopback test
This section includes the code to copy all the data received in the receive buffer parse it and take the decision on which kind of gesture was received and take the appropriate decision. The loop back section picks up anything written on the Tx lineEdit and displays the received buffer on the rx lineEdit.

Open Image
The logic to switch between images and to loop through the images is implemented here.

Scroll
The position of the vertical slider is changed depending on the up/down gesture, the LCDnumber display is used to display the position (value) of the vertical slider. The value ranges from 0-99 and loops through if the value is greater 99 or smaller than 0.

Sensor setup

Screenshots of Qt application