View source for S15: Hand Gesture Recognition using IR Sensors



== Abstract ==
<p style="text-indent: 1em; text-align: justify;">
The aim of the project is to develop hand gesture recognition system using grid of IR proximity sensors.  Various hand gestures like swipe, pan etc. can be recognized. These gestures can be used to control different devices or can be used in various applications. The system will recognize different hand gestures based on the values received from IR proximity sensors. We have used QT to develop the application to demonstrate the working of the project.
</p>

== Objectives & Introduction ==
<p style="text-indent: 1em; text-align: justify;">
We use various hand gestures in our day-to-day life to communicate while trying to explain someone something, direct them somewhere etc. It would be so cool if we could communicate with various applications running on the computers or different devices around us understand the hand gestures and give the expected output. 

In order to achieve this, we are using a 3-by-3 grid of analog IR proximity sensors and connecting these sensors via multiplexers to the ADC pins on SJOne Board. As a hand is moved in front of the sensors, the sensor values would in a particular pattern enabling us to detect the gesture and instruct the application to perform the corresponding action.
</p>
=== Team Members & Responsibilities ===
*  Harita Parekh
**   Implementing algorithm for gesture recognition
**   Implementation of sensor data filters
*  Shruti Rao
**   Implementing algorithm for gesture recognition
**   Interfacing of sensors, multiplexers and controller
*  Sushant Potdar
**   Implementation of final sensor grid
**   Development of the application module

== Schedule ==

{| class="wikitable"
|-
! scope="col"| Week#
! scope="col"| Start Date
! scope="col"| End Date
! scope="col"| Task
! scope="col" width=70pt| Status
! scope="col"| Actual Completion Date
|-
|-
| 1
| align="center" |3/22/2015
| align="center" |3/28/2015
| Research on the sensors, order sensors and multiplexers
| Completed
| align="center" |3/28/2015
|-
|-
| 2
| align="center" |3/29/2015
| align="center" |4/4/2015
| Read the data sheet for sensors and understand its working. Test multiplexers
| Completed
| align="center" |4/04/2015
|-
|-
| 3
| align="center" |4/05/2015
| align="center" |4/11/2015
| Interfacing of sensors, multiplexers and controller
| Completed
| align="center" |4/15/2015
|-
|-
| 4
| align="center" |4/12/2015
| align="center" |4/18/2015
| 
* Implementation of sensor data filters

* Implement algorithm to recognize left-to-right movement
| Completed
| align="center" |4/25/2015
|-
|-
| 5
| align="center" |4/19/2015
| align="center" |4/25/2015
| 
* Implementation of final sensor grid

* Implement algorithm to recognize up-to-down movement

* Implement algorithm to recognize right-to-left movement
| Completed
| align="center" |5/02/2015
|-
|-
| 6
| align="center" |4/26/2015
| align="center" |5/02/2015
| 
* Implement algorithm to recognize down-to-up movement

* Develop the QT application
| Completed
| align="center" |5/09/2015
|-
|-
| 7
| align="center" |5/03/2015
| align="center" |5/09/2015
| Testing and bug fixes
| Completed
| align="center" |5/15/2015
|-
|-
| 8
| align="center" |5/10/2015
| align="center" |5/16/2015
| Testing and final touches
| Completed
| align="center" |5/22/2015
|-
|-
| 9
| align="center" |5/21/2015
| align="center" |5/24/2015
| Report Completion
| Completed
| align="center" |5/24/2015
|-
|-
| 10
| align="center" |5/25/2015
| align="center" |5/25/2015
| Final demo
| Scheduled
| align="center" |5/25/2015
|-
|}

== Parts List & Cost ==
{| class="wikitable"
|-
! scope="col"| SR#
! scope="col"| Component Name
! scope="col"| Quantity
! scope="col"| Price per component
! scope="col"| Total Price
|-
! scope="row"| 1
| Sharp Distance Measuring Sensor Unit (GP2Y0A21YK0F)
| align="center" |9
| $14.95
| $134.55
|-
! scope="row"| 2
| STMicroelectronics Dual 4-Channel Analog Multiplexer/Demultiplexer (M74HC4052)
| align="center" |3
| $
| $
|-
! scope="row"| 3
| SJ-One Board
| align="center" |1
| $80
| $80
|-
! scope="row"| 4
| USB-to-UART converter
| align="center" |1
| $7
| $7
|-
! align="left" colspan="4" | Total
| $
|}

== Design & Implementation ==
=== Hardware Design ===
The image shows the setup of the project.
<table>
<tr valign="top" align="justify">
<b>System Block Diagram:</b>
</tr>
<br>
<tr>
<td valign="top" align="center">
[[File:S15 244 Grp10 Ges block diagram.png|700px]]<br>
</td>
</tr>

<tr align="justify">
The system consists of 9 IR proximity sensors, which are arranged in 3x3 grid. The output of the sensors is given to the Analog-to-Digital convertor on the SJOne Board to get the digital equivalent of the voltage given by the sensors. Since there are only 3 ADC channels exposed on the pins on the board, we cannot connect all the sensors directly to the board. For these we have used three multiplexers, which has 3 sensors each connected to its input. The output of the multiplexers is connected to ADC. SJOne board is connected to the laptop via UART-to-USB connection.
</tr>

<tr>
<td valign="top" align="justify" width=950px>
<b>Proximity Sensor:</b>
<br>
This sensor by Sharp measures the distance from an obstacle by bouncing IR rays off the obstacle. This sensor can measure distances from 10 to 80 cms. The sensor returns an analog voltage corresponding to the distance from the obstacle. Depending on which sensor returns valid values, validations could be made and hand movement can be determined. There is no external circuitry required for this sensor. The operating voltage recommended for this sensor is 4.5V to 5.5V.
</td>
<td>
[[File:S15 244 G10 Ges sensor.jpg|200px]]
</td>
</tr>

<tr>
<td valign="top" align="justify" width=950px>
<b>Multiplexer:</b>
<br>
The chip used in the project is M74HC4052 from STMicroelectronics. This is a dual 4-channel multiplexer/demultiplexer. Due to shortage of ADC pins to interface with the sensors, use of multiplexer is required. The multiplexer takes input from three sensors and enables only one of them at the output. The program logic decides which sensor’s output should be enabled at the multiplexer’s output. A and B control signals select one of the channel out of the four. The operating voltage for the multiplexer is 2 to 6V.
</td>
<td>
[[File:S15 244 G10 Ges mux.jpg|200px]]
</td>
</tr>

<tr>
<td valign="top" align="justify" width=950px>
<b>USB-to-UART converter:</b>
<br>
To communicate to SJONE board over UART there is a need an USB to serial converter and a MAX232 circuit to convert the voltage levels to TTL, which the SJONE board understands. Instead it’s better to use a UART-to-USB converter to avoid the multiple conversions. This is done using CP2102 IC, which is similar to a FTDI chip.
</td>
<td>
[[File:S15_244_Grp10_Ges_UART_to_USB.JPG|200px]]
</td>
</tr>
</table>

=== Hardware Interface ===
<table>
<tr>
<td valign="top" align="center">
Pin connections for IR Sensor to Multiplexer:<br>
[[File:S15 244 Grp10 Ges sensor-to-mux.png|500px]]
</td>
</tr>
<tr>
<td valign="top" align="center">
Pin connections on SJOne board:<br>
[[File:S15 244 Grp10 Ges sjone pinouts.png|400px]]
</td>
<td valign="top" align="center">
Connections between SJOne board and USB-to-UART Converter:<br>
[[File:S15 244 Grp10 Ges SJOne to UART.JPG|360px|x400px]]
</td>
</tr>
</table>

=== Software Design ===
<table>
<tr>
<td>

</td>
<td valign="top" align="justify" width=400px>
<b>Filter Algorithm</b>
</tr>
</table>

<table>
<tr>
<td valign="top" align="justify" width=700px>
<b>State Machine Diagram:</b>




<br>
</td>
</tr>
<tr>
<td valign="top" align="justify" width=400px>
<b>Application Development</b><br>
<i>QT Software</i><br>
QT is a cross-platform application framework that is widely used in developing application software that can be run on various software and hardware platforms with little or no change in the underlying codebase while having the speed and the power of native application. [1]
It is mainly used to make GUI based applications but there can be applications such as consoles or command-line applications developed in QT. QT is preferred by many application programmers as it helps in developing GUI applications in C++ as it uses the standard C++ libraries for backend. 
Platforms supported by Qt are:
* Android
* Embedded Linux
* Integrity
* iOS
* OSX
* QNX
* VxWorks
* Waylands
* Windows
* Windows CE
* Windows RT
* X11
QT applications are highly portable from one platform to other as QT first runs a Qmake function before compiling the source code. It is very similar to ‘cmake’ which is used for cross platform compilation of any source code. The qmake auto generates a makefile depending on the operating system and the compiler used for the project. So if a project is to be ported from windows to linux based system then the qmake auto generated a new makefile with arguments and parameters that the g++ compiler expects.
</td>
<td width=300px align="center">
[[File:S15 244 Grp10 Ges qtdevices.png|300px]]<br>
[[File:S15 244 Grp10 Ges qt-sdk.png|300px]]
</td>
</tr>
<tr>
<td valign="top" align="justify" width=700px>
<i>Gesture Recognition application on QT</i><br>
Once the gesture is recognized on the SJONE board, a message is sent over UART2 stating which gesture was sensed. The Qt application gets this message from the COMM port and scrolls the images left and right based on the left/right gesture and it moves a vertical slider up and down which in turn changes the value on a LCD screen display.

The application opens up in a window that has 2 tabs, config and App. The config tab includes the fields required to open the COMM port and test the COMM port using a loopback connection.<br>
[[File:S15 244 Grp10 Ges qt tab1.JPG|center|650px]]<br>
This tab has the following QObjects<br>
[[File:S15 244 Grp10 Ges table1.png|center|650px|x480px]]<br>
The App tab includes the objects required to change images and change the value in the vertical slider and lcd number display.<br>
[[File:S15 244 Grp10 Ges qt tab2.JPG|center|650px]]<br>
This tab has the following Qobjects<br>
[[File:S15 244 Grp10 Ges table2.png|center|650px|x420px]]<br>


</td>
</tr>
</table>

=== Implementation ===
This section includes implementation, but again, not the details, just the high level.  For example, you can list the steps it takes to communicate over a sensor, or the steps needed to write a page of memory onto SPI Flash.  You can include sub-sections for each of your component implementation.

== Testing & Technical Challenges ==
Describe the challenges of your project.  What advise would you give yourself or someone else if your project can be started from scratch again?
Make a smooth transition to testing section and described what it took to test your project.

Include sub-sections that list out a problem and solution, such as:

=== My Issue #1 ===
Discuss the issue and resolution.

== Conclusion ==
Conclude your project here.  You can recap your testing and problems.  You should address the "so what" part here to indicate what you ultimately learnt from this project.  How has this project increased your knowledge?

=== Project Video ===
Upload a video of your project and post the link here.

=== Project Source Code ===
*  [https://sourceforge.net/projects/sjsu/files/CmpE_SJSU_S2015/ Sourceforge Source Code Link]

== References ==
=== Acknowledgement ===
<p style="text-indent: 1em; text-align: justify;">
All the components where procurred from Amazon, Adafruit and digikey. We are thankful to Preet for his continuous guidance during the project.
</p>

=== References Used ===
<p style="width: 700px; text-align: justify;">
[http://www.sharpsma.com/webfm_send/1208 IR Sensor Data Sheet]<br>
[http://www.nxp.com/documents/data_sheet/LPC1769_68_67_66_65_64_63.pdf LPC_USER_MANUAL]<br>
[http://www.st.com/web/en/resource/technical/document/datasheet/CD00002398.pdf Multiplexer Data Sheet]<br>
[http://en.wikipedia.org/wiki/Qt_%28software%29 QT Software]<br>
[http://www.socialledge.com/sjsu/index.php?title=S14:_Virtual_Dog Filter code refered from Spring'14 project Virtual Dog]
</p>