View source for S18: Hand gesture controlled multiplayer game

== Project Title ==
===Hand gesture controlled multiplayer game===

== Abstract ==
Human-machine interfaces are constantly evolving. The interfaces used in HMI, based on natural gestures of users, allow converting these movements into commands for a computer system. In this project we developed a multiplayer Ping-Pong game using a 16x32 RGB LED Matrix, accelerometer, wireless connectivity, FreeRTOS APIs and SJOne boards. The game works as a classic ping pong game in which there are two sliders which are controlled wirelessly by two hand gesture controllers. If the ball hits any of the vertical boundaries of the matrix then the player loses the game. Each hand gesture controller has an accelerometer sensor interfaced to SJOne board and it transmits data wirelessly to display node. The display node has the LED matrix interfaced to SJOne board and it receives the commands wirelessly from the hand gesture controllers. A wireless mesh network is created between the display node and the hand gesture boards to send accelerometer values. We have designed three screens for the game: Start screen, Game screen, and Result screen. This project helps us understand the communication and synchronization of FreeRTOS tasks in an interesting way.

== Objectives & Introduction ==
=== Objective ===
This project involves learning what goes behind developing a 2D Multiplayer game. Translating the onboard acceleration sensor in a way to be meaningful and measurable for the display node to display the slider movement from the two players over the onboard wireless Nordic transceiver.

===Introduction ===
The overall architecture of the design includes a display node, two hand-gesture control modules, and wireless communication module.

'''Display node:''' It has a 16x32 LED Matrix interfaced to SJOne board. It is responsible for driving the LED matrix

'''Hand-gesture control boards:''' The two hand-gesture control boards are used to control the game. They have SJOne board with an onboard accelerometer. The microcontroller in the SJOne board continuously reads the data from the accelerometer and sends signals wirelessly to the display node

'''Wireless module:''' It is used as a communication medium between the display node and the hand-gesture controlled boards`


[[File:Architecture_Overview.png|800px|thumb|center|Architecture Overview]]

== Team Members & Responsibilities ==
**'''''[https://www.linkedin.com/in/akil-khan/ Akil Khan]'''''
**   Design and development of the RGB Led Matrix driver
**   Game logic development, implementation, QA, and testing.
**'''''[https://www.linkedin.com/in/abhilash-tuse/ Abhilash Tuse]'''''
**   Game logic development
**   Logic Development for communication between hand gesture control board and controller board
**'''''[https://www.linkedin.com/in/dishapatil1494/ Disha Patil]'''''
**   Design of PCB using Eagle for Power Supply
**   Hand gesture control logic development using an accelerometer
**'''''[https://www.linkedin.com/in/omkar-kale/ Omkar Kale]'''''
**   Hand gesture control logic development using an accelerometer
**   Logic Development for communication between hand gesture control board and controller board
**'''''[https://www.linkedin.com/in/vishalrshrivastava/ Vishal Shrivastava]'''''
**   Design and development of the RGB Led Matrix driver
**   Design of PCB using Eagle for Power Supply

== Schedule ==
{| class="wikitable"
|-
! scope="col"| Week#
! scope="col"| Date
! scope="col"| Task 
! scope="col"| Status
! scope="col"| Actual Completion Date
|-
|-
! scope="row"| 1
| 04/10/2018
| 
* Submission of Project Proposals
* Research on project requirements.
* Order components and distribute project modules.
| 
* Completed
* Completed
* Completed
| 
* 03/24/2018
* 04/10/2018
* 04/10/2018
|-
! scope="row"| 2
| 04/17/2018
| 
* Developing logic for hand gesture control using accelerometer sensor.
* Power Supply circuit design and simulation using Multisim
* Write basic LED display driver to blink individual pixels and set of pixels.
* Establishing basic wireless communication between 2 SJOne boards.
|
* Completed
* Completed
* Completed
* Completed
|
* 04/15/2018
* 04/16/2018
* 04/17/2018
* 04/18/2018
|- 
! scope="row"| 3
| 04/24/2018
| 
* Develop advanced API's on top of LED display driver: Draw game objects such as sliders and ball.
* PCB layout design using Eagle and finalizing the schematic.
* Establish communication between the hand-gesture controller and display node using wireless mesh module.
* Project report update on the wiki.
|
* Completed
* Completed
* Completed
* Completed
|
* 05/01/2018
* 04/29/2018
* 04/24/2018
* 05/23/2018
|- 
! scope="row"| 4
| 05/01/2018
| 
* Developing logic for the ball movement and translating hand gesture control into slider movement.
* Soldering components and hardware testing on PCB
* Packaging of hardware board and related components.
* Project report update on the wiki.
|
* Completed
* Completed
* Completed
* Completed
|
* 05/01/2018
* 05/19/2018
* 05/09/2018
* 05/19/2018
|- 
! scope="row"| 5
| 05/08/2018
| 
* Integrate ball movement and slider movement logic into display node algorithm.
* Project report update on the wiki.
|
* Completed
* Completed
|
* 05/05/2018
* 05/19/2018
|- 
! scope="row"| 6
| 04/15/2018
| 
* Final testing and bug fixes.
* Complete wiki report and final demo.
|
* Completed
* Completed
|
* 05/23/2018
* 05/23/2018
|- 
|}

== Parts List & Cost ==
{| class="wikitable"
|-
! scope="col"| Item#
! scope="col"| Part
! scope="col"| Manufacturer
! scope="col"| Quantity
! scope="col"| Cost($)
|-
! scope="row"| 1
| SJ One Board
|[http://www.socialledge.com/sjsu/index.php?title=SJ_One_Board Preet]
| 3
| 80.00
|-
! scope="row"| 2
| Adafruit RGB LED Matrix
|[https://www.adafruit.com/product/420 LED Matrix]
| 1
| 35.00
|-
! scope="row"| 3
| Power Adapter
|[https://www.adafruit.com/product/276 Power Supply]
|1 
|7.95
|-
! scope="row"| 4 
| Power Bank 800mah
|[https://www.amazon.com/dp/B073P83PHF/ref=sspa_dk_detail_0?psc=1&pd_rd_i=B073P83PHF&pd_rd_wg=ubl19&pd_rd_r=DAQV9R6RCAW5N9XK7WE1&pd_rd_w=4WrKg Aibocn]
| 2
| 11.95
|-

! scope="row"| 5
| PCB Board
| [https://www.pcbway.com/ PCBway]
| 1
| 23.00
|-
! scope="row"| 6
| Miscellaneous (Barrel Jack, Jumper wires, Connectors)
| Excess Solution
| 
| 5.00
|-

|}
*Total Cost:334.85$

== Design & Implementation ==

=== Hardware Design ===
[[File:CmpE244_S18_System_Block_Diagram.jpg|800px|thumb|center|System block diagram]]

==== Accelerometer ====
The SJOne board has an accelerometer which is interfaced on the I2C2 bus. Accelerometers are electromechanical devices that sense either static or dynamic forces of acceleration. Static forces include gravity, while dynamic forces can include vibrations and movement. The measurements are on 3-axis and these values can be calibrated to find the desired values.

In our project, an accelerometer is used for detecting the up-down movement of the wrist. We are using accelerometers on two SJ-One Boards for gesture recognition. The movement from accelerometer sensor will be used to control the slider movement on the RGB LED Matrix.
<center>
<table>
<tr>
<td>
[[File:CmpE244_S18_Accelerometer.png|200px|thumb|left|Accelerometer on board]]
</td>
<td>
</td>
<td>
[[File:CmpE244_S18_Detectable_Accelerations.png|200px|thumb|right|Accelerometer Detection]]
</td>
<td>
[[File:CmpE244_S18_Hand_Gesture_Controller.png|150px|thumb|right|Hand-Gesture Controller]]
</td>
</tr>
</table>
</center>

==== Wireless Module====
[[File:CmpE244_S18_Nordic_Wireless.jpg|300px|thumb|SJone wireless remote]]
Nordic wireless chip nRF24L01+ is interfaced with SJone board using SPI bus protocol.

The nRF24L01+ is a single chip 2.4GHz transceiver with an embedded baseband protocol engine, suitable for ultra low
power wireless applications. The nRF24L01+ is designed for operation in the frequency band of 2.400 - 2.483GHz. The
high air data rate combined with two power saving modes make the nRF24L01+ very suitable for ultra-low power designs.  The nRF24L01 integrates a complete 2.4GHz RF transceiver, RF synthesizer, and baseband logic including the Enhanced ShockBurst™ hardware protocol accelerator supporting a high-speed SPI interface for the application controller. No external loop filter, resonators, or VCO varactor diodes are required, only a low-cost ±60ppm crystal, matching circuitry, and antenna.

==== RGB LED Matrix====
[[File:RGB_LED_Matrix.PNG|300px|thumb|16x32 RGB LED Matrix]]

The RGB LED matrix panels are normally used to make video walls to display animations or short video clips. It has 512 bright RGB LEDs arranged in a 16x32 grid. These panels require 12 digital pins (6-bit data, 6-bit control) and a good 5V power supply, at least 2 amps per panel. A single 32x16 RGB matrix, running full tilt (all pixels set white), can require nearly 4 Amps of current.

All 512 LEDs can’t be driven at once. One reason is that would require a lot of current, another reason is that it would be expensive to have so many pins. Instead, the matrix is divided into 8 interleaved sections/strips. The first section is the 1st 'line' and the 9th 'line' (32 x 2 RGB LEDs = 64 RGB LEDs), the second is the 2nd and 10th line, etc until the last section which is the 7th and 16th line. The reason it is done in this way is so that the lines are interleaved and look better when refreshed, otherwise, we'd see the stripes more clearly. So, on the PCB is 12 LED driver chips. These are like 74HC595s but they have 16 outputs and they require constant current. 16 outputs * 12 chips = 192 LEDs that can be controlled at once, and 64 * 3 (R G and B) = 192. There are 192 outputs that can control one line at a time, with each of 192 R, G and B LEDs either on or off. The controller selects which section to draw (using A, B, and C address pins - 3 bits can have 8 values). Once the address is set, the controller clocks out 192 bits of data (24 bytes) and latches it. Then it increments the address and clocks out another 192 bits, etc until it gets to address #7, then it sets the address back to #0

We are driving the matrix using 12 GPIO pins of SJOne board.

=== Hardware Interface Diagram===

[[File:CMPE244_S18_SS_hw_interface.png | 600px | thumb | center | '''Hardware Interface of RGB LED matrix''']]

Pin mapping of RGB LED matrix with SJ One board:
{| class="wikitable"
|-
! align="center"|RGB LED matrix Pins
! align="center"|SJ One Board Pins
! align="center"|Function
|-
| scope="row" align="center"|R1
| scope="row" align="center"|P1.19
| scope="row" align="center"|Red Data top half
|-
| scope="row" align="center"|G1
| scope="row" align="center"|P1.20
| scope="row" align="center"|Green Data top half
|-
| scope="row" align="center"|B1
| scope="row" align="center"|P1.22
| scope="row" align="center"|Blue Data top half
|-
| scope="row" align="center"|R2
| scope="row" align="center"|P1.23
| scope="row" align="center"|Red Data bottom half
|-
| scope="row" align="center"|G2
| scope="row" align="center"|P1.28
| scope="row" align="center"|Green Data bottom half
|-
| scope="row" align="center"|B2
| scope="row" align="center"|P1.29
| scope="row" align="center"|Blue Data bottom half
|-
| scope="row" align="center"|A
| scope="row" align="center"|P2.0
| scope="row" align="center"|Row select A
|-
| scope="row" align="center"|B
| scope="row" align="center"|P2.1
| scope="row" align="center"|Row select B
|-
| scope="row" align="center"|C
| scope="row" align="center"|P2.2
| scope="row" align="center"|Row select C
|-
| scope="row" align="center"|CLK
| scope="row" align="center"|P0.1
| scope="row" align="center"|The CLK (clock) signal marks 
the arrival of each bit of data.
|-
| scope="row" align="center"|OE
| scope="row" align="center"|P0.0
| scope="row" align="center"|OE (output enable) switches the LEDs off 
when transitioning from one row to the next.
|-
| scope="row" align="center"|LAT
| scope="row" align="center"|P0.26
| scope="row" align="center"|The LAT (latch) signal marks the end 
of a row of data.
|-
|}

=== Printed Circuit Board(PCB) Design===

The PCB Layout is designed using the Eagle Software v8.7.1. 
Board consists of 2 power supplies which are powering: 
*SJone board (5V, 500mA)
*RGB LED Matrix (5V,2A)
The Power Supply circuit has an LM317 voltage regulator IC and a voltage divider to fulfill the specific power requirements.
LM317 is a TI (Texas Instruments) based IC which has an Output Voltage Range Adjustable from 1.25 V to 37 V and is suitable for our application.
We have used a 12V adapter in order to power our board. This serves for both the current requirements.
The circuit was simulated using MultiSim v14.1 software by NI (National Instruments). 
The simulation helped us understand the working of our circuit before we built and tested it.

Printed Circuit Board also has provisions for the connection of the SJone pin headers and RGB LED Matrix's connector. 
The connections are as shown in the hardware description diagram. The PCB also has a power switch and LED for indication.
[[File:CmpeE244_S18_PCB.jpg|300px|thumb|left|Printed Circuit Board]]
[[File:CmpeE244_S18_Eagle_board_design.png|300px|thumb|center|Board Design]]

=== Software Design ===
==== Display Node ====
===== RGB LED Matrix =====
This 16x32 5V,2A supply RGB LED Matrix panels from Adafruit require 12 digital pins (6-bit data, 6-bit control). They have 512 bright RGB LEDs arranged in a 16x32 grid on the front. These displays are 'chainable' - connect the output to the next input. This display requires about 800 bytes of RAM to buffer the 12-bit color image. As the matrix panel has a 6-bit parallel interface, SPI hardware won't help and hence simple GPIO interface has been used to communicate with the RGB LED Matrix. As these displays don't have built-in PWM control of any kind, the driver is supposed to redraw the screen over and over to 'manually' PWM the whole thing. 

Pins R1, G1, and B1 deliver data to the top half of the display whereas R2, G2, and B2 deliver data to the bottom half of the display. Pins A, B, C, and D select which two rows of the display are currently lit. The LAT signal acts as the strobe and marks the end of a row of data. The CLK signal marks the arrival of each bit of data. OE (output enable) is an active low signal that switches the LEDs off when transitioning from one row to the next.

The implementation consists of 768-byte buffer for storing the pixel values for all LEDs on the display. Every line is scanned and the values for the pixels are updated every 1 ms. Between every transition of the clock from low to high, the values for the pixels are updated. As the screen is refreshed and updated very frequently, it gives the impression of a moving image.

The basic process used to refresh the display when using three bits-per-pixel color (one bit for red; one bit for green; and one bit for blue) is the following:

*Shift the pixel data for row 0 into the top column drivers and the pixel data for row 16 into the bottom column drivers using the R0, G0, B0, R1, G1, and B1 data inputs and the SCLK shift clock signal.
*Assert the blanking signal to blank the display.
*Set the address input to 0.
*Latch the contents of the column drivers‘ shift registers into the column drivers‘ output registers using the LATCH signal.
*Deassert the blanking signal to display rows 0 and 16.
*Wait some fixed amount of time.
*Repeat the process for each of the pairs of rows in the display.
*Repeat the entire process at least 100 to 200 times per second to prevent flicker.

Supported APIs:

*DrawPixel(uint8_t x, uint8_t y, uint16_t c);
*DrawLine(int8_t x0, int8_t y0, int8_t x1, int8_t y1, uint16_t c);
*DrawRectangle(uint8_t x, uint8_t y, uint8_t w, uint8_t h, uint16_t color);
*FillRectangle(uint8_t x, uint8_t y, uint8_t w, uint8_t h, uint16_t color);
*DrawChar(uint8_t x, uint8_t y, char c, uint16_t color, uint8_t size);
*UpdateDisplay();
*DrawCircle(uint8_t x0, uint8_t y0, uint8_t r, uint16_t color);
*Fill(uint16_t c);
*Fill_slider(uint16_t c);
*Fill_ball(uint16_t c);
*SetCursor(uint8_t x, uint8_t y);


[[File:class_diagram_display_node.png|700px|thumb|center|Display Node Class Diagram]]

===== Wireless Receiver =====
The SJ-One board connected to LED display uses onboard Nordic wireless to communicate with player boards. Initially, it waits for players to press the start button on their controllers which sends commands to the receiver board. Once it receives the commands to start, it displays green signal on the display and changes the screen to game screen after that. When the game is running, receiver continuously receives the acceleration values from player boards which it then sends further to slider task for movement on the display. We referred '''[http://socialledge.com/sjsu/index.php/Low_Powered_Mesh_Network_stack Low Powered Mesh Network stack]''' for building wireless mesh network between the display board and player boards.

==== Hand Gesture Control Logic ====
===== Flowchart =====
[[File:CmpE244_S18_AccSensor_flowchart.PNG|500px|thumb|center|Flowchart for Hand Gesture Module]]
===== Accelerometer Value Calculation =====
After carefully looking at the board design and how the board will be placed on a person's hand, we found out that the values received along one axis(X-axis) will be enough to capture the wrist movement.
The range of accelerometer values shows how sensitive the accelerometer sensor is. We wanted the code to be flexible in terms of the range of the values. So that we did not have to hard code the sensor values and instead used a code based on maximum and minimum range values which could be easily configurable.
The piece of code mentioned below shows how the values are calculated.
[[File:CmpE244_S18_AccValue.PNG|500px|thumb|center|Accelerometer Value Calculations]]


[[File:player1_class_diagram.png|500px|thumb|left|Player 1 Class Diagram]]
[[File:player2_class_diagram.png|500px|thumb|center|Player 2 Class Diagram]]



===== Wireless Transmission =====
We used Nordic Wireless for transmitting the accelerometer data.
Each node has an unique node address and a node name. For the mesh network, we had to use the same channel number and data transmission rate.
The below figure shows the node addresses allocated to the two player nodes and the control node.
[[File:CmpE244_S18_Wireless_Node_Addresses.PNG|200px|thumb|center|Flowchart for Hand Gesture Module]]

=== Game logic and Implementation ===

The game implementation involves creating multiple tasks for control of independent entities and also making those entities work together.

[[File:Game_Flow_Chart.PNG|900px|thumb|center|Game Flowchart]]

The game starts off with the start screen task, which displays the game title and status for both players. As soon as both the players have pressed the button1 (to indicate that the player is ready), after a 3 second counter, the ball and slider tasks are resumed. Initially, all the tasks are suspended except the start screen task. Slider task takes the input acceleration sensor values from both the players and translates the values into appropriate movement on the display whereas the ball tasks move the ball on the display. If any of the player miss to hit the ball, the player loses and the score display task will be displayed and all other tasks would be suspended. After displaying the score for 3 seconds, we reset the ball position to randomly start from the center of the screen (4 possible directions) and start the round again. A player wins if he wins 9 rounds, and the score display screen then displays the winner along with the scores. As the game is over, the game will be restarted by suspending all tasks and resuming the start screen task again.

Remote wireless task retrieves the slider values from both the players over Nordic wireless transceiver and passes it to the slider task for producing the hand gesture movement on to the display. To support the wireless communication, background Nordic wireless task participates in the mesh network and handles retry logic such that packets are resent if an ACK has not been received.

== Testing ==
=== Module testing ===
==== Acceleration Measurement ====
Hand gesture recognition was rigorously tested. The values received from accelerometer sensor were within a range. So, we had to test the range of values received at which the slider on screen looked responsive and smooth. We tested values ranging from -1000 to +1000 to find the right range of values. The hand gesture module transmits uint8_t. So, we checked whether the output at the minimum and maximum input values were within valid range. We also checked the uint8_t value(index) transmitted depending upon the inputs received from the accelerometer sensor.

==== LCD Display ====
At first, we implemented the slider movement logic and tested the slider movement on the display. This testing helped us decide the size of the slider and make sure that when the slider is moved using the accelerometer, the change in the position is perfectly reflected on the LED Matrix

We wanted to make sure, whenever the ball touches the horizontal boundaries of the LED matrix it bounces back with the proper angle. And if the ball touches the vertical boundaries of the matrix, the matrix should display the score and start the game again. To ensure the correctness of the ball movement, we implemented the ball movement logic and tested it independently on the LED Matrix.
[[File:CmpeE244_S18_Slider_Movement.gif|150px|thumb|left|Slider Movement Testing]]
[[File:CmpeE244_S18_Ball_Movement.gif|200px|thumb|center|Ball Movement Testing]]





Also, we designed the Start screen and Game End screen and tested them on the LED display. While testing the screens we focused on having proper text font and placement of the text on the display.
[[File:CmpE244_S18_LED_Matrix_Start_Screen.png|200px|thumb|left|Start Screen]]
[[File:CmpE244_S18_LED_Matrix_Game_End_Screen.png|200px|thumb|center|Game End Screeen]]



==== PCB Testing ====

The PCB was tested by checking the connectivity of the path on the copper clad tracks using a Digital Multimeter.
The circuit on the PCB was simulated using MultiSim v14.1 Software and results were verified on the actual board.
All the connector-related bugs were fixed by inspection and drilling the PCB wherever necessary. 
The board was tested for any short circuit by checking the connectivity between the Supply tracks and common ground tracks.

=== Integration testing ===
After testing the individual modules, we decided to integrate display node and left-hand control board first and test it. Once we got the desired result, we integrated the right-hand board with the display node. This methodology helped us identify the wireless communication challenges. 
After testing the display nodes with the hand control boards, we decided to integrate the ball movement logic with. This approach helped us in deciding the suitable speed of the ball and displaying the score if any player loses. Also, the integration testing helped us identify some major issues such as flickering of the LED matrix.

== Technical Challenges ==
=== Display driver implementation ===

We initially started to mimic the Adafruit library, but the library was originally designed to work with Arduino (AVR), and it consisted of some assembly language instructions to save CPU cycle. Hence, we were unsuccessful in porting the library entirely. We read more about the RGB LED matrix and decided to get control over a single row first and over a single pixel. After some trial and error bit-banging, we were successful in gaining control over a single row and a single pixel on the Matrix display. Once that was achieved, we moved on to implementing higher level APIs on top of it like DrawLine, DrawRectangle, etc. Thought was also given on interfacing the display over SPI, but the display needs 6 parallel data outputs at a time, so driving through SPI wouldn't have been the optimized solution for this problem.

=== RGB LED Matrix Flickering Issue ===

We need to refresh the LED matrix to display the new data. Earlier we were refreshing the matrix after every 100 milliseconds. Because the refresh rate was low we saw there was some flickering in the matrix. So, we decided to increase the refresh rate and kept it at 1 milliseconds. This refresh rate is very small for the human eyes to detect, the flickering in the matrix stopped appearing.

=== PCB Designing ===

The toughest challenge in designing the PCB Layout and Schematic was to select the exact Pin Header Libraries of the accurate measurement of a 3.5mm spacing.
This problem especially occurred for the power connectors the project needed for powering up the RGB LED matrix as the connector is slightly unconventional compared to the normal 2mm spacing.
To suffice this need, a new library was created in Eagle v8.2.1 Software by taking a reference of an existing library which has the same spacing between the pins on the header.

== Conclusion ==
The project was a success both in terms of output and learnings. We were able to play a multiplayer ping-pong game by wirelessly controlling the display on the LED Matrix by using hand gesture. This project helped us learn the working of the following modules in particular: RGB LED matrix, Nordic wireless unit, accelerometer on the SJ-One board. We could identify synchronization issues and tried to fix them by unit testing. The hardware which included a PCB and LED matrix had to be designed and packaged to give an overall compact look to the project so that it looks more like a gaming console product.
We enjoyed doing this project and getting better at using FreeRTOS APIs, debugging and hardware designing.

=== Project Video ===
Game Video - [https://www.youtube.com/watch?v=rA3D4OwXwDc&feature=youtu.be Hand Gesture Controlled Ping Pong Game]

=== Project Source Code ===
*  [https://gitlab.com/suicide_squad/hand-gesture-controlled-multiplayer-game Github Source Code Link]

== References ==
=== Acknowledgement ===
We would like to thank our Professor Preetpal Kang for all his teachings and inspirational lectures. Not only did we enjoy working though out this project but also gave us an overall learning experience and precious life lessons. We would also like to thank the ISA members for always being ready to help with whatever issues we faced.

=== References Used ===
[1] [http://www.socialledge.com/sjsu/index.php?title=Realtime_OS_on_Embedded_Systems FreeRTOS documentations]

[2] [https://github.com/adafruit/Adafruit-GFX-Library Adafruit LCD library]

[3] CMPE 244 Lecture notes from Preetpal Kang, Computer Engineering, San Jose State University. Feb - May 2018.

[4] [https://www.sparkfun.com/datasheets/Components/SMD/nRF24L01Pluss_Preliminary_Product_Specification_v1_0.pdf Nordic wireless datasheet]