Difference between revisions of "S18: Hand gesture controlled multiplayer game"

From Embedded Systems Learning Academy
Jump to: navigation, search
(Hand Gesture Control Logic)
(Hand-gesture control module)
 
(163 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
[[File:CmpE244_Viserion_ProjectImage.PNG|thumb|589x615px|right|Ping Pong]]
 +
 
== Project Title ==
 
== Project Title ==
===Hand gesture controlled multiplayer game===
+
===Hand-gesture controlled multiplayer game===
  
 
== Abstract ==
 
== Abstract ==
Human-machine interfaces are constantly evolving. The interfaces used in HMI based on natural gestures of users allow converting these movements into commands for a computer system. This project involves the implementation of a wireless hand-gesture controlled multiplayer Ping Pong (2D) game using the 16x32 RGB LED Matrix, accelerometer, wireless connectivity and FreeRTOS APIs. It incorporates the onboard RF Nordic wireless transceiver of SJ-One board for wireless communication and the onboard acceleration sensor to translate the hand moves into the slider movement on the display. We have designed this game using FreeRTOS APIs. The game works as a classic ping pong game in which there are two sliders which are controlled wirelessly by hand gesture controller. If the ball hits any of the vertical boundaries of the screen then the player loses the game. We have designed four screens: Start screen, Game screen, Score screen and Result screen. A wireless mesh network is created between display node and player boards to send accelerometer values. This project helps us understand the communication and synchronization of FreeRTOS tasks in an interesting way.
+
Human-machine interfaces are constantly evolving. The interfaces used in HMI, based on natural gestures of users, allow converting these movements into commands for a computer system. In this project, we developed a multiplayer Ping-Pong game using a 16x32 RGB LED Matrix, accelerometer, wireless connectivity, FreeRTOS APIs and SJOne boards. The game works as a classic ping pong game in which there are two sliders which are controlled wirelessly by two hand-gesture controllers. If the ball hits any of the vertical boundaries of the matrix then the player loses the game. Each hand-gesture controller has an accelerometer sensor interfaced to SJOne board and it transmits data wirelessly to display node. The display node has the LED matrix interfaced to SJOne board and it receives the commands wirelessly from the hand-gesture controllers. A wireless mesh network is created between the display node and the hand-gesture boards to send accelerometer values. We have designed three screens for the game: Start screen, Game screen, and Result screen. This project helps us understand the communication and synchronization of FreeRTOS tasks in an interesting way.
  
 
== Objectives & Introduction ==
 
== Objectives & Introduction ==
 +
[[File:Architecture_Overview.png|550px|thumb|right|Architecture Overview]]
 
=== Objective ===
 
=== Objective ===
 
This project involves learning what goes behind developing a 2D Multiplayer game. Translating the onboard acceleration sensor in a way to be meaningful and measurable for the display node to display the slider movement from the two players over the onboard wireless Nordic transceiver.
 
This project involves learning what goes behind developing a 2D Multiplayer game. Translating the onboard acceleration sensor in a way to be meaningful and measurable for the display node to display the slider movement from the two players over the onboard wireless Nordic transceiver.
  
 
===Introduction ===
 
===Introduction ===
 +
 
The overall architecture of the design includes a display node, two hand-gesture control modules, and wireless communication module.
 
The overall architecture of the design includes a display node, two hand-gesture control modules, and wireless communication module.
  
'''Display node:''' It has a 16x32 LED Matrix interfaced to SJOne board. It is responsible for driving the LED matrix
+
'''Display Node:''' It has a 16x32 LED Matrix interfaced to SJOne board. It is responsible for driving the LED matrix
  
'''Hand-gesture control boards:''' The two hand-gesture control boards are used to control the game. They have SJOne board with an onboard accelerometer. The microcontroller in the SJOne board continuously reads the data from the accelerometer and sends signals wirelessly to the display node
+
'''Hand-gesture control module:''' The two hand-gesture control boards are used to control the game. They have SJOne board with an onboard accelerometer. The microcontroller in the SJOne board continuously reads the data from the accelerometer and sends signals wirelessly to the display node
  
 
'''Wireless module:''' It is used as a communication medium between the display node and the hand-gesture controlled boards`
 
'''Wireless module:''' It is used as a communication medium between the display node and the hand-gesture controlled boards`
 
 
[[File:Architecture_Overview.png|800px|thumb|center|Architecture Overview]]
 
  
 
== Team Members & Responsibilities ==
 
== Team Members & Responsibilities ==
**'''''[https://www.linkedin.com/in/akil-khan/ Akil Khan]'''''
+
*'''''[https://www.linkedin.com/in/akil-khan/ Akil Khan]'''''
 
**  Design and development of the RGB Led Matrix driver
 
**  Design and development of the RGB Led Matrix driver
 
**  Game logic development, implementation, QA, and testing.
 
**  Game logic development, implementation, QA, and testing.
**'''''[https://www.linkedin.com/in/abhilash-tuse/ Abhilash Tuse]'''''
+
*'''''[https://www.linkedin.com/in/abhilash-tuse/ Abhilash Tuse]'''''
 
**  Game logic development
 
**  Game logic development
**  Logic Development for communication between hand gesture control board and controller board
+
**  Logic Development for communication between hand-gesture control board and controller board
**'''''[https://www.linkedin.com/in/dishapatil1494/ Disha Patil]'''''
+
*'''''[https://www.linkedin.com/in/dishapatil1494/ Disha Patil]'''''
 +
**  Hand-gesture control logic development using an accelerometer
 
**  Design of PCB using Eagle for Power Supply
 
**  Design of PCB using Eagle for Power Supply
**  Hand gesture control logic development using an accelerometer
+
*'''''[https://www.linkedin.com/in/omkar-kale/ Omkar Kale]'''''
**'''''[https://www.linkedin.com/in/omkar-kale/ Omkar Kale]'''''
+
**  Hand-gesture control logic development using an accelerometer
**  Hand gesture control logic development using an accelerometer
+
**  Logic Development for communication between hand-gesture control board and controller board
**  Logic Development for communication between hand gesture control board and controller board
+
*'''''[https://www.linkedin.com/in/vishalrshrivastava/ Vishal Shrivastava]'''''
**'''''[https://www.linkedin.com/in/vishalrshrivastava/ Vishal Shrivastava]'''''
 
 
**  Design and development of the RGB Led Matrix driver
 
**  Design and development of the RGB Led Matrix driver
 
**  Design of PCB using Eagle for Power Supply
 
**  Design of PCB using Eagle for Power Supply
Line 66: Line 67:
 
| 04/17/2018
 
| 04/17/2018
 
|  
 
|  
* Developing logic for hand gesture control using accelerometer sensor.
+
* Developing logic for hand-gesture control using accelerometer sensor.
 
* Power Supply circuit design and simulation using Multisim
 
* Power Supply circuit design and simulation using Multisim
 
* Write basic LED display driver to blink individual pixels and set of pixels.
 
* Write basic LED display driver to blink individual pixels and set of pixels.
Line 102: Line 103:
 
| 05/01/2018
 
| 05/01/2018
 
|  
 
|  
* Developing logic for the ball movement and translating hand gesture control into slider movement.
+
* Developing logic for the ball movement and translating hand-gesture control into slider movement.
 
* Soldering components and hardware testing on PCB
 
* Soldering components and hardware testing on PCB
 
* Packaging of hardware board and related components.
 
* Packaging of hardware board and related components.
Line 196: Line 197:
  
 
=== Hardware Design ===
 
=== Hardware Design ===
[[File:CmpE244_S18_System_Block_Diagram.jpg|800px|thumb|center|System block diagram]]
 
  
==== Accelerometer ====
+
[[File:CmpE244_S18_System_Block_Diagram.jpg|570px|thumb|center|'''System block diagram''']]
The SJOne board has an accelerometer which is interfaced on the I2C2 bus. Accelerometers are electromechanical devices that sense either static or dynamic forces of acceleration. Static forces include gravity, while dynamic forces can include vibrations and movement. The measurements are on 3-axis and these values can be calibrated to find the desired values.
+
 
 +
 
  
In our project, an accelerometer is used for detecting the up-down movement of the wrist. We are using accelerometers on two SJ-One Boards for gesture recognition. The movement from accelerometer sensor will be used to control the slider movement on the RGB LED Matrix.
+
Hardware design diagram gives an overview of the entire system which consists of the main 3 SJ-One controllers: 2 player boards and 1 board as the display node
<center>
+
 
<table>
+
*Player boards use the onboard accelerometer on SJ-One which is interfaced using the I2C protocol. Each of the player boards read these accelerometer values and determine the position of the slider on the display.
<tr>
+
*As we are implementing a wireless multiplayer game, the accelerometer values are transmitted using yet another onboard module, the Nordic wireless module which interfaces with the wireless module on the third controller used for the display.
<td>
+
*The Display node SJ-One board is used to control a 16*32 RGB LED Matrix. This matrix displays 2 sliders and a ball representing the players in a Ping-Pong game. Slider locations are decided by the values obtained wirelessly from Player boards.
[[File:CmpE244_S18_Accelerometer.png|200px|thumb|left|Accelerometer on board]]
+
*RGB LED Matrix displays three screens: Start Screen, Game Screen, End Screen which displays the final score of the players and the winner as well. The hardware description of the LED matrix is explained later in this section.  
</td>
 
<td>
 
</td>
 
<td>
 
[[File:CmpE244_S18_Detectable_Accelerations.png|200px|thumb|right|Accelerometer Detection]]
 
</td>
 
<td>
 
[[File:CmpE244_S18_Hand_Gesture_Controller.png|150px|thumb|right|Hand-Gesture Controller]]
 
</td>
 
</tr>
 
</table>
 
</center>
 
  
==== Wireless Module====
 
[[File:CmpE244_S18_Nordic_Wireless.jpg|300px|thumb|SJone wireless remote]]
 
Nordic wireless chip nRF24L01+ is interfaced with SJone board using SPI bus protocol.
 
  
The nRF24L01+ is a single chip 2.4GHz transceiver with an embedded baseband protocol engine, suitable for ultra low
+
==== Display Node ====
power wireless applications. The nRF24L01+ is designed for operation in the frequency band of 2.400 - 2.483GHz. The
 
high air data rate combined with two power saving modes make the nRF24L01+ very suitable for ultra-low power designs.  The nRF24L01 integrates a complete 2.4GHz RF transceiver, RF synthesizer, and baseband logic including the Enhanced ShockBurst™ hardware protocol accelerator supporting a high-speed SPI interface for the application controller. No external loop filter, resonators, or VCO varactor diodes are required, only a low-cost ±60ppm crystal, matching circuitry, and antenna.
 
  
==== RGB LED Matrix====
+
===== RGB LED Matrix =====
 
[[File:RGB_LED_Matrix.PNG|300px|thumb|16x32 RGB LED Matrix]]
 
[[File:RGB_LED_Matrix.PNG|300px|thumb|16x32 RGB LED Matrix]]
  
 
The RGB LED matrix panels are normally used to make video walls to display animations or short video clips. It has 512 bright RGB LEDs arranged in a 16x32 grid. These panels require 12 digital pins (6-bit data, 6-bit control) and a good 5V power supply, at least 2 amps per panel. A single 32x16 RGB matrix, running full tilt (all pixels set white), can require nearly 4 Amps of current.
 
The RGB LED matrix panels are normally used to make video walls to display animations or short video clips. It has 512 bright RGB LEDs arranged in a 16x32 grid. These panels require 12 digital pins (6-bit data, 6-bit control) and a good 5V power supply, at least 2 amps per panel. A single 32x16 RGB matrix, running full tilt (all pixels set white), can require nearly 4 Amps of current.
  
All 512 LEDs can’t be driven at once. One reason is that would require a lot of current, another reason is that it would be expensive to have so many pins. Instead, the matrix is divided into 8 interleaved sections/strips. The first section is the 1st 'line' and the 9th 'line' (32 x 2 RGB LEDs = 64 RGB LEDs), the second is the 2nd and 10th line, etc until the last section which is the 7th and 16th line. The reason it is done in this way is so that the lines are interleaved and look better when refreshed, otherwise, we'd see the stripes more clearly. So, on the PCB is 12 LED driver chips. These are like 74HC595s but they have 16 outputs and they require constant current. 16 outputs * 12 chips = 192 LEDs that can be controlled at once, and 64 * 3 (R G and B) = 192. There are 192 outputs that can control one line at a time, with each of 192 R, G and B LEDs either on or off. The controller selects which section to draw (using A, B, and C address pins - 3 bits can have 8 values). Once the address is set, the controller clocks out 192 bits of data (24 bytes) and latches it. Then it increments the address and clocks out another 192 bits, etc until it gets to address #7, then it sets the address back to #0
+
All 512 LEDs can’t be driven at once. One reason is that would require a lot of current, another reason is that it would be expensive to have so many pins. Instead, the matrix is divided into 8 interleaved sections/strips. The first section is the 1st 'line' and the 9th 'line' (32 x 2 RGB LEDs = 64 RGB LEDs), the second is the 2nd and 10th line, etc until the last section which is the 7th and 16th line. The reason it is done in this way is so that the lines are interleaved and look better when refreshed, otherwise, we'd see the stripes more clearly. So, on the PCB is 12 LED driver chips. These are like 74HC595s but they have 16 outputs and they require constant current. 16 outputs * 12 chips = 192 LEDs that can be controlled at once, and 64 * 3 (R G and B) = 192. There are 192 outputs that can control one line at a time, with each of 192 R, G and B LEDs either on or off. The controller selects which section to draw (using A, B, and C address pins - 3 bits can have 8 values). Once the address is set, the controller clocks out 192 bits of data (24 bytes) and latches it. Then it increments the address and clocks out another 192 bits, etc until it gets to address #7, then it sets the address back to #0.
  
 
We are driving the matrix using 12 GPIO pins of SJOne board.
 
We are driving the matrix using 12 GPIO pins of SJOne board.
 
=== Hardware Interface Diagram===
 
 
[[File:CMPE244_S18_SS_hw_interface.png | 600px | thumb | center | '''Hardware Interface of RGB LED matrix''']]
 
  
 
Pin mapping of RGB LED matrix with SJ One board:
 
Pin mapping of RGB LED matrix with SJ One board:
 +
[[File:Hw_interface.png | 750px | thumb | right | '''Hardware Interface of RGB LED matrix''']]
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
Line 291: Line 272:
 
| scope="row" align="center"|OE
 
| scope="row" align="center"|OE
 
| scope="row" align="center"|P0.0
 
| scope="row" align="center"|P0.0
| scope="row" align="center"|OE (output enable) switches the LEDs off  
+
| scope="row" align="center"|OE (output enable) switches the
when transitioning from one row to the next.
+
LEDs off when transitioning
 +
from one row to the next.
 
|-
 
|-
 
| scope="row" align="center"|LAT
 
| scope="row" align="center"|LAT
 
| scope="row" align="center"|P0.26
 
| scope="row" align="center"|P0.26
| scope="row" align="center"|The LAT (latch) signal marks the end  
+
| scope="row" align="center"|The LAT (latch) signal marks the
of a row of data.
+
end of a row of data.
 
|-
 
|-
 
|}
 
|}
  
=== Printed Circuit Board(PCB) Design===
+
 
 +
==== Hand-gesture control module ====
 +
[[File:CmpE244_S18_Hand_Gesture_Controller.png|400px|thumb|center|Hand-Gesture Controller]]
 +
===== Accelerometer =====
 +
The SJ-One board has an accelerometer which is interfaced on the I2C bus. Accelerometers are electromechanical devices that sense either static or dynamic forces of acceleration. Static forces include gravity, while dynamic forces can include vibrations and movement. The measurements are on 3-axis and these values can be calibrated to find the desired values.
 +
 
 +
In our project, an accelerometer is used for detecting the up-down movement of the wrist. We are using accelerometers on two SJ-One Boards for gesture recognition. The movement from accelerometer sensor will be used to control the slider movement on the RGB LED Matrix.
 +
{|
 +
|[[File:CMPE244_S18_SS_Accelerometer.PNG|200px|thumb|left|Accelerometer on board]]
 +
|[[File:Accelerometer_Interface.PNG|400px|thumb|center|Accelerometer Intefaced With Microcontroller]]
 +
|[[File:CmpE244_S18_Detectable_Accelerations.png|400px|thumb|right|Accelerometer Detection]]
 +
|}
 +
 
 +
==== Wireless Module====
 +
{|
 +
|Nordic wireless chip nRF24L01+ is interfaced with SJone board using SPI bus protocol.
 +
The nRF24L01+ is a single chip 2.4GHz transceiver with an embedded baseband protocol engine, suitable for ultra low
 +
power wireless applications. The nRF24L01+ is designed for operation in the frequency band of 2.400 - 2.483GHz. The
 +
high air data rate combined with two power saving modes make the nRF24L01+ very suitable for ultra-low power designs.  The nRF24L01 integrates a complete 2.4GHz RF transceiver, RF synthesizer, and baseband logic including the Enhanced ShockBurst™ hardware protocol accelerator supporting a high-speed SPI interface for the application controller. No external loop filter, resonators, or VCO varactor diodes are required, only a low-cost ±60ppm crystal, matching circuitry, and antenna.
 +
|[[File:CmpE244_S18_Nordic_Wireless.jpg|350px|thumb|right|'''Nordic Wireless Chip nRF24L01+''' ]]
 +
|[[File:Wireless_Chip_Interface.PNG|350px|thumb|center|'''Nordic Wireless Chip Interface with Microcontroller''']]
 +
|}
 +
 
 +
==== Printed Circuit Board(PCB) Design ====
  
 
The PCB Layout is designed using the Eagle Software v8.7.1.  
 
The PCB Layout is designed using the Eagle Software v8.7.1.  
Line 315: Line 320:
 
Printed Circuit Board also has provisions for the connection of the SJone pin headers and RGB LED Matrix's connector.  
 
Printed Circuit Board also has provisions for the connection of the SJone pin headers and RGB LED Matrix's connector.  
 
The connections are as shown in the hardware description diagram. The PCB also has a power switch and LED for indication.
 
The connections are as shown in the hardware description diagram. The PCB also has a power switch and LED for indication.
[[File:CmpeE244_S18_PCB.jpg|300px|thumb|left|Printed Circuit Board]]
+
The values for the resistors and other active or passive components are as shown in the eagle schematic diagram.
[[File:CmpeE244_S18_Eagle_board_design.png|300px|thumb|center|Board Design]]
+
{|
 +
|[[File:CmpeE244_S18_PCB.jpeg|500px|thumb|center|Printed Circuit Board]]
 +
|[[File:CmpeE244_S18_Eagle_board_design.png|500px|thumb|center|Board Design]]
 +
|}
 +
 
  
 
=== Software Design ===
 
=== Software Design ===
 +
 
==== Display Node ====
 
==== Display Node ====
 +
 
===== RGB LED Matrix =====
 
===== RGB LED Matrix =====
 +
 +
 +
[[File:CMPE244_S18_LedMatrixFlowchart.png | 600px | thumb | left| '''Software Flowchart for the LED Matrix Display''']]
 +
[[File:class_diagram_display_node.png|450px|thumb|center|'''Display Node Class Diagram''']]
 +
 +
 
This 16x32 5V,2A supply RGB LED Matrix panels from Adafruit require 12 digital pins (6-bit data, 6-bit control). They have 512 bright RGB LEDs arranged in a 16x32 grid on the front. These displays are 'chainable' - connect the output to the next input. This display requires about 800 bytes of RAM to buffer the 12-bit color image. As the matrix panel has a 6-bit parallel interface, SPI hardware won't help and hence simple GPIO interface has been used to communicate with the RGB LED Matrix. As these displays don't have built-in PWM control of any kind, the driver is supposed to redraw the screen over and over to 'manually' PWM the whole thing.  
 
This 16x32 5V,2A supply RGB LED Matrix panels from Adafruit require 12 digital pins (6-bit data, 6-bit control). They have 512 bright RGB LEDs arranged in a 16x32 grid on the front. These displays are 'chainable' - connect the output to the next input. This display requires about 800 bytes of RAM to buffer the 12-bit color image. As the matrix panel has a 6-bit parallel interface, SPI hardware won't help and hence simple GPIO interface has been used to communicate with the RGB LED Matrix. As these displays don't have built-in PWM control of any kind, the driver is supposed to redraw the screen over and over to 'manually' PWM the whole thing.  
  
Line 338: Line 355:
 
*Repeat the entire process at least 100 to 200 times per second to prevent flicker.
 
*Repeat the entire process at least 100 to 200 times per second to prevent flicker.
  
Supported APIs:
+
==== Hand-gesture Control Logic ====
  
*DrawPixel(uint8_t x, uint8_t y, uint16_t c);
 
*DrawLine(int8_t x0, int8_t y0, int8_t x1, int8_t y1, uint16_t c);
 
*DrawRectangle(uint8_t x, uint8_t y, uint8_t w, uint8_t h, uint16_t color);
 
*FillRectangle(uint8_t x, uint8_t y, uint8_t w, uint8_t h, uint16_t color);
 
*DrawChar(uint8_t x, uint8_t y, char c, uint16_t color, uint8_t size);
 
*UpdateDisplay();
 
*DrawCircle(uint8_t x0, uint8_t y0, uint8_t r, uint16_t color);
 
*Fill(uint16_t c);
 
*Fill_slider(uint16_t c);
 
*Fill_ball(uint16_t c);
 
*SetCursor(uint8_t x, uint8_t y);
 
  
  
[[File:class_diagram_display_node.png|700px|thumb|center|Display Node Class Diagram]]
 
  
==== Hand Gesture Control Logic ====
+
 
===== Flowchart =====
+
 
[[File:CmpE244_S18_AccSensor_flowchart.PNG|500px|thumb|center|Flowchart for Hand Gesture Module]]
+
[[File:player1_class_diagram.png|325px|thumb|right|'''Hand-gesture Control Board Class Diagram''']]
 +
 
 +
 
 +
[[File:CmpE244_S18_AccSensor_flowchart.PNG|900px|thumb|center|'''Flowchart for Hand-gesture Module''']]
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
[[File:CmpE244_S18_AccValue.PNG|500px|thumb|right|'''Accelerometer Value Calculations''']]
 +
 
 +
 
 +
The gesture controlled process which acts as an interface between the players and the display node is explained as follows:
 +
*Hand gesture module begins with the initialization of the onboard accelerometer sensor which is interfaced using I2C bus to the controller.
 +
*The Start condition is satisfied as soon as both the players press the corresponding switches on the respective boards as a sign of being ready.
 +
*Switch press is acknowledged and the required start code for the start screen is transmitted. The start screen displays a countdown from 3 to 1 to give a dramatic effect.
 +
*After the start screen is displayed, the values are received from the onboard accelerometer sensor and transmitted to the Master Node i.e. the RGB LED Matrix.
 +
 
 
===== Accelerometer Value Calculation =====
 
===== Accelerometer Value Calculation =====
 +
 
After carefully looking at the board design and how the board will be placed on a person's hand, we found out that the values received along one axis(X-axis) will be enough to capture the wrist movement.
 
After carefully looking at the board design and how the board will be placed on a person's hand, we found out that the values received along one axis(X-axis) will be enough to capture the wrist movement.
 
The range of accelerometer values shows how sensitive the accelerometer sensor is. We wanted the code to be flexible in terms of the range of the values. So that we did not have to hard code the sensor values and instead used a code based on maximum and minimum range values which could be easily configurable.
 
The range of accelerometer values shows how sensitive the accelerometer sensor is. We wanted the code to be flexible in terms of the range of the values. So that we did not have to hard code the sensor values and instead used a code based on maximum and minimum range values which could be easily configurable.
 
The piece of code mentioned below shows how the values are calculated.
 
The piece of code mentioned below shows how the values are calculated.
[[File:CmpE244_S18_AccValue.PNG|500px|thumb|center|Accelerometer Value Calculations]]
 
  
 +
==== Wireless Module ====
  
[[File:player1_class_diagram.png|500px|thumb|left|Player 1 Class Diagram]]
+
[[File:Wireless_Interface.PNG|400px|thumb|center|Wireless Communication Between the Boards]]
[[File:player2_class_diagram.png|500px|thumb|center|Player 2 Class Diagram]]
+
 
==== Wireless Transmission ====
+
We referred '''[http://socialledge.com/sjsu/index.php/Low_Powered_Mesh_Network_stack Low Powered Mesh Network stack]''' for building wireless mesh network between the display board and player boards.
We used Nordic Wireless for transmitting the accelerometer data.
+
 
Each node has an unique node address and a node name. For the mesh network, we had to use the same channel number and data transmission rate.
+
===== Wireless Receiver =====
The below figure shows the node addresses allocated to the two player nodes and the control node.
+
 
[[File:CmpE244_S18_Wireless_Node_Addresses.PNG|200px|thumb|center|Flowchart for Hand Gesture Module]]
+
The SJ-One board connected to LED display uses onboard Nordic wireless to communicate with player boards. Initially, it waits for players to press the start button on their controllers which then sends commands to the receiver board. Once it receives the commands to start, it displays green signal on the display and changes the screen to game screen. While the game is running, receiver continuously receives the acceleration values at every 50 ms from player boards which it then sends further to the slider task.
 +
 
 +
===== Wireless Transmission =====
 +
 
 +
We used Nordic Wireless for transmitting the accelerometer data. Both the player 1 and the player 2 boards send the accelerometer data to the Display node. We created a remote wireless task which runs on medium priority and sends an integer value to the Display node at every 100 ms.
 +
Both the boards have a unique node address and a node name. For the mesh network, we had to use the same channel number and data transmission rate for both the player boards and Display node.
  
 
=== Game logic and Implementation ===
 
=== Game logic and Implementation ===
 +
 +
 +
 +
 +
 +
 +
[[File:Game_Flow_Chart.PNG|450px|thumb|center|'''Game Flowchart''']]
 +
  
 
The game implementation involves creating multiple tasks for control of independent entities and also making those entities work together.
 
The game implementation involves creating multiple tasks for control of independent entities and also making those entities work together.
 
[[File:Game_Flow_Chart.PNG|900px|thumb|center|Game Flowchart]]
 
  
 
The game starts off with the start screen task, which displays the game title and status for both players. As soon as both the players have pressed the button1 (to indicate that the player is ready), after a 3 second counter, the ball and slider tasks are resumed. Initially, all the tasks are suspended except the start screen task. Slider task takes the input acceleration sensor values from both the players and translates the values into appropriate movement on the display whereas the ball tasks move the ball on the display. If any of the player miss to hit the ball, the player loses and the score display task will be displayed and all other tasks would be suspended. After displaying the score for 3 seconds, we reset the ball position to randomly start from the center of the screen (4 possible directions) and start the round again. A player wins if he wins 9 rounds, and the score display screen then displays the winner along with the scores. As the game is over, the game will be restarted by suspending all tasks and resuming the start screen task again.
 
The game starts off with the start screen task, which displays the game title and status for both players. As soon as both the players have pressed the button1 (to indicate that the player is ready), after a 3 second counter, the ball and slider tasks are resumed. Initially, all the tasks are suspended except the start screen task. Slider task takes the input acceleration sensor values from both the players and translates the values into appropriate movement on the display whereas the ball tasks move the ball on the display. If any of the player miss to hit the ball, the player loses and the score display task will be displayed and all other tasks would be suspended. After displaying the score for 3 seconds, we reset the ball position to randomly start from the center of the screen (4 possible directions) and start the round again. A player wins if he wins 9 rounds, and the score display screen then displays the winner along with the scores. As the game is over, the game will be restarted by suspending all tasks and resuming the start screen task again.
  
Remote wireless task retrieves the slider values from both the players over Nordic wireless transceiver and passes it to the slider task for producing the hand gesture movement on to the display. To support the wireless communication, background Nordic wireless task participates in the mesh network and handles retry logic such that packets are resent if an ACK has not been received.
+
Remote wireless task retrieves the slider values from both the players over Nordic wireless transceiver and passes it to the slider task for producing the hand-gesture movement on to the display. To support the wireless communication, background Nordic wireless task participates in the mesh network and handles retry logic such that packets are resent if an ACK has not been received.
  
 
== Testing ==
 
== Testing ==
 
=== Module testing ===
 
=== Module testing ===
==== Acceleration Measurement ====
 
Hand gesture recognition was rigorously tested. The values received from accelerometer sensor were within a range. So, we had to test the range of values received at which the slider on screen looked responsive and smooth. We tested values ranging from -1000 to +1000 to find the right range of values. The hand gesture module transmits uint8_t. So, we checked whether the output at the minimum and maximum input values were within valid range. We also checked the uint8_t value(index) transmitted depending upon the inputs received from the accelerometer sensor.
 
 
==== LCD Display ====
 
At first, we implemented the slider movement logic and tested the slider movement on the display. This testing helped us decide the size of the slider and make sure that when the slider is moved using the accelerometer, the change in the position is perfectly reflected on the LED Matrix
 
  
We wanted to make sure, whenever the ball touches the horizontal boundaries of the LED matrix it bounces back with the proper angle. And if the ball touches the vertical boundaries of the matrix, the matrix should display the score and start the game again. To ensure the correctness of the ball movement, we implemented the ball movement logic and tested it independently on the LED Matrix.
+
==== Wireless Module ====
[[File:CmpeE244_S18_Slider_Movement.gif|150px|thumb|left|Slider Movement Testing]]
+
We started to test the basic wireless communication between two boards. Initially, we sent a single data packet from one board to the other. Once the receiver board receives the packet, it prints the content with the address of the transmitting node.
[[File:CmpeE244_S18_Ball_Movement.gif|200px|thumb|center|Ball Movement Testing]]
 
  
 +
After this, we tested with 3 boards. This time we had 2 transmitters and 1 receiver. We had to set different addresses for each node to establish mesh communication between all 3 nodes. We were sending one packet from each transmitter. The receiver board receives both the packets and prints the content of the packet with the address of the transmitting node. In this way we verified that the wireless communication between the boards is working successfully.
  
 +
==== Display Node ====
  
 +
At first, we implemented the slider movement logic and tested the slider movement on the display. This testing helped us decide the size of the slider and make sure that when the slider is moved using the accelerometer, the change in the position is perfectly reflected on the LED Matrix
  
 +
We wanted to make sure, whenever the ball touches the horizontal boundaries of the LED matrix it bounces back with the proper angle. And if the ball touches the vertical boundaries of the matrix, the matrix should display the score and start the game again. To ensure the correctness of the ball movement, we implemented the ball movement logic and tested it independently on the LED Matrix.
  
 
Also, we designed the Start screen and Game End screen and tested them on the LED display. While testing the screens we focused on having proper text font and placement of the text on the display.
 
Also, we designed the Start screen and Game End screen and tested them on the LED display. While testing the screens we focused on having proper text font and placement of the text on the display.
[[File:CmpE244_S18_LED_Matrix_Start_Screen.png|200px|thumb|left|Start Screen]]
 
[[File:CmpE244_S18_LED_Matrix_Game_End_Screen.png|200px|thumb|center|Game End Screeen]]
 
  
 +
{|
 +
|[[File:CmpE244_S18_LED_Matrix_Start_Screen.png|300px|thumb|left|'''Start Screen''']]
 +
|[[File:CmpeE244_S18_Slider_Movement.gif|300px|thumb|'''Slider Movement Testing''']]
 +
|[[File:CmpeE244_S18_Ball_Movement.gif|300px|thumb|right|'''Ball Movement Testing''']]
 +
|[[File:CmpE244_S18_LED_Matrix_Game_End_Screen.png|300px|thumb|right|'''Game End Screeen''']]
 +
|}
  
 +
==== Hand-gesture Control Module ====
 +
Hand-gesture recognition was rigorously tested. The values received from accelerometer sensor were within a range. So, we had to test the range of values received at which the slider on screen looked responsive and smooth. We tested values ranging from -1000 to +1000 to find the right range of values. The hand-gesture module transmits uint8_t. So, we checked whether the output at the minimum and maximum input values were within valid range. We also checked the uint8_t value(index) transmitted depending upon the inputs received from the accelerometer sensor.
  
 
==== PCB Testing ====
 
==== PCB Testing ====
Line 413: Line 452:
  
 
=== Integration testing ===
 
=== Integration testing ===
After testing the individual modules, we decided to integrate display node and left-hand control board first and test it. Once we got the desired result, we integrated the right-hand board with the display node. This methodology helped us identify the wireless communication challenges.  
+
After testing the individual modules, we decided to integrate display node and left-hand control board first and test it. Once we were able to control left slider satisfactorily, we integrated the right-hand control board with the display node. This methodology helped us identify that there was a time lag in receiving the accelerometer data from the hand-gesture controller boards.  
After testing the display nodes with the hand control boards, we decided to integrate the ball movement logic with. This approach helped us in deciding the suitable speed of the ball and displaying the score if any player loses. Also, the integration testing helped us identify some major issues such as flickering of the LED matrix.
+
After testing the display nodes with the hand control boards, we decided to integrate the ball movement logic with them. This approach helped us in deciding the suitable speed of the ball.
  
 
== Technical Challenges ==
 
== Technical Challenges ==
=== Display driver implementation ===
 
  
We initially started to mimic the Adafruit library, but the library was originally designed to work with Arduino (AVR), and it consisted of some assembly language instructions to save CPU cycle. Hence, we were unsuccessful in porting the library entirely. We read more about the RGB LED matrix and decided to get control over a single row first and over a single pixel. After some trial and error bit-banging, we were successful in gaining control over a single row and a single pixel on the Matrix display. Once that was achieved, we moved on to implementing higher level APIs on top of it like DrawLine, DrawRectangle, etc. Thought was also given on interfacing the display over SPI, but the display needs 6 parallel data outputs at a time, so driving through SPI wouldn't have been the optimized solution for this problem.
+
=== Display Driver Implementation ===
 +
 
 +
We initially started to mimic the Adafruit library, but the library was originally designed to work with Arduino (AVR), and it consisted of some assembly language instructions to save CPU cycle. Hence, we were unsuccessful in porting the library entirely. We read more about the RGB LED matrix and decided to get control over a single row first and then over a single pixel. We first implemented a function to first select the row according to A, B, and C signals and write the corresponding pixel values of all LEDs in those rows with the updated values from the frame buffer. The values of the pixels were set only within the clock trigger. In the end, the pixel values were latched before the output enable signal was asserted again. This function was called from a periodic task with the 1 millisecond delay in order to get the refresh rate of the display as 1 millisecond. This helped us to get the control over individual pixels of the matrix.
  
 
=== RGB LED Matrix Flickering Issue ===
 
=== RGB LED Matrix Flickering Issue ===
  
 
We need to refresh the LED matrix to display the new data. Earlier we were refreshing the matrix after every 100 milliseconds. Because the refresh rate was low we saw there was some flickering in the matrix. So, we decided to increase the refresh rate and kept it at 1 milliseconds. This refresh rate is very small for the human eyes to detect, the flickering in the matrix stopped appearing.
 
We need to refresh the LED matrix to display the new data. Earlier we were refreshing the matrix after every 100 milliseconds. Because the refresh rate was low we saw there was some flickering in the matrix. So, we decided to increase the refresh rate and kept it at 1 milliseconds. This refresh rate is very small for the human eyes to detect, the flickering in the matrix stopped appearing.
 +
 +
=== Time lag between reading the accelerometer data and displaying it on the LED matrix ===
 +
 +
[[File:CmpE244_S18_LookupTable.PNG|500px|thumb|right|Lookup Table For Drawing A Line]]
 +
There was a noticeable lag between the detection of wrist movement and the time at which it was displayed on the RGB LED Matrix. In the first code, we had used strings for transmitting data from player nodes to control node. We were transmitting actual values. These two things were the reason for the lag.
 +
So, we made the following changes:
 +
 +
1] We transmitted uint8_t values from player nodes to control nodes which reduced the int to string conversion at both the ends.
 +
 +
2] We had to draw lines for the slider movement for both the players on the RGB LED matrix. Drawing a line depends on two points, so we needed the start index and the end index. The calculation of these indexes was done on the control node side. Instead of performing operations on raw data at the control node side, we performed operations on the player node side and transmitted an uint8_t value to the control node. The control node has a two-dimensional lookup table which contains the start and the end index for drawing a line. This two changes reduced the lag considerably.
  
 
=== PCB Designing ===
 
=== PCB Designing ===
Line 429: Line 479:
 
The toughest challenge in designing the PCB Layout and Schematic was to select the exact Pin Header Libraries of the accurate measurement of a 3.5mm spacing.
 
The toughest challenge in designing the PCB Layout and Schematic was to select the exact Pin Header Libraries of the accurate measurement of a 3.5mm spacing.
 
This problem especially occurred for the power connectors the project needed for powering up the RGB LED matrix as the connector is slightly unconventional compared to the normal 2mm spacing.
 
This problem especially occurred for the power connectors the project needed for powering up the RGB LED matrix as the connector is slightly unconventional compared to the normal 2mm spacing.
To suffice this need, a new library was created in Eagle v8.2.1 Software by taking a reference of an existing library which has the same spacing between the pins on the header.
+
To suffice this need, a new library was created in Eagle v8.2.1 Software by taking a reference to an existing library which has the same spacing between the pins on the header.
  
 
== Conclusion ==
 
== Conclusion ==
The project was a success both in terms of output and learnings. We were able to play a multiplayer ping-pong game by wirelessly controlling the display on the LED Matrix by using hand gesture. This project helped us learn the working of the following modules in particular: RGB LED matrix, Nordic wireless unit, accelerometer on the SJ-One board. We could identify synchronization issues and tried to fix them by unit testing. The hardware which included a PCB and LED matrix had to be designed and packaged to give an overall compact look to the project so that it looks more like a gaming console product.
+
The project was a success both in terms of output and learnings. We were able to build a multiplayer ping-pong game which is controlled wirelessly using hand gestures. It provided us an exposure of working with RGB LED matrix, Nordic wireless chip, and accelerometer. We learned to write a driver for an RGB LED matrix. It helped us utilize the knowledge of FreeRTOS which we gained through the CMPE-244 class. We were able to give an overall compact look to the display unit and hand gesture control board to make it a complete gaming console product. Also, using Git for version control helped us keep the track of individual's work progress and it became easier for us to share the code with the team.
We enjoyed doing this project and getting better at using FreeRTOS APIs, debugging and hardware designing.
+
 
 +
Working as a team and conducting the brainstorming sessions helped us in coming up with some good ideas. Also, it helped in expediting the integration testing and debugging of the project during the final days. Overall, we got an experience of working in a team and developing a complete product using professional tools and methodologies.
 +
 
  
 
=== Project Video ===
 
=== Project Video ===
Game Video - [https://www.youtube.com/watch?v=rA3D4OwXwDc&feature=youtu.be Hand Gesture Controlled Ping Pong Game]
+
* [https://www.youtube.com/watch?v=-Gn6-So41Qk Hand-gesture Controlled Ping Pong Game]
  
 
=== Project Source Code ===
 
=== Project Source Code ===
Line 453: Line 505:
  
 
[4] [https://www.sparkfun.com/datasheets/Components/SMD/nRF24L01Pluss_Preliminary_Product_Specification_v1_0.pdf Nordic wireless datasheet]
 
[4] [https://www.sparkfun.com/datasheets/Components/SMD/nRF24L01Pluss_Preliminary_Product_Specification_v1_0.pdf Nordic wireless datasheet]
 +
 +
[5] [https://github.com/adafruit/Adafruit-GFX-Library Adafruit GFX Library]

Latest revision as of 09:52, 27 May 2018

Ping Pong

Project Title

Hand-gesture controlled multiplayer game

Abstract

Human-machine interfaces are constantly evolving. The interfaces used in HMI, based on natural gestures of users, allow converting these movements into commands for a computer system. In this project, we developed a multiplayer Ping-Pong game using a 16x32 RGB LED Matrix, accelerometer, wireless connectivity, FreeRTOS APIs and SJOne boards. The game works as a classic ping pong game in which there are two sliders which are controlled wirelessly by two hand-gesture controllers. If the ball hits any of the vertical boundaries of the matrix then the player loses the game. Each hand-gesture controller has an accelerometer sensor interfaced to SJOne board and it transmits data wirelessly to display node. The display node has the LED matrix interfaced to SJOne board and it receives the commands wirelessly from the hand-gesture controllers. A wireless mesh network is created between the display node and the hand-gesture boards to send accelerometer values. We have designed three screens for the game: Start screen, Game screen, and Result screen. This project helps us understand the communication and synchronization of FreeRTOS tasks in an interesting way.

Objectives & Introduction

Architecture Overview

Objective

This project involves learning what goes behind developing a 2D Multiplayer game. Translating the onboard acceleration sensor in a way to be meaningful and measurable for the display node to display the slider movement from the two players over the onboard wireless Nordic transceiver.

Introduction

The overall architecture of the design includes a display node, two hand-gesture control modules, and wireless communication module.

Display Node: It has a 16x32 LED Matrix interfaced to SJOne board. It is responsible for driving the LED matrix

Hand-gesture control module: The two hand-gesture control boards are used to control the game. They have SJOne board with an onboard accelerometer. The microcontroller in the SJOne board continuously reads the data from the accelerometer and sends signals wirelessly to the display node

Wireless module: It is used as a communication medium between the display node and the hand-gesture controlled boards`

Team Members & Responsibilities

  • Akil Khan
    • Design and development of the RGB Led Matrix driver
    • Game logic development, implementation, QA, and testing.
  • Abhilash Tuse
    • Game logic development
    • Logic Development for communication between hand-gesture control board and controller board
  • Disha Patil
    • Hand-gesture control logic development using an accelerometer
    • Design of PCB using Eagle for Power Supply
  • Omkar Kale
    • Hand-gesture control logic development using an accelerometer
    • Logic Development for communication between hand-gesture control board and controller board
  • Vishal Shrivastava
    • Design and development of the RGB Led Matrix driver
    • Design of PCB using Eagle for Power Supply

Schedule

Week# Date Task Status Actual Completion Date
1 04/10/2018
  • Submission of Project Proposals
  • Research on project requirements.
  • Order components and distribute project modules.
  • Completed
  • Completed
  • Completed
  • 03/24/2018
  • 04/10/2018
  • 04/10/2018
2 04/17/2018
  • Developing logic for hand-gesture control using accelerometer sensor.
  • Power Supply circuit design and simulation using Multisim
  • Write basic LED display driver to blink individual pixels and set of pixels.
  • Establishing basic wireless communication between 2 SJOne boards.
  • Completed
  • Completed
  • Completed
  • Completed
  • 04/15/2018
  • 04/16/2018
  • 04/17/2018
  • 04/18/2018
3 04/24/2018
  • Develop advanced API's on top of LED display driver: Draw game objects such as sliders and ball.
  • PCB layout design using Eagle and finalizing the schematic.
  • Establish communication between the hand-gesture controller and display node using wireless mesh module.
  • Project report update on the wiki.
  • Completed
  • Completed
  • Completed
  • Completed
  • 05/01/2018
  • 04/29/2018
  • 04/24/2018
  • 05/23/2018
4 05/01/2018
  • Developing logic for the ball movement and translating hand-gesture control into slider movement.
  • Soldering components and hardware testing on PCB
  • Packaging of hardware board and related components.
  • Project report update on the wiki.
  • Completed
  • Completed
  • Completed
  • Completed
  • 05/01/2018
  • 05/19/2018
  • 05/09/2018
  • 05/19/2018
5 05/08/2018
  • Integrate ball movement and slider movement logic into display node algorithm.
  • Project report update on the wiki.
  • Completed
  • Completed
  • 05/05/2018
  • 05/19/2018
6 04/15/2018
  • Final testing and bug fixes.
  • Complete wiki report and final demo.
  • Completed
  • Completed
  • 05/23/2018
  • 05/23/2018

Parts List & Cost

Item# Part Manufacturer Quantity Cost($)
1 SJ One Board Preet 3 80.00
2 Adafruit RGB LED Matrix LED Matrix 1 35.00
3 Power Adapter Power Supply 1 7.95
4 Power Bank 800mah Aibocn 2 11.95
5 PCB Board PCBway 1 23.00
6 Miscellaneous (Barrel Jack, Jumper wires, Connectors) Excess Solution 5.00
  • Total Cost:334.85$

Design & Implementation

Hardware Design

System block diagram


Hardware design diagram gives an overview of the entire system which consists of the main 3 SJ-One controllers: 2 player boards and 1 board as the display node

  • Player boards use the onboard accelerometer on SJ-One which is interfaced using the I2C protocol. Each of the player boards read these accelerometer values and determine the position of the slider on the display.
  • As we are implementing a wireless multiplayer game, the accelerometer values are transmitted using yet another onboard module, the Nordic wireless module which interfaces with the wireless module on the third controller used for the display.
  • The Display node SJ-One board is used to control a 16*32 RGB LED Matrix. This matrix displays 2 sliders and a ball representing the players in a Ping-Pong game. Slider locations are decided by the values obtained wirelessly from Player boards.
  • RGB LED Matrix displays three screens: Start Screen, Game Screen, End Screen which displays the final score of the players and the winner as well. The hardware description of the LED matrix is explained later in this section.


Display Node

RGB LED Matrix
16x32 RGB LED Matrix

The RGB LED matrix panels are normally used to make video walls to display animations or short video clips. It has 512 bright RGB LEDs arranged in a 16x32 grid. These panels require 12 digital pins (6-bit data, 6-bit control) and a good 5V power supply, at least 2 amps per panel. A single 32x16 RGB matrix, running full tilt (all pixels set white), can require nearly 4 Amps of current.

All 512 LEDs can’t be driven at once. One reason is that would require a lot of current, another reason is that it would be expensive to have so many pins. Instead, the matrix is divided into 8 interleaved sections/strips. The first section is the 1st 'line' and the 9th 'line' (32 x 2 RGB LEDs = 64 RGB LEDs), the second is the 2nd and 10th line, etc until the last section which is the 7th and 16th line. The reason it is done in this way is so that the lines are interleaved and look better when refreshed, otherwise, we'd see the stripes more clearly. So, on the PCB is 12 LED driver chips. These are like 74HC595s but they have 16 outputs and they require constant current. 16 outputs * 12 chips = 192 LEDs that can be controlled at once, and 64 * 3 (R G and B) = 192. There are 192 outputs that can control one line at a time, with each of 192 R, G and B LEDs either on or off. The controller selects which section to draw (using A, B, and C address pins - 3 bits can have 8 values). Once the address is set, the controller clocks out 192 bits of data (24 bytes) and latches it. Then it increments the address and clocks out another 192 bits, etc until it gets to address #7, then it sets the address back to #0.

We are driving the matrix using 12 GPIO pins of SJOne board.

Pin mapping of RGB LED matrix with SJ One board:

Hardware Interface of RGB LED matrix
RGB LED matrix Pins SJ One Board Pins Function
R1 P1.19 Red Data top half
G1 P1.20 Green Data top half
B1 P1.22 Blue Data top half
R2 P1.23 Red Data bottom half
G2 P1.28 Green Data bottom half
B2 P1.29 Blue Data bottom half
A P2.0 Row select A
B P2.1 Row select B
C P2.2 Row select C
CLK P0.1 The CLK (clock) signal marks

the arrival of each bit of data.

OE P0.0 OE (output enable) switches the

LEDs off when transitioning from one row to the next.

LAT P0.26 The LAT (latch) signal marks the

end of a row of data.


Hand-gesture control module

Hand-Gesture Controller
Accelerometer

The SJ-One board has an accelerometer which is interfaced on the I2C bus. Accelerometers are electromechanical devices that sense either static or dynamic forces of acceleration. Static forces include gravity, while dynamic forces can include vibrations and movement. The measurements are on 3-axis and these values can be calibrated to find the desired values.

In our project, an accelerometer is used for detecting the up-down movement of the wrist. We are using accelerometers on two SJ-One Boards for gesture recognition. The movement from accelerometer sensor will be used to control the slider movement on the RGB LED Matrix.

Accelerometer on board
Accelerometer Intefaced With Microcontroller
Accelerometer Detection

Wireless Module

Nordic wireless chip nRF24L01+ is interfaced with SJone board using SPI bus protocol.

The nRF24L01+ is a single chip 2.4GHz transceiver with an embedded baseband protocol engine, suitable for ultra low power wireless applications. The nRF24L01+ is designed for operation in the frequency band of 2.400 - 2.483GHz. The high air data rate combined with two power saving modes make the nRF24L01+ very suitable for ultra-low power designs. The nRF24L01 integrates a complete 2.4GHz RF transceiver, RF synthesizer, and baseband logic including the Enhanced ShockBurst™ hardware protocol accelerator supporting a high-speed SPI interface for the application controller. No external loop filter, resonators, or VCO varactor diodes are required, only a low-cost ±60ppm crystal, matching circuitry, and antenna.

Nordic Wireless Chip nRF24L01+
Nordic Wireless Chip Interface with Microcontroller

Printed Circuit Board(PCB) Design

The PCB Layout is designed using the Eagle Software v8.7.1. Board consists of 2 power supplies which are powering:

  • SJone board (5V, 500mA)
  • RGB LED Matrix (5V,2A)

The Power Supply circuit has an LM317 voltage regulator IC and a voltage divider to fulfill the specific power requirements. LM317 is a TI (Texas Instruments) based IC which has an Output Voltage Range Adjustable from 1.25 V to 37 V and is suitable for our application. We have used a 12V adapter in order to power our board. This serves for both the current requirements. The circuit was simulated using MultiSim v14.1 software by NI (National Instruments). The simulation helped us understand the working of our circuit before we built and tested it.

Printed Circuit Board also has provisions for the connection of the SJone pin headers and RGB LED Matrix's connector. The connections are as shown in the hardware description diagram. The PCB also has a power switch and LED for indication. The values for the resistors and other active or passive components are as shown in the eagle schematic diagram.

Printed Circuit Board
Board Design


Software Design

Display Node

RGB LED Matrix
Software Flowchart for the LED Matrix Display
Display Node Class Diagram


This 16x32 5V,2A supply RGB LED Matrix panels from Adafruit require 12 digital pins (6-bit data, 6-bit control). They have 512 bright RGB LEDs arranged in a 16x32 grid on the front. These displays are 'chainable' - connect the output to the next input. This display requires about 800 bytes of RAM to buffer the 12-bit color image. As the matrix panel has a 6-bit parallel interface, SPI hardware won't help and hence simple GPIO interface has been used to communicate with the RGB LED Matrix. As these displays don't have built-in PWM control of any kind, the driver is supposed to redraw the screen over and over to 'manually' PWM the whole thing.

Pins R1, G1, and B1 deliver data to the top half of the display whereas R2, G2, and B2 deliver data to the bottom half of the display. Pins A, B, C, and D select which two rows of the display are currently lit. The LAT signal acts as the strobe and marks the end of a row of data. The CLK signal marks the arrival of each bit of data. OE (output enable) is an active low signal that switches the LEDs off when transitioning from one row to the next.

The implementation consists of 768-byte buffer for storing the pixel values for all LEDs on the display. Every line is scanned and the values for the pixels are updated every 1 ms. Between every transition of the clock from low to high, the values for the pixels are updated. As the screen is refreshed and updated very frequently, it gives the impression of a moving image.

The basic process used to refresh the display when using three bits-per-pixel color (one bit for red; one bit for green; and one bit for blue) is the following:

  • Shift the pixel data for row 0 into the top column drivers and the pixel data for row 16 into the bottom column drivers using the R0, G0, B0, R1, G1, and B1 data inputs and the SCLK shift clock signal.
  • Assert the blanking signal to blank the display.
  • Set the address input to 0.
  • Latch the contents of the column drivers‘ shift registers into the column drivers‘ output registers using the LATCH signal.
  • Deassert the blanking signal to display rows 0 and 16.
  • Wait some fixed amount of time.
  • Repeat the process for each of the pairs of rows in the display.
  • Repeat the entire process at least 100 to 200 times per second to prevent flicker.

Hand-gesture Control Logic

Hand-gesture Control Board Class Diagram


Flowchart for Hand-gesture Module




Accelerometer Value Calculations


The gesture controlled process which acts as an interface between the players and the display node is explained as follows:

  • Hand gesture module begins with the initialization of the onboard accelerometer sensor which is interfaced using I2C bus to the controller.
  • The Start condition is satisfied as soon as both the players press the corresponding switches on the respective boards as a sign of being ready.
  • Switch press is acknowledged and the required start code for the start screen is transmitted. The start screen displays a countdown from 3 to 1 to give a dramatic effect.
  • After the start screen is displayed, the values are received from the onboard accelerometer sensor and transmitted to the Master Node i.e. the RGB LED Matrix.
Accelerometer Value Calculation

After carefully looking at the board design and how the board will be placed on a person's hand, we found out that the values received along one axis(X-axis) will be enough to capture the wrist movement. The range of accelerometer values shows how sensitive the accelerometer sensor is. We wanted the code to be flexible in terms of the range of the values. So that we did not have to hard code the sensor values and instead used a code based on maximum and minimum range values which could be easily configurable. The piece of code mentioned below shows how the values are calculated.

Wireless Module

Wireless Communication Between the Boards

We referred Low Powered Mesh Network stack for building wireless mesh network between the display board and player boards.

Wireless Receiver

The SJ-One board connected to LED display uses onboard Nordic wireless to communicate with player boards. Initially, it waits for players to press the start button on their controllers which then sends commands to the receiver board. Once it receives the commands to start, it displays green signal on the display and changes the screen to game screen. While the game is running, receiver continuously receives the acceleration values at every 50 ms from player boards which it then sends further to the slider task.

Wireless Transmission

We used Nordic Wireless for transmitting the accelerometer data. Both the player 1 and the player 2 boards send the accelerometer data to the Display node. We created a remote wireless task which runs on medium priority and sends an integer value to the Display node at every 100 ms. Both the boards have a unique node address and a node name. For the mesh network, we had to use the same channel number and data transmission rate for both the player boards and Display node.

Game logic and Implementation

Game Flowchart


The game implementation involves creating multiple tasks for control of independent entities and also making those entities work together.

The game starts off with the start screen task, which displays the game title and status for both players. As soon as both the players have pressed the button1 (to indicate that the player is ready), after a 3 second counter, the ball and slider tasks are resumed. Initially, all the tasks are suspended except the start screen task. Slider task takes the input acceleration sensor values from both the players and translates the values into appropriate movement on the display whereas the ball tasks move the ball on the display. If any of the player miss to hit the ball, the player loses and the score display task will be displayed and all other tasks would be suspended. After displaying the score for 3 seconds, we reset the ball position to randomly start from the center of the screen (4 possible directions) and start the round again. A player wins if he wins 9 rounds, and the score display screen then displays the winner along with the scores. As the game is over, the game will be restarted by suspending all tasks and resuming the start screen task again.

Remote wireless task retrieves the slider values from both the players over Nordic wireless transceiver and passes it to the slider task for producing the hand-gesture movement on to the display. To support the wireless communication, background Nordic wireless task participates in the mesh network and handles retry logic such that packets are resent if an ACK has not been received.

Testing

Module testing

Wireless Module

We started to test the basic wireless communication between two boards. Initially, we sent a single data packet from one board to the other. Once the receiver board receives the packet, it prints the content with the address of the transmitting node.

After this, we tested with 3 boards. This time we had 2 transmitters and 1 receiver. We had to set different addresses for each node to establish mesh communication between all 3 nodes. We were sending one packet from each transmitter. The receiver board receives both the packets and prints the content of the packet with the address of the transmitting node. In this way we verified that the wireless communication between the boards is working successfully.

Display Node

At first, we implemented the slider movement logic and tested the slider movement on the display. This testing helped us decide the size of the slider and make sure that when the slider is moved using the accelerometer, the change in the position is perfectly reflected on the LED Matrix

We wanted to make sure, whenever the ball touches the horizontal boundaries of the LED matrix it bounces back with the proper angle. And if the ball touches the vertical boundaries of the matrix, the matrix should display the score and start the game again. To ensure the correctness of the ball movement, we implemented the ball movement logic and tested it independently on the LED Matrix.

Also, we designed the Start screen and Game End screen and tested them on the LED display. While testing the screens we focused on having proper text font and placement of the text on the display.

Start Screen
Slider Movement Testing
Ball Movement Testing
Game End Screeen

Hand-gesture Control Module

Hand-gesture recognition was rigorously tested. The values received from accelerometer sensor were within a range. So, we had to test the range of values received at which the slider on screen looked responsive and smooth. We tested values ranging from -1000 to +1000 to find the right range of values. The hand-gesture module transmits uint8_t. So, we checked whether the output at the minimum and maximum input values were within valid range. We also checked the uint8_t value(index) transmitted depending upon the inputs received from the accelerometer sensor.

PCB Testing

The PCB was tested by checking the connectivity of the path on the copper clad tracks using a Digital Multimeter. The circuit on the PCB was simulated using MultiSim v14.1 Software and results were verified on the actual board. All the connector-related bugs were fixed by inspection and drilling the PCB wherever necessary. The board was tested for any short circuit by checking the connectivity between the Supply tracks and common ground tracks.

Integration testing

After testing the individual modules, we decided to integrate display node and left-hand control board first and test it. Once we were able to control left slider satisfactorily, we integrated the right-hand control board with the display node. This methodology helped us identify that there was a time lag in receiving the accelerometer data from the hand-gesture controller boards. After testing the display nodes with the hand control boards, we decided to integrate the ball movement logic with them. This approach helped us in deciding the suitable speed of the ball.

Technical Challenges

Display Driver Implementation

We initially started to mimic the Adafruit library, but the library was originally designed to work with Arduino (AVR), and it consisted of some assembly language instructions to save CPU cycle. Hence, we were unsuccessful in porting the library entirely. We read more about the RGB LED matrix and decided to get control over a single row first and then over a single pixel. We first implemented a function to first select the row according to A, B, and C signals and write the corresponding pixel values of all LEDs in those rows with the updated values from the frame buffer. The values of the pixels were set only within the clock trigger. In the end, the pixel values were latched before the output enable signal was asserted again. This function was called from a periodic task with the 1 millisecond delay in order to get the refresh rate of the display as 1 millisecond. This helped us to get the control over individual pixels of the matrix.

RGB LED Matrix Flickering Issue

We need to refresh the LED matrix to display the new data. Earlier we were refreshing the matrix after every 100 milliseconds. Because the refresh rate was low we saw there was some flickering in the matrix. So, we decided to increase the refresh rate and kept it at 1 milliseconds. This refresh rate is very small for the human eyes to detect, the flickering in the matrix stopped appearing.

Time lag between reading the accelerometer data and displaying it on the LED matrix

Lookup Table For Drawing A Line

There was a noticeable lag between the detection of wrist movement and the time at which it was displayed on the RGB LED Matrix. In the first code, we had used strings for transmitting data from player nodes to control node. We were transmitting actual values. These two things were the reason for the lag. So, we made the following changes:

1] We transmitted uint8_t values from player nodes to control nodes which reduced the int to string conversion at both the ends.

2] We had to draw lines for the slider movement for both the players on the RGB LED matrix. Drawing a line depends on two points, so we needed the start index and the end index. The calculation of these indexes was done on the control node side. Instead of performing operations on raw data at the control node side, we performed operations on the player node side and transmitted an uint8_t value to the control node. The control node has a two-dimensional lookup table which contains the start and the end index for drawing a line. This two changes reduced the lag considerably.

PCB Designing

The toughest challenge in designing the PCB Layout and Schematic was to select the exact Pin Header Libraries of the accurate measurement of a 3.5mm spacing. This problem especially occurred for the power connectors the project needed for powering up the RGB LED matrix as the connector is slightly unconventional compared to the normal 2mm spacing. To suffice this need, a new library was created in Eagle v8.2.1 Software by taking a reference to an existing library which has the same spacing between the pins on the header.

Conclusion

The project was a success both in terms of output and learnings. We were able to build a multiplayer ping-pong game which is controlled wirelessly using hand gestures. It provided us an exposure of working with RGB LED matrix, Nordic wireless chip, and accelerometer. We learned to write a driver for an RGB LED matrix. It helped us utilize the knowledge of FreeRTOS which we gained through the CMPE-244 class. We were able to give an overall compact look to the display unit and hand gesture control board to make it a complete gaming console product. Also, using Git for version control helped us keep the track of individual's work progress and it became easier for us to share the code with the team.

Working as a team and conducting the brainstorming sessions helped us in coming up with some good ideas. Also, it helped in expediting the integration testing and debugging of the project during the final days. Overall, we got an experience of working in a team and developing a complete product using professional tools and methodologies.


Project Video

Project Source Code

References

Acknowledgement

We would like to thank our Professor Preetpal Kang for all his teachings and inspirational lectures. Not only did we enjoy working though out this project but also gave us an overall learning experience and precious life lessons. We would also like to thank the ISA members for always being ready to help with whatever issues we faced.

References Used

[1] FreeRTOS documentations

[2] Adafruit LCD library

[3] CMPE 244 Lecture notes from Preetpal Kang, Computer Engineering, San Jose State University. Feb - May 2018.

[4] Nordic wireless datasheet

[5] Adafruit GFX Library