Difference between revisions of "S18: Hand gesture controlled multiplayer game"

Revision as of 20:35, 24 April 2018 (view source) Proj user10 (talk | contribs) (→‎Parts List & Cost) ← Older edit		Latest revision as of 09:52, 27 May 2018 (view source) Proj user10 (talk | contribs) (→‎Hand-gesture control module)
(308 intermediate revisions by the same user not shown)
Line 1:		Line 1:
−	=== Grading Criteria ===	+	[[File:CmpE244_Viserion_ProjectImage.PNG|thumb|589x615px|right|Ping Pong]]
−	<font color="green">
−	*  How well is Software & Hardware Design described?
−	*  How well can this report be used to reproduce this project?
−	*  Code Quality
−	*  Overall Report Quality:
−	**  Software Block Diagrams
−	**  Hardware Block Diagrams
−	**:  Schematic Quality
−	**  Quality of technical challenges and solutions adopted.
−	</font>
	== Project Title ==		== Project Title ==
−	Hand gesture controlled multiplayer game	+	===Hand-gesture controlled multiplayer game===
	== Abstract ==		== Abstract ==
−	This project involves implementation of a wireless and hand-gesture controlled multiplayer ping pong (2D) game using the Adafruit's 16x32 RGB LED Matrix. It incorporates the on-board RF Nordic wireless transceiver for wireless communication and the on-board acceleration sensor to translate the hand movement into the slider movement on the display.	+	Human-machine interfaces are constantly evolving. The interfaces used in HMI, based on natural gestures of users, allow converting these movements into commands for a computer system. In this project, we developed a multiplayer Ping-Pong game using a 16x32 RGB LED Matrix, accelerometer, wireless connectivity, FreeRTOS APIs and SJOne boards. The game works as a classic ping pong game in which there are two sliders which are controlled wirelessly by two hand-gesture controllers. If the ball hits any of the vertical boundaries of the matrix then the player loses the game. Each hand-gesture controller has an accelerometer sensor interfaced to SJOne board and it transmits data wirelessly to display node. The display node has the LED matrix interfaced to SJOne board and it receives the commands wirelessly from the hand-gesture controllers. A wireless mesh network is created between the display node and the hand-gesture boards to send accelerometer values. We have designed three screens for the game: Start screen, Game screen, and Result screen. This project helps us understand the communication and synchronization of FreeRTOS tasks in an interesting way.
	== Objectives & Introduction ==		== Objectives & Introduction ==
−	Show list of your objectives. This section includes the high level details of your project. You can write about the various sensors or peripherals you used to get your project completed.	+	[[File:Architecture_Overview.png|550px|thumb|right|Architecture Overview]]
		+	=== Objective ===
		+	This project involves learning what goes behind developing a 2D Multiplayer game. Translating the onboard acceleration sensor in a way to be meaningful and measurable for the display node to display the slider movement from the two players over the onboard wireless Nordic transceiver.
−	=== Team Members & Responsibilities ===	+	===Introduction ===
−	* Akil Khan	+
		+	The overall architecture of the design includes a display node, two hand-gesture control modules, and wireless communication module.
		+
		+	'''Display Node:''' It has a 16x32 LED Matrix interfaced to SJOne board. It is responsible for driving the LED matrix
		+
		+	'''Hand-gesture control module:''' The two hand-gesture control boards are used to control the game. They have SJOne board with an onboard accelerometer. The microcontroller in the SJOne board continuously reads the data from the accelerometer and sends signals wirelessly to the display node
		+
		+	'''Wireless module:''' It is used as a communication medium between the display node and the hand-gesture controlled boards`
		+
		+	== Team Members & Responsibilities ==
		+	*'''''[https://www.linkedin.com/in/akil-khan/ Akil Khan]'''''
	**  Design and development of the RGB Led Matrix driver		**  Design and development of the RGB Led Matrix driver
		+	**  Game logic development, implementation, QA, and testing.
		+	*'''''[https://www.linkedin.com/in/abhilash-tuse/ Abhilash Tuse]'''''
	**  Game logic development		**  Game logic development
−	* Abhilash Tuse	+	**  Logic Development for communication between hand-gesture control board and controller board
−	**	+	*'''''[https://www.linkedin.com/in/dishapatil1494/ Disha Patil]'''''
−	* Omkar Kale	+	**  Hand-gesture control logic development using an accelerometer
−	**  Hand gesture control logic development using accelerometer	+	**  Design of PCB using Eagle for Power Supply
−	**  Logic Development for communication between hand gesture control and LED Matrix display modules	+	*'''''[https://www.linkedin.com/in/omkar-kale/ Omkar Kale]'''''
−	* Disha Patil	+	**  Hand-gesture control logic development using an accelerometer
		+	**  Logic Development for communication between hand-gesture control board and controller board
		+	*'''''[https://www.linkedin.com/in/vishalrshrivastava/ Vishal Shrivastava]'''''
		+	**  Design and development of the RGB Led Matrix driver
	**  Design of PCB using Eagle for Power Supply		**  Design of PCB using Eagle for Power Supply
−	**  Hand gesture control logic development using accelerometer
−	*  Vishal Shrivastava
−	**
	== Schedule ==		== Schedule ==
−	Show a simple table or figures that show your scheduled as planned before you started working on the project.  Then in another table column, write down the actual schedule so that readers can see the planned vs. actual goals.  The point of the schedule is for readers to assess how to pace themselves if they are doing a similar project.
−
	{| class="wikitable"		{| class="wikitable"
	|-		|-
Line 48:		Line 50:
	|-		|-
	! scope="row"| 1		! scope="row"| 1
−	| 10/04/2018	+	| 04/10/2018
	|		|
		+	* Submission of Project Proposals
	* Research on project requirements.		* Research on project requirements.
	* Order components and distribute project modules.		* Order components and distribute project modules.
	|		|
		+	* Completed
	* Completed		* Completed
	* Completed		* Completed
	|		|
−	* 10/04/2018	+	* 03/24/2018
−	* 10/04/2018	+	* 04/10/2018
		+	* 04/10/2018
	|-		|-
	! scope="row"| 2		! scope="row"| 2
−	| 17/04/2018	+	| 04/17/2018
	|		|
−	* Developing logic for hand gesture control using accelerometer sensor.	+	* Developing logic for hand-gesture control using accelerometer sensor.
−	* Power Supply circuit design using Multisim	+	* Power Supply circuit design and simulation using Multisim
	* Write basic LED display driver to blink individual pixels and set of pixels.		* Write basic LED display driver to blink individual pixels and set of pixels.
	* Establishing basic wireless communication between 2 SJOne boards.		* Establishing basic wireless communication between 2 SJOne boards.
	|		|
	* Completed		* Completed
−	*	+	* Completed
−	*	+	* Completed
	* Completed		* Completed
	|		|
−	* 17/04/2018	+	* 04/15/2018
−	*	+	* 04/16/2018
−	*	+	* 04/17/2018
−	* 18/04/2018	+	* 04/18/2018
	|-		|-
	! scope="row"| 3		! scope="row"| 3
−	| 24/04/2018	+	| 04/24/2018
	|		|
−	* LED display driver: Draw game objects such as player lines and ball over LED matrix.	+	* Develop advanced API's on top of LED display driver: Draw game objects such as sliders and ball.
−	* PCB design using Eagle.	+	* PCB layout design using Eagle and finalizing the schematic.
−	* Establish communication between hand gesture and controller boards using wireless mesh module.	+	* Establish communication between the hand-gesture controller and display node using wireless mesh module.
	* Project report update on the wiki.		* Project report update on the wiki.
	|		|
−	*	+	* Completed
−	*	+	* Completed
	* Completed		* Completed
	* Completed		* Completed
	|		|
−	*	+	* 05/01/2018
−	*	+	* 04/29/2018
−	* 24/04/2018	+	* 04/24/2018
−	* 24/04/2018	+	* 05/23/2018
	|-		|-
	! scope="row"| 4		! scope="row"| 4
−	| 01/05/2018	+	| 05/01/2018
	|		|
−	* Game algorithm design and implementation.	+	* Developing logic for the ball movement and translating hand-gesture control into slider movement.
	* Soldering components and hardware testing on PCB		* Soldering components and hardware testing on PCB
−	* Identify extra features/options to be provided in the game.	+	* Packaging of hardware board and related components.
	* Project report update on the wiki.		* Project report update on the wiki.
	|		|
−	*	+	* Completed
−	*	+	* Completed
−	*	+	* Completed
−	*	+	* Completed
	|		|
−	*	+	* 05/01/2018
−	*	+	* 05/19/2018
−	*	+	* 05/09/2018
−	*	+	* 05/19/2018
	|-		|-
	! scope="row"| 5		! scope="row"| 5
−	| 08/05/2018	+	| 05/08/2018
	|		|
−	* Integrate modules and test gameplay.	+	* Integrate ball movement and slider movement logic into display node algorithm.
−	* Control game options using switches.
	* Project report update on the wiki.		* Project report update on the wiki.
	|		|
−	*	+	* Completed
−	*	+	* Completed
−	*
	|		|
−	*	+	* 05/05/2018
−	*	+	* 05/19/2018
−	*
	|-		|-
	! scope="row"| 6		! scope="row"| 6
−	| 15/05/2018	+	| 04/15/2018
	|		|
	* Final testing and bug fixes.		* Final testing and bug fixes.
	* Complete wiki report and final demo.		* Complete wiki report and final demo.
	|		|
−	*	+	* Completed
−	*	+	* Completed
	|		|
−	*	+	* 05/23/2018
−	*	+	* 05/23/2018
	|-		|-
	|}		|}
Line 148:		Line 150:
	! scope="col"| Part		! scope="col"| Part
	! scope="col"| Manufacturer		! scope="col"| Manufacturer
−	! scope="col"| Part number
	! scope="col"| Quantity		! scope="col"| Quantity
−	! scope="col"| Total Cost($)	+	! scope="col"| Cost($)
	|-		|-
	! scope="row"| 1		! scope="row"| 1
	| SJ One Board		| SJ One Board
	|[http://www.socialledge.com/sjsu/index.php?title=SJ_One_Board Preet]		|[http://www.socialledge.com/sjsu/index.php?title=SJ_One_Board Preet]
−	| N/A
	| 3		| 3
−	| 240.00	+	| 80.00
	|-		|-
	! scope="row"| 2		! scope="row"| 2
	| Adafruit RGB LED Matrix		| Adafruit RGB LED Matrix
	|[https://www.adafruit.com/product/420 LED Matrix]		|[https://www.adafruit.com/product/420 LED Matrix]
−	|
	| 1		| 1
	| 35.00		| 35.00
	|-		|-
	! scope="row"| 3		! scope="row"| 3
−	| Power Supply For LED Matrix	+	| Power Adapter
	|[https://www.adafruit.com/product/276 Power Supply]		|[https://www.adafruit.com/product/276 Power Supply]
−	|	+	|1
−	| 7.95	+	|7.95
	|-		|-
−	! scope="row"| 4	+	! scope="row"| 4
−	| 10K Precision Epoxy Thermistor - 3950 NTC	+	| Power Bank 800mah
−	|[https://www.adafruit.com/product/372 Adafruit]	+	|[https://www.amazon.com/dp/B073P83PHF/ref=sspa_dk_detail_0?psc=1&pd_rd_i=B073P83PHF&pd_rd_wg=ubl19&pd_rd_r=DAQV9R6RCAW5N9XK7WE1&pd_rd_w=4WrKg Aibocn]
−	|[https://cdn-shop.adafruit.com/datasheets/103_3950_lookuptable.pdf Lookup Table]	+	| 2
−	| 4.00	+	| 11.95
	|-		|-
		+
	! scope="row"| 5		! scope="row"| 5
−	| LCD module	+	| PCB Board
−	|[https://www.aliexpress.com/item/2-8-inch-TFT-LCD-SPI-modules-240x320-ILI9340-or-equal-IC-with-PCB/32791470288.html Aliexpress]	+	| [https://www.pcbway.com/ PCBway]
−	| SEN-10988	+	| 1
−	| 32.30	+	| 23.00
	|-		|-
	! scope="row"| 6		! scope="row"| 6
−	| Hardware	+	| Miscellaneous (Barrel Jack, Jumper wires, Connectors)
	| Excess Solution		| Excess Solution
−	| Berg strips, Jumper wires, Connectors, Standoffs	+	|
	| 5.00		| 5.00
	|-		|-
−	! scope="row"| 7	+
−	| 9V Battery
−	| [https://www.amazon.com/AmazonBasics-Everyday-Alkaline-Batteries-8-Pack/dp/B00MH4QM1S/ref=sr_1_3_a_it?ie=UTF8&qid=1495354532&sr=8-3&keywords=9v+battery Amazon]
−	|
−	| 9.50
−	|-
	|}		|}
−	*Total Cost: 157.35$	+	*Total Cost:334.85$
	== Design & Implementation ==		== Design & Implementation ==
−	The design section can go over your hardware and software design.  Organize this section using sub-sections that go over your design and implementation.
	=== Hardware Design ===		=== Hardware Design ===
−	Discuss your hardware design here.  Show detailed schematics, and the interface here.
−	=== Hardware Interface ===	+	[[File:CmpE244_S18_System_Block_Diagram.jpg|570px|thumb|center|'''System block diagram''']]
−	In this section, you can describe how your hardware communicates, such as which BUSes used.  You can discuss your driver implementation here, such that the '''Software Design''' section is isolated to talk about high level workings rather than inner working of your project.	+
		+
		+
		+	Hardware design diagram gives an overview of the entire system which consists of the main 3 SJ-One controllers: 2 player boards and 1 board as the display node
		+
		+	*Player boards use the onboard accelerometer on SJ-One which is interfaced using the I2C protocol. Each of the player boards read these accelerometer values and determine the position of the slider on the display.
		+	*As we are implementing a wireless multiplayer game, the accelerometer values are transmitted using yet another onboard module, the Nordic wireless module which interfaces with the wireless module on the third controller used for the display.
		+	*The Display node SJ-One board is used to control a 16*32 RGB LED Matrix. This matrix displays 2 sliders and a ball representing the players in a Ping-Pong game. Slider locations are decided by the values obtained wirelessly from Player boards.
		+	*RGB LED Matrix displays three screens: Start Screen, Game Screen, End Screen which displays the final score of the players and the winner as well. The hardware description of the LED matrix is explained later in this section.
		+
		+
		+	==== Display Node ====
		+
		+	===== RGB LED Matrix =====
		+	[[File:RGB_LED_Matrix.PNG|300px|thumb|16x32 RGB LED Matrix]]
		+
		+	The RGB LED matrix panels are normally used to make video walls to display animations or short video clips. It has 512 bright RGB LEDs arranged in a 16x32 grid. These panels require 12 digital pins (6-bit data, 6-bit control) and a good 5V power supply, at least 2 amps per panel. A single 32x16 RGB matrix, running full tilt (all pixels set white), can require nearly 4 Amps of current.
		+
		+	All 512 LEDs can’t be driven at once. One reason is that would require a lot of current, another reason is that it would be expensive to have so many pins. Instead, the matrix is divided into 8 interleaved sections/strips. The first section is the 1st 'line' and the 9th 'line' (32 x 2 RGB LEDs = 64 RGB LEDs), the second is the 2nd and 10th line, etc until the last section which is the 7th and 16th line. The reason it is done in this way is so that the lines are interleaved and look better when refreshed, otherwise, we'd see the stripes more clearly. So, on the PCB is 12 LED driver chips. These are like 74HC595s but they have 16 outputs and they require constant current. 16 outputs * 12 chips = 192 LEDs that can be controlled at once, and 64 * 3 (R G and B) = 192. There are 192 outputs that can control one line at a time, with each of 192 R, G and B LEDs either on or off. The controller selects which section to draw (using A, B, and C address pins - 3 bits can have 8 values). Once the address is set, the controller clocks out 192 bits of data (24 bytes) and latches it. Then it increments the address and clocks out another 192 bits, etc until it gets to address #7, then it sets the address back to #0.
		+
		+	We are driving the matrix using 12 GPIO pins of SJOne board.
		+
		+	Pin mapping of RGB LED matrix with SJ One board:
		+	[[File:Hw_interface.png | 750px | thumb | right | '''Hardware Interface of RGB LED matrix''']]
		+	{| class="wikitable"
		+	|-
		+	! align="center"|RGB LED matrix Pins
		+	! align="center"|SJ One Board Pins
		+	! align="center"|Function
		+	|-
		+	| scope="row" align="center"|R1
		+	| scope="row" align="center"|P1.19
		+	| scope="row" align="center"|Red Data top half
		+	|-
		+	| scope="row" align="center"|G1
		+	| scope="row" align="center"|P1.20
		+	| scope="row" align="center"|Green Data top half
		+	|-
		+	| scope="row" align="center"|B1
		+	| scope="row" align="center"|P1.22
		+	| scope="row" align="center"|Blue Data top half
		+	|-
		+	| scope="row" align="center"|R2
		+	| scope="row" align="center"|P1.23
		+	| scope="row" align="center"|Red Data bottom half
		+	|-
		+	| scope="row" align="center"|G2
		+	| scope="row" align="center"|P1.28
		+	| scope="row" align="center"|Green Data bottom half
		+	|-
		+	| scope="row" align="center"|B2
		+	| scope="row" align="center"|P1.29
		+	| scope="row" align="center"|Blue Data bottom half
		+	|-
		+	| scope="row" align="center"|A
		+	| scope="row" align="center"|P2.0
		+	| scope="row" align="center"|Row select A
		+	|-
		+	| scope="row" align="center"|B
		+	| scope="row" align="center"|P2.1
		+	| scope="row" align="center"|Row select B
		+	|-
		+	| scope="row" align="center"|C
		+	| scope="row" align="center"|P2.2
		+	| scope="row" align="center"|Row select C
		+	|-
		+	| scope="row" align="center"|CLK
		+	| scope="row" align="center"|P0.1
		+	| scope="row" align="center"|The CLK (clock) signal marks
		+	the arrival of each bit of data.
		+	|-
		+	| scope="row" align="center"|OE
		+	| scope="row" align="center"|P0.0
		+	| scope="row" align="center"|OE (output enable) switches the
		+	LEDs off when transitioning
		+	from one row to the next.
		+	|-
		+	| scope="row" align="center"|LAT
		+	| scope="row" align="center"|P0.26
		+	| scope="row" align="center"|The LAT (latch) signal marks the
		+	end of a row of data.
		+	|-
		+	|}
		+
		+
		+	==== Hand-gesture control module ====
		+	[[File:CmpE244_S18_Hand_Gesture_Controller.png|400px|thumb|center|Hand-Gesture Controller]]
		+	===== Accelerometer =====
		+	The SJ-One board has an accelerometer which is interfaced on the I2C bus. Accelerometers are electromechanical devices that sense either static or dynamic forces of acceleration. Static forces include gravity, while dynamic forces can include vibrations and movement. The measurements are on 3-axis and these values can be calibrated to find the desired values.
		+
		+	In our project, an accelerometer is used for detecting the up-down movement of the wrist. We are using accelerometers on two SJ-One Boards for gesture recognition. The movement from accelerometer sensor will be used to control the slider movement on the RGB LED Matrix.
		+	{|
		+	|[[File:CMPE244_S18_SS_Accelerometer.PNG|200px|thumb|left|Accelerometer on board]]
		+	|[[File:Accelerometer_Interface.PNG|400px|thumb|center|Accelerometer Intefaced With Microcontroller]]
		+	|[[File:CmpE244_S18_Detectable_Accelerations.png|400px|thumb|right|Accelerometer Detection]]
		+	|}
		+
		+	==== Wireless Module====
		+	{|
		+	|Nordic wireless chip nRF24L01+ is interfaced with SJone board using SPI bus protocol.
		+	The nRF24L01+ is a single chip 2.4GHz transceiver with an embedded baseband protocol engine, suitable for ultra low
		+	power wireless applications. The nRF24L01+ is designed for operation in the frequency band of 2.400 - 2.483GHz. The
		+	high air data rate combined with two power saving modes make the nRF24L01+ very suitable for ultra-low power designs.  The nRF24L01 integrates a complete 2.4GHz RF transceiver, RF synthesizer, and baseband logic including the Enhanced ShockBurst™ hardware protocol accelerator supporting a high-speed SPI interface for the application controller. No external loop filter, resonators, or VCO varactor diodes are required, only a low-cost ±60ppm crystal, matching circuitry, and antenna.
		+	|[[File:CmpE244_S18_Nordic_Wireless.jpg|350px|thumb|right|'''Nordic Wireless Chip nRF24L01+''' ]]
		+	|[[File:Wireless_Chip_Interface.PNG|350px|thumb|center|'''Nordic Wireless Chip Interface with Microcontroller''']]
		+	|}
		+
		+	==== Printed Circuit Board(PCB) Design ====
		+
		+	The PCB Layout is designed using the Eagle Software v8.7.1.
		+	Board consists of 2 power supplies which are powering:
		+	*SJone board (5V, 500mA)
		+	*RGB LED Matrix (5V,2A)
		+	The Power Supply circuit has an LM317 voltage regulator IC and a voltage divider to fulfill the specific power requirements.
		+	LM317 is a TI (Texas Instruments) based IC which has an Output Voltage Range Adjustable from 1.25 V to 37 V and is suitable for our application.
		+	We have used a 12V adapter in order to power our board. This serves for both the current requirements.
		+	The circuit was simulated using MultiSim v14.1 software by NI (National Instruments).
		+	The simulation helped us understand the working of our circuit before we built and tested it.
		+
		+	Printed Circuit Board also has provisions for the connection of the SJone pin headers and RGB LED Matrix's connector.
		+	The connections are as shown in the hardware description diagram. The PCB also has a power switch and LED for indication.
		+	The values for the resistors and other active or passive components are as shown in the eagle schematic diagram.
		+	{|
		+	|[[File:CmpeE244_S18_PCB.jpeg|500px|thumb|center|Printed Circuit Board]]
		+	|[[File:CmpeE244_S18_Eagle_board_design.png|500px|thumb|center|Board Design]]
		+	|}
		+
	=== Software Design ===		=== Software Design ===
−	Show your software design.  For example, if you are designing an MP3 Player, show the tasks that you are using, and what they are doing at a high level.  Do not show the details of the code.  For example, do not show exact code, but you may show psuedocode and fragments of code.  Keep in mind that you are showing DESIGN of your software, not the inner workings of it.
−	=== Implementation ===	+	==== Display Node ====
−	This section includes implementation, but again, not the details, just the high level. For example, you can list the steps it takes to communicate over a sensor, or the steps needed to write a page of memory onto SPI Flash. You can include sub-sections for each of your component implementation.	+
		+	===== RGB LED Matrix =====
		+
		+
		+	[[File:CMPE244_S18_LedMatrixFlowchart.png | 600px | thumb | left| '''Software Flowchart for the LED Matrix Display''']]
		+	[[File:class_diagram_display_node.png|450px|thumb|center|'''Display Node Class Diagram''']]
		+
		+
		+	This 16x32 5V,2A supply RGB LED Matrix panels from Adafruit require 12 digital pins (6-bit data, 6-bit control). They have 512 bright RGB LEDs arranged in a 16x32 grid on the front. These displays are 'chainable' - connect the output to the next input. This display requires about 800 bytes of RAM to buffer the 12-bit color image. As the matrix panel has a 6-bit parallel interface, SPI hardware won't help and hence simple GPIO interface has been used to communicate with the RGB LED Matrix. As these displays don't have built-in PWM control of any kind, the driver is supposed to redraw the screen over and over to 'manually' PWM the whole thing.
		+
		+	Pins R1, G1, and B1 deliver data to the top half of the display whereas R2, G2, and B2 deliver data to the bottom half of the display. Pins A, B, C, and D select which two rows of the display are currently lit. The LAT signal acts as the strobe and marks the end of a row of data. The CLK signal marks the arrival of each bit of data. OE (output enable) is an active low signal that switches the LEDs off when transitioning from one row to the next.
		+
		+	The implementation consists of 768-byte buffer for storing the pixel values for all LEDs on the display. Every line is scanned and the values for the pixels are updated every 1 ms. Between every transition of the clock from low to high, the values for the pixels are updated. As the screen is refreshed and updated very frequently, it gives the impression of a moving image.
		+
		+	The basic process used to refresh the display when using three bits-per-pixel color (one bit for red; one bit for green; and one bit for blue) is the following:
		+
		+	*Shift the pixel data for row 0 into the top column drivers and the pixel data for row 16 into the bottom column drivers using the R0, G0, B0, R1, G1, and B1 data inputs and the SCLK shift clock signal.
		+	*Assert the blanking signal to blank the display.
		+	*Set the address input to 0.
		+	*Latch the contents of the column drivers‘ shift registers into the column drivers‘ output registers using the LATCH signal.
		+	*Deassert the blanking signal to display rows 0 and 16.
		+	*Wait some fixed amount of time.
		+	*Repeat the process for each of the pairs of rows in the display.
		+	*Repeat the entire process at least 100 to 200 times per second to prevent flicker.
		+
		+	==== Hand-gesture Control Logic ====
		+
		+
		+
		+
		+
		+
		+	[[File:player1_class_diagram.png|325px|thumb|right|'''Hand-gesture Control Board Class Diagram''']]
		+
		+
		+	[[File:CmpE244_S18_AccSensor_flowchart.PNG|900px|thumb|center|'''Flowchart for Hand-gesture Module''']]
		+
		+
		+
		+
		+
		+
		+	[[File:CmpE244_S18_AccValue.PNG|500px|thumb|right|'''Accelerometer Value Calculations''']]
		+
		+
		+	The gesture controlled process which acts as an interface between the players and the display node is explained as follows:
		+	*Hand gesture module begins with the initialization of the onboard accelerometer sensor which is interfaced using I2C bus to the controller.
		+	*The Start condition is satisfied as soon as both the players press the corresponding switches on the respective boards as a sign of being ready.
		+	*Switch press is acknowledged and the required start code for the start screen is transmitted. The start screen displays a countdown from 3 to 1 to give a dramatic effect.
		+	*After the start screen is displayed, the values are received from the onboard accelerometer sensor and transmitted to the Master Node i.e. the RGB LED Matrix.
		+
		+	===== Accelerometer Value Calculation =====
		+
		+	After carefully looking at the board design and how the board will be placed on a person's hand, we found out that the values received along one axis(X-axis) will be enough to capture the wrist movement.
		+	The range of accelerometer values shows how sensitive the accelerometer sensor is. We wanted the code to be flexible in terms of the range of the values. So that we did not have to hard code the sensor values and instead used a code based on maximum and minimum range values which could be easily configurable.
		+	The piece of code mentioned below shows how the values are calculated.
		+
		+	==== Wireless Module ====
		+
		+	[[File:Wireless_Interface.PNG|400px|thumb|center|Wireless Communication Between the Boards]]
		+
		+	We referred '''[http://socialledge.com/sjsu/index.php/Low_Powered_Mesh_Network_stack Low Powered Mesh Network stack]''' for building wireless mesh network between the display board and player boards.
		+
		+	===== Wireless Receiver =====
		+
		+	The SJ-One board connected to LED display uses onboard Nordic wireless to communicate with player boards. Initially, it waits for players to press the start button on their controllers which then sends commands to the receiver board. Once it receives the commands to start, it displays green signal on the display and changes the screen to game screen. While the game is running, receiver continuously receives the acceleration values at every 50 ms from player boards which it then sends further to the slider task.
		+
		+	===== Wireless Transmission =====
		+
		+	We used Nordic Wireless for transmitting the accelerometer data. Both the player 1 and the player 2 boards send the accelerometer data to the Display node. We created a remote wireless task which runs on medium priority and sends an integer value to the Display node at every 100 ms.
		+	Both the boards have a unique node address and a node name. For the mesh network, we had to use the same channel number and data transmission rate for both the player boards and Display node.
		+
		+	=== Game logic and Implementation ===
		+
		+
		+
		+
		+
		+
		+	[[File:Game_Flow_Chart.PNG|450px|thumb|center|'''Game Flowchart''']]
		+
		+
		+	The game implementation involves creating multiple tasks for control of independent entities and also making those entities work together.
		+
		+	The game starts off with the start screen task, which displays the game title and status for both players. As soon as both the players have pressed the button1 (to indicate that the player is ready), after a 3 second counter, the ball and slider tasks are resumed. Initially, all the tasks are suspended except the start screen task. Slider task takes the input acceleration sensor values from both the players and translates the values into appropriate movement on the display whereas the ball tasks move the ball on the display. If any of the player miss to hit the ball, the player loses and the score display task will be displayed and all other tasks would be suspended. After displaying the score for 3 seconds, we reset the ball position to randomly start from the center of the screen (4 possible directions) and start the round again. A player wins if he wins 9 rounds, and the score display screen then displays the winner along with the scores. As the game is over, the game will be restarted by suspending all tasks and resuming the start screen task again.
		+
		+	Remote wireless task retrieves the slider values from both the players over Nordic wireless transceiver and passes it to the slider task for producing the hand-gesture movement on to the display. To support the wireless communication, background Nordic wireless task participates in the mesh network and handles retry logic such that packets are resent if an ACK has not been received.
		+
		+	== Testing ==
		+	=== Module testing ===
−	== Testing & Technical Challenges ==	+	==== Wireless Module ====
−	Describe the challenges of your project. What advise would you give yourself or someone else if your project can be started from scratch again?	+	We started to test the basic wireless communication between two boards. Initially, we sent a single data packet from one board to the other. Once the receiver board receives the packet, it prints the content with the address of the transmitting node.
−	Make a smooth transition to testing section and described what it took to test your project.
−	Include sub-sections that list out a problem and solution, such as:	+	After this, we tested with 3 boards. This time we had 2 transmitters and 1 receiver. We had to set different addresses for each node to establish mesh communication between all 3 nodes. We were sending one packet from each transmitter. The receiver board receives both the packets and prints the content of the packet with the address of the transmitting node. In this way we verified that the wireless communication between the boards is working successfully.
−	=== <Bug/issue name> ===	+	==== Display Node ====
−	Discuss the issue and resolution.	+
		+	At first, we implemented the slider movement logic and tested the slider movement on the display. This testing helped us decide the size of the slider and make sure that when the slider is moved using the accelerometer, the change in the position is perfectly reflected on the LED Matrix
		+
		+	We wanted to make sure, whenever the ball touches the horizontal boundaries of the LED matrix it bounces back with the proper angle. And if the ball touches the vertical boundaries of the matrix, the matrix should display the score and start the game again. To ensure the correctness of the ball movement, we implemented the ball movement logic and tested it independently on the LED Matrix.
		+
		+	Also, we designed the Start screen and Game End screen and tested them on the LED display. While testing the screens we focused on having proper text font and placement of the text on the display.
		+
		+	{|
		+	|[[File:CmpE244_S18_LED_Matrix_Start_Screen.png|300px|thumb|left|'''Start Screen''']]
		+	|[[File:CmpeE244_S18_Slider_Movement.gif|300px|thumb|'''Slider Movement Testing''']]
		+	|[[File:CmpeE244_S18_Ball_Movement.gif|300px|thumb|right|'''Ball Movement Testing''']]
		+	|[[File:CmpE244_S18_LED_Matrix_Game_End_Screen.png|300px|thumb|right|'''Game End Screeen''']]
		+	|}
		+
		+	==== Hand-gesture Control Module ====
		+	Hand-gesture recognition was rigorously tested. The values received from accelerometer sensor were within a range. So, we had to test the range of values received at which the slider on screen looked responsive and smooth. We tested values ranging from -1000 to +1000 to find the right range of values. The hand-gesture module transmits uint8_t. So, we checked whether the output at the minimum and maximum input values were within valid range. We also checked the uint8_t value(index) transmitted depending upon the inputs received from the accelerometer sensor.
		+
		+	==== PCB Testing ====
		+
		+	The PCB was tested by checking the connectivity of the path on the copper clad tracks using a Digital Multimeter.
		+	The circuit on the PCB was simulated using MultiSim v14.1 Software and results were verified on the actual board.
		+	All the connector-related bugs were fixed by inspection and drilling the PCB wherever necessary.
		+	The board was tested for any short circuit by checking the connectivity between the Supply tracks and common ground tracks.
		+
		+	=== Integration testing ===
		+	After testing the individual modules, we decided to integrate display node and left-hand control board first and test it. Once we were able to control left slider satisfactorily, we integrated the right-hand control board with the display node. This methodology helped us identify that there was a time lag in receiving the accelerometer data from the hand-gesture controller boards.
		+	After testing the display nodes with the hand control boards, we decided to integrate the ball movement logic with them. This approach helped us in deciding the suitable speed of the ball.
		+
		+	== Technical Challenges ==
		+
		+	=== Display Driver Implementation ===
		+
		+	We initially started to mimic the Adafruit library, but the library was originally designed to work with Arduino (AVR), and it consisted of some assembly language instructions to save CPU cycle. Hence, we were unsuccessful in porting the library entirely. We read more about the RGB LED matrix and decided to get control over a single row first and then over a single pixel. We first implemented a function to first select the row according to A, B, and C signals and write the corresponding pixel values of all LEDs in those rows with the updated values from the frame buffer. The values of the pixels were set only within the clock trigger. In the end, the pixel values were latched before the output enable signal was asserted again. This function was called from a periodic task with the 1 millisecond delay in order to get the refresh rate of the display as 1 millisecond. This helped us to get the control over individual pixels of the matrix.
		+
		+	=== RGB LED Matrix Flickering Issue ===
		+
		+	We need to refresh the LED matrix to display the new data. Earlier we were refreshing the matrix after every 100 milliseconds. Because the refresh rate was low we saw there was some flickering in the matrix. So, we decided to increase the refresh rate and kept it at 1 milliseconds. This refresh rate is very small for the human eyes to detect, the flickering in the matrix stopped appearing.
		+
		+	=== Time lag between reading the accelerometer data and displaying it on the LED matrix ===
		+
		+	[[File:CmpE244_S18_LookupTable.PNG|500px|thumb|right|Lookup Table For Drawing A Line]]
		+	There was a noticeable lag between the detection of wrist movement and the time at which it was displayed on the RGB LED Matrix. In the first code, we had used strings for transmitting data from player nodes to control node. We were transmitting actual values. These two things were the reason for the lag.
		+	So, we made the following changes:
		+
		+	1] We transmitted uint8_t values from player nodes to control nodes which reduced the int to string conversion at both the ends.
		+
		+	2] We had to draw lines for the slider movement for both the players on the RGB LED matrix. Drawing a line depends on two points, so we needed the start index and the end index. The calculation of these indexes was done on the control node side. Instead of performing operations on raw data at the control node side, we performed operations on the player node side and transmitted an uint8_t value to the control node. The control node has a two-dimensional lookup table which contains the start and the end index for drawing a line. This two changes reduced the lag considerably.
		+
		+	=== PCB Designing ===
		+
		+	The toughest challenge in designing the PCB Layout and Schematic was to select the exact Pin Header Libraries of the accurate measurement of a 3.5mm spacing.
		+	This problem especially occurred for the power connectors the project needed for powering up the RGB LED matrix as the connector is slightly unconventional compared to the normal 2mm spacing.
		+	To suffice this need, a new library was created in Eagle v8.2.1 Software by taking a reference to an existing library which has the same spacing between the pins on the header.
	== Conclusion ==		== Conclusion ==
−	Conclude your project here. You can recap your testing and problems. You should address the "so what" part here to indicate what you ultimately learnt from this project. How has this project increased your knowledge?	+	The project was a success both in terms of output and learnings. We were able to build a multiplayer ping-pong game which is controlled wirelessly using hand gestures. It provided us an exposure of working with RGB LED matrix, Nordic wireless chip, and accelerometer. We learned to write a driver for an RGB LED matrix. It helped us utilize the knowledge of FreeRTOS which we gained through the CMPE-244 class. We were able to give an overall compact look to the display unit and hand gesture control board to make it a complete gaming console product. Also, using Git for version control helped us keep the track of individual's work progress and it became easier for us to share the code with the team.
		+
		+	Working as a team and conducting the brainstorming sessions helped us in coming up with some good ideas. Also, it helped in expediting the integration testing and debugging of the project during the final days. Overall, we got an experience of working in a team and developing a complete product using professional tools and methodologies.
		+
	=== Project Video ===		=== Project Video ===
−	Upload a video of your project and post the link here.	+	* [https://www.youtube.com/watch?v=-Gn6-So41Qk Hand-gesture Controlled Ping Pong Game]
	=== Project Source Code ===		=== Project Source Code ===
−	*  [https://gitlab.com/suicide_squad/hand-gesture-controlled-multiplayer-game]	+	*  [https://gitlab.com/suicide_squad/hand-gesture-controlled-multiplayer-game Github Source Code Link]
	== References ==		== References ==
	=== Acknowledgement ===		=== Acknowledgement ===
−	Any acknowledgement that you may wish to provide can be included here.	+	We would like to thank our Professor Preetpal Kang for all his teachings and inspirational lectures. Not only did we enjoy working though out this project but also gave us an overall learning experience and precious life lessons. We would also like to thank the ISA members for always being ready to help with whatever issues we faced.
	=== References Used ===		=== References Used ===
−	List any references used in project.	+	[1] [http://www.socialledge.com/sjsu/index.php?title=Realtime_OS_on_Embedded_Systems FreeRTOS documentations]
		+
		+	[2] [https://github.com/adafruit/Adafruit-GFX-Library Adafruit LCD library]
		+
		+	[3] CMPE 244 Lecture notes from Preetpal Kang, Computer Engineering, San Jose State University. Feb - May 2018.
		+
		+	[4] [https://www.sparkfun.com/datasheets/Components/SMD/nRF24L01Pluss_Preliminary_Product_Specification_v1_0.pdf Nordic wireless datasheet]
−	=== Appendix ===	+	[5] [https://github.com/adafruit/Adafruit-GFX-Library Adafruit GFX Library]
−	You can list the references you used.

---

Latest revision as of 09:52, 27 May 2018

Project Title

Hand-gesture controlled multiplayer game

Abstract

Human-machine interfaces are constantly evolving. The interfaces used in HMI, based on natural gestures of users, allow converting these movements into commands for a computer system. In this project, we developed a multiplayer Ping-Pong game using a 16x32 RGB LED Matrix, accelerometer, wireless connectivity, FreeRTOS APIs and SJOne boards. The game works as a classic ping pong game in which there are two sliders which are controlled wirelessly by two hand-gesture controllers. If the ball hits any of the vertical boundaries of the matrix then the player loses the game. Each hand-gesture controller has an accelerometer sensor interfaced to SJOne board and it transmits data wirelessly to display node. The display node has the LED matrix interfaced to SJOne board and it receives the commands wirelessly from the hand-gesture controllers. A wireless mesh network is created between the display node and the hand-gesture boards to send accelerometer values. We have designed three screens for the game: Start screen, Game screen, and Result screen. This project helps us understand the communication and synchronization of FreeRTOS tasks in an interesting way.

Objectives & Introduction

Objective

This project involves learning what goes behind developing a 2D Multiplayer game. Translating the onboard acceleration sensor in a way to be meaningful and measurable for the display node to display the slider movement from the two players over the onboard wireless Nordic transceiver.

Introduction

The overall architecture of the design includes a display node, two hand-gesture control modules, and wireless communication module.

Display Node: It has a 16x32 LED Matrix interfaced to SJOne board. It is responsible for driving the LED matrix

Hand-gesture control module: The two hand-gesture control boards are used to control the game. They have SJOne board with an onboard accelerometer. The microcontroller in the SJOne board continuously reads the data from the accelerometer and sends signals wirelessly to the display node

Wireless module: It is used as a communication medium between the display node and the hand-gesture controlled boards`

Team Members & Responsibilities

• Akil Khan
  • Design and development of the RGB Led Matrix driver
  • Game logic development, implementation, QA, and testing.
• Abhilash Tuse
  • Game logic development
  • Logic Development for communication between hand-gesture control board and controller board
• Disha Patil
  • Hand-gesture control logic development using an accelerometer
  • Design of PCB using Eagle for Power Supply
• Omkar Kale
  • Hand-gesture control logic development using an accelerometer
  • Logic Development for communication between hand-gesture control board and controller board
• Vishal Shrivastava
  • Design and development of the RGB Led Matrix driver
  • Design of PCB using Eagle for Power Supply

Schedule

Week#	Date	Task	Status	Actual Completion Date
1	04/10/2018	Submission of Project Proposals Research on project requirements. Order components and distribute project modules.	Completed Completed Completed	03/24/2018 04/10/2018 04/10/2018
2	04/17/2018	Developing logic for hand-gesture control using accelerometer sensor. Power Supply circuit design and simulation using Multisim Write basic LED display driver to blink individual pixels and set of pixels. Establishing basic wireless communication between 2 SJOne boards.	Completed Completed Completed Completed	04/15/2018 04/16/2018 04/17/2018 04/18/2018
3	04/24/2018	Develop advanced API's on top of LED display driver: Draw game objects such as sliders and ball. PCB layout design using Eagle and finalizing the schematic. Establish communication between the hand-gesture controller and display node using wireless mesh module. Project report update on the wiki.	Completed Completed Completed Completed	05/01/2018 04/29/2018 04/24/2018 05/23/2018
4	05/01/2018	Developing logic for the ball movement and translating hand-gesture control into slider movement. Soldering components and hardware testing on PCB Packaging of hardware board and related components. Project report update on the wiki.	Completed Completed Completed Completed	05/01/2018 05/19/2018 05/09/2018 05/19/2018
5	05/08/2018	Integrate ball movement and slider movement logic into display node algorithm. Project report update on the wiki.	Completed Completed	05/05/2018 05/19/2018
6	04/15/2018	Final testing and bug fixes. Complete wiki report and final demo.	Completed Completed	05/23/2018 05/23/2018

Parts List & Cost

Item#	Part	Manufacturer	Quantity	Cost($)
1	SJ One Board	Preet	3	80.00
2	Adafruit RGB LED Matrix	LED Matrix	1	35.00
3	Power Adapter	Power Supply	1	7.95
4	Power Bank 800mah	Aibocn	2	11.95
5	PCB Board	PCBway	1	23.00
6	Miscellaneous (Barrel Jack, Jumper wires, Connectors)	Excess Solution		5.00

• Total Cost:334.85$

Design & Implementation

Hardware Design

Hardware design diagram gives an overview of the entire system which consists of the main 3 SJ-One controllers: 2 player boards and 1 board as the display node

• Player boards use the onboard accelerometer on SJ-One which is interfaced using the I2C protocol. Each of the player boards read these accelerometer values and determine the position of the slider on the display.
• As we are implementing a wireless multiplayer game, the accelerometer values are transmitted using yet another onboard module, the Nordic wireless module which interfaces with the wireless module on the third controller used for the display.
• The Display node SJ-One board is used to control a 16*32 RGB LED Matrix. This matrix displays 2 sliders and a ball representing the players in a Ping-Pong game. Slider locations are decided by the values obtained wirelessly from Player boards.
• RGB LED Matrix displays three screens: Start Screen, Game Screen, End Screen which displays the final score of the players and the winner as well. The hardware description of the LED matrix is explained later in this section.

Display Node

RGB LED Matrix

The RGB LED matrix panels are normally used to make video walls to display animations or short video clips. It has 512 bright RGB LEDs arranged in a 16x32 grid. These panels require 12 digital pins (6-bit data, 6-bit control) and a good 5V power supply, at least 2 amps per panel. A single 32x16 RGB matrix, running full tilt (all pixels set white), can require nearly 4 Amps of current.

All 512 LEDs can’t be driven at once. One reason is that would require a lot of current, another reason is that it would be expensive to have so many pins. Instead, the matrix is divided into 8 interleaved sections/strips. The first section is the 1st 'line' and the 9th 'line' (32 x 2 RGB LEDs = 64 RGB LEDs), the second is the 2nd and 10th line, etc until the last section which is the 7th and 16th line. The reason it is done in this way is so that the lines are interleaved and look better when refreshed, otherwise, we'd see the stripes more clearly. So, on the PCB is 12 LED driver chips. These are like 74HC595s but they have 16 outputs and they require constant current. 16 outputs * 12 chips = 192 LEDs that can be controlled at once, and 64 * 3 (R G and B) = 192. There are 192 outputs that can control one line at a time, with each of 192 R, G and B LEDs either on or off. The controller selects which section to draw (using A, B, and C address pins - 3 bits can have 8 values). Once the address is set, the controller clocks out 192 bits of data (24 bytes) and latches it. Then it increments the address and clocks out another 192 bits, etc until it gets to address #7, then it sets the address back to #0.

We are driving the matrix using 12 GPIO pins of SJOne board.

Pin mapping of RGB LED matrix with SJ One board:

RGB LED matrix Pins	SJ One Board Pins	Function
R1	P1.19	Red Data top half
G1	P1.20	Green Data top half
B1	P1.22	Blue Data top half
R2	P1.23	Red Data bottom half
G2	P1.28	Green Data bottom half
B2	P1.29	Blue Data bottom half
A	P2.0	Row select A
B	P2.1	Row select B
C	P2.2	Row select C
CLK	P0.1	The CLK (clock) signal marks the arrival of each bit of data.
OE	P0.0	OE (output enable) switches the LEDs off when transitioning from one row to the next.
LAT	P0.26	The LAT (latch) signal marks the end of a row of data.

Hand-gesture control module

Accelerometer

The SJ-One board has an accelerometer which is interfaced on the I2C bus. Accelerometers are electromechanical devices that sense either static or dynamic forces of acceleration. Static forces include gravity, while dynamic forces can include vibrations and movement. The measurements are on 3-axis and these values can be calibrated to find the desired values.

In our project, an accelerometer is used for detecting the up-down movement of the wrist. We are using accelerometers on two SJ-One Boards for gesture recognition. The movement from accelerometer sensor will be used to control the slider movement on the RGB LED Matrix.

Accelerometer on board

Accelerometer Intefaced With Microcontroller

Accelerometer Detection

Wireless Module

Nordic wireless chip nRF24L01+ is interfaced with SJone board using SPI bus protocol. The nRF24L01+ is a single chip 2.4GHz transceiver with an embedded baseband protocol engine, suitable for ultra low power wireless applications. The nRF24L01+ is designed for operation in the frequency band of 2.400 - 2.483GHz. The high air data rate combined with two power saving modes make the nRF24L01+ very suitable for ultra-low power designs. The nRF24L01 integrates a complete 2.4GHz RF transceiver, RF synthesizer, and baseband logic including the Enhanced ShockBurst™ hardware protocol accelerator supporting a high-speed SPI interface for the application controller. No external loop filter, resonators, or VCO varactor diodes are required, only a low-cost ±60ppm crystal, matching circuitry, and antenna.

Nordic Wireless Chip nRF24L01+

Nordic Wireless Chip Interface with Microcontroller

Printed Circuit Board(PCB) Design

The PCB Layout is designed using the Eagle Software v8.7.1. Board consists of 2 power supplies which are powering:

• SJone board (5V, 500mA)
• RGB LED Matrix (5V,2A)

The Power Supply circuit has an LM317 voltage regulator IC and a voltage divider to fulfill the specific power requirements. LM317 is a TI (Texas Instruments) based IC which has an Output Voltage Range Adjustable from 1.25 V to 37 V and is suitable for our application. We have used a 12V adapter in order to power our board. This serves for both the current requirements. The circuit was simulated using MultiSim v14.1 software by NI (National Instruments). The simulation helped us understand the working of our circuit before we built and tested it.

Printed Circuit Board also has provisions for the connection of the SJone pin headers and RGB LED Matrix's connector. The connections are as shown in the hardware description diagram. The PCB also has a power switch and LED for indication. The values for the resistors and other active or passive components are as shown in the eagle schematic diagram.

Printed Circuit Board

Board Design

Software Design

Display Node

RGB LED Matrix

This 16x32 5V,2A supply RGB LED Matrix panels from Adafruit require 12 digital pins (6-bit data, 6-bit control). They have 512 bright RGB LEDs arranged in a 16x32 grid on the front. These displays are 'chainable' - connect the output to the next input. This display requires about 800 bytes of RAM to buffer the 12-bit color image. As the matrix panel has a 6-bit parallel interface, SPI hardware won't help and hence simple GPIO interface has been used to communicate with the RGB LED Matrix. As these displays don't have built-in PWM control of any kind, the driver is supposed to redraw the screen over and over to 'manually' PWM the whole thing.

Pins R1, G1, and B1 deliver data to the top half of the display whereas R2, G2, and B2 deliver data to the bottom half of the display. Pins A, B, C, and D select which two rows of the display are currently lit. The LAT signal acts as the strobe and marks the end of a row of data. The CLK signal marks the arrival of each bit of data. OE (output enable) is an active low signal that switches the LEDs off when transitioning from one row to the next.

The implementation consists of 768-byte buffer for storing the pixel values for all LEDs on the display. Every line is scanned and the values for the pixels are updated every 1 ms. Between every transition of the clock from low to high, the values for the pixels are updated. As the screen is refreshed and updated very frequently, it gives the impression of a moving image.

The basic process used to refresh the display when using three bits-per-pixel color (one bit for red; one bit for green; and one bit for blue) is the following:

• Shift the pixel data for row 0 into the top column drivers and the pixel data for row 16 into the bottom column drivers using the R0, G0, B0, R1, G1, and B1 data inputs and the SCLK shift clock signal.
• Assert the blanking signal to blank the display.
• Set the address input to 0.
• Latch the contents of the column drivers‘ shift registers into the column drivers‘ output registers using the LATCH signal.
• Deassert the blanking signal to display rows 0 and 16.
• Wait some fixed amount of time.
• Repeat the process for each of the pairs of rows in the display.
• Repeat the entire process at least 100 to 200 times per second to prevent flicker.

Hand-gesture Control Logic

The gesture controlled process which acts as an interface between the players and the display node is explained as follows:

• Hand gesture module begins with the initialization of the onboard accelerometer sensor which is interfaced using I2C bus to the controller.
• The Start condition is satisfied as soon as both the players press the corresponding switches on the respective boards as a sign of being ready.
• Switch press is acknowledged and the required start code for the start screen is transmitted. The start screen displays a countdown from 3 to 1 to give a dramatic effect.
• After the start screen is displayed, the values are received from the onboard accelerometer sensor and transmitted to the Master Node i.e. the RGB LED Matrix.

Accelerometer Value Calculation

After carefully looking at the board design and how the board will be placed on a person's hand, we found out that the values received along one axis(X-axis) will be enough to capture the wrist movement. The range of accelerometer values shows how sensitive the accelerometer sensor is. We wanted the code to be flexible in terms of the range of the values. So that we did not have to hard code the sensor values and instead used a code based on maximum and minimum range values which could be easily configurable. The piece of code mentioned below shows how the values are calculated.

Wireless Module

We referred Low Powered Mesh Network stack for building wireless mesh network between the display board and player boards.

Wireless Receiver

The SJ-One board connected to LED display uses onboard Nordic wireless to communicate with player boards. Initially, it waits for players to press the start button on their controllers which then sends commands to the receiver board. Once it receives the commands to start, it displays green signal on the display and changes the screen to game screen. While the game is running, receiver continuously receives the acceleration values at every 50 ms from player boards which it then sends further to the slider task.

Wireless Transmission

We used Nordic Wireless for transmitting the accelerometer data. Both the player 1 and the player 2 boards send the accelerometer data to the Display node. We created a remote wireless task which runs on medium priority and sends an integer value to the Display node at every 100 ms. Both the boards have a unique node address and a node name. For the mesh network, we had to use the same channel number and data transmission rate for both the player boards and Display node.

Game logic and Implementation

The game implementation involves creating multiple tasks for control of independent entities and also making those entities work together.

The game starts off with the start screen task, which displays the game title and status for both players. As soon as both the players have pressed the button1 (to indicate that the player is ready), after a 3 second counter, the ball and slider tasks are resumed. Initially, all the tasks are suspended except the start screen task. Slider task takes the input acceleration sensor values from both the players and translates the values into appropriate movement on the display whereas the ball tasks move the ball on the display. If any of the player miss to hit the ball, the player loses and the score display task will be displayed and all other tasks would be suspended. After displaying the score for 3 seconds, we reset the ball position to randomly start from the center of the screen (4 possible directions) and start the round again. A player wins if he wins 9 rounds, and the score display screen then displays the winner along with the scores. As the game is over, the game will be restarted by suspending all tasks and resuming the start screen task again.

Remote wireless task retrieves the slider values from both the players over Nordic wireless transceiver and passes it to the slider task for producing the hand-gesture movement on to the display. To support the wireless communication, background Nordic wireless task participates in the mesh network and handles retry logic such that packets are resent if an ACK has not been received.

Testing

Module testing

Wireless Module

We started to test the basic wireless communication between two boards. Initially, we sent a single data packet from one board to the other. Once the receiver board receives the packet, it prints the content with the address of the transmitting node.

After this, we tested with 3 boards. This time we had 2 transmitters and 1 receiver. We had to set different addresses for each node to establish mesh communication between all 3 nodes. We were sending one packet from each transmitter. The receiver board receives both the packets and prints the content of the packet with the address of the transmitting node. In this way we verified that the wireless communication between the boards is working successfully.

Display Node

At first, we implemented the slider movement logic and tested the slider movement on the display. This testing helped us decide the size of the slider and make sure that when the slider is moved using the accelerometer, the change in the position is perfectly reflected on the LED Matrix

We wanted to make sure, whenever the ball touches the horizontal boundaries of the LED matrix it bounces back with the proper angle. And if the ball touches the vertical boundaries of the matrix, the matrix should display the score and start the game again. To ensure the correctness of the ball movement, we implemented the ball movement logic and tested it independently on the LED Matrix.

Also, we designed the Start screen and Game End screen and tested them on the LED display. While testing the screens we focused on having proper text font and placement of the text on the display.

Start Screen

Slider Movement Testing

Ball Movement Testing

Game End Screeen

Hand-gesture Control Module

Hand-gesture recognition was rigorously tested. The values received from accelerometer sensor were within a range. So, we had to test the range of values received at which the slider on screen looked responsive and smooth. We tested values ranging from -1000 to +1000 to find the right range of values. The hand-gesture module transmits uint8_t. So, we checked whether the output at the minimum and maximum input values were within valid range. We also checked the uint8_t value(index) transmitted depending upon the inputs received from the accelerometer sensor.

PCB Testing

The PCB was tested by checking the connectivity of the path on the copper clad tracks using a Digital Multimeter. The circuit on the PCB was simulated using MultiSim v14.1 Software and results were verified on the actual board. All the connector-related bugs were fixed by inspection and drilling the PCB wherever necessary. The board was tested for any short circuit by checking the connectivity between the Supply tracks and common ground tracks.

Integration testing

After testing the individual modules, we decided to integrate display node and left-hand control board first and test it. Once we were able to control left slider satisfactorily, we integrated the right-hand control board with the display node. This methodology helped us identify that there was a time lag in receiving the accelerometer data from the hand-gesture controller boards. After testing the display nodes with the hand control boards, we decided to integrate the ball movement logic with them. This approach helped us in deciding the suitable speed of the ball.

Technical Challenges

Display Driver Implementation

We initially started to mimic the Adafruit library, but the library was originally designed to work with Arduino (AVR), and it consisted of some assembly language instructions to save CPU cycle. Hence, we were unsuccessful in porting the library entirely. We read more about the RGB LED matrix and decided to get control over a single row first and then over a single pixel. We first implemented a function to first select the row according to A, B, and C signals and write the corresponding pixel values of all LEDs in those rows with the updated values from the frame buffer. The values of the pixels were set only within the clock trigger. In the end, the pixel values were latched before the output enable signal was asserted again. This function was called from a periodic task with the 1 millisecond delay in order to get the refresh rate of the display as 1 millisecond. This helped us to get the control over individual pixels of the matrix.

RGB LED Matrix Flickering Issue

We need to refresh the LED matrix to display the new data. Earlier we were refreshing the matrix after every 100 milliseconds. Because the refresh rate was low we saw there was some flickering in the matrix. So, we decided to increase the refresh rate and kept it at 1 milliseconds. This refresh rate is very small for the human eyes to detect, the flickering in the matrix stopped appearing.

Time lag between reading the accelerometer data and displaying it on the LED matrix

There was a noticeable lag between the detection of wrist movement and the time at which it was displayed on the RGB LED Matrix. In the first code, we had used strings for transmitting data from player nodes to control node. We were transmitting actual values. These two things were the reason for the lag. So, we made the following changes:

1] We transmitted uint8_t values from player nodes to control nodes which reduced the int to string conversion at both the ends.

2] We had to draw lines for the slider movement for both the players on the RGB LED matrix. Drawing a line depends on two points, so we needed the start index and the end index. The calculation of these indexes was done on the control node side. Instead of performing operations on raw data at the control node side, we performed operations on the player node side and transmitted an uint8_t value to the control node. The control node has a two-dimensional lookup table which contains the start and the end index for drawing a line. This two changes reduced the lag considerably.

PCB Designing

The toughest challenge in designing the PCB Layout and Schematic was to select the exact Pin Header Libraries of the accurate measurement of a 3.5mm spacing. This problem especially occurred for the power connectors the project needed for powering up the RGB LED matrix as the connector is slightly unconventional compared to the normal 2mm spacing. To suffice this need, a new library was created in Eagle v8.2.1 Software by taking a reference to an existing library which has the same spacing between the pins on the header.

Conclusion

The project was a success both in terms of output and learnings. We were able to build a multiplayer ping-pong game which is controlled wirelessly using hand gestures. It provided us an exposure of working with RGB LED matrix, Nordic wireless chip, and accelerometer. We learned to write a driver for an RGB LED matrix. It helped us utilize the knowledge of FreeRTOS which we gained through the CMPE-244 class. We were able to give an overall compact look to the display unit and hand gesture control board to make it a complete gaming console product. Also, using Git for version control helped us keep the track of individual's work progress and it became easier for us to share the code with the team.

Working as a team and conducting the brainstorming sessions helped us in coming up with some good ideas. Also, it helped in expediting the integration testing and debugging of the project during the final days. Overall, we got an experience of working in a team and developing a complete product using professional tools and methodologies.

Project Video

• Hand-gesture Controlled Ping Pong Game

Project Source Code

• Github Source Code Link

References

Acknowledgement

We would like to thank our Professor Preetpal Kang for all his teachings and inspirational lectures. Not only did we enjoy working though out this project but also gave us an overall learning experience and precious life lessons. We would also like to thank the ISA members for always being ready to help with whatever issues we faced.

References Used

[1] FreeRTOS documentations

[2] Adafruit LCD library

[3] CMPE 244 Lecture notes from Preetpal Kang, Computer Engineering, San Jose State University. Feb - May 2018.

[4] Nordic wireless datasheet

[5] Adafruit GFX Library