Difference between revisions of "S18: Smart Rock Paper Scissors"

From Embedded Systems Learning Academy
Jump to: navigation, search
(Bluetooth Gloves)
 
(173 intermediate revisions by the same user not shown)
Line 1: Line 1:
=== Grading Criteria ===
+
== Smart Rock Paper Scissors ==
<font color="green">
+
Hand gesture controlled rock paper scissors game.
*  How well is Software & Hardware Design described?
 
*  How well can this report be used to reproduce this project?
 
*  Code Quality
 
*  Overall Report Quality:
 
**  Software Block Diagrams
 
**  Hardware Block Diagrams
 
**:  Schematic Quality
 
**  Quality of technical challenges and solutions adopted.
 
</font>
 
  
== Project Title ==
+
[[File:CmpE244 S18 Cumulonimbus SystemPicture.PNG]]
Hand gesture controlled rock paper scissors game
+
 
 +
'''Figure 1: The Smart Rock Paper Scissors System'''
  
 
== Abstract ==
 
== Abstract ==
This section should be a couple lines to describe what your project does.
+
This project recreates the classic Rock,Paper, Scissors game using a LED Matrix to display and keep track of scoring, and wireless gloves to record the user input.  The LED Matrix helps users to know what round they are on, what the score is, as well as provide prompts of when to give the Rock, Paper or Scissors signal.  The Smart aspect of the system also allows users to play against a computer player.  The wireless gloves utilize Bluetooth modules and flex sensors to detect the player input and feed that input into the game logic and game controller.
  
 
== Objectives & Introduction ==
 
== Objectives & Introduction ==
Show list of your objectives.  This section includes the high level details of your project.  You can write about the various sensors or peripherals you used to get your project completed.
+
* Create LED display to guide players through the game
 +
* Create wireless gloves to allow players to give input
 +
* Create feedback to the gloves to enhance the game experience
 +
* Make the game platform modular, so other games could be easily built with the system
  
 
=== Team Members & Responsibilities ===
 
=== Team Members & Responsibilities ===
Line 34: Line 29:
 
*  Sarvpreet Singh
 
*  Sarvpreet Singh
 
**LED Display
 
**LED Display
 +
 
== Schedule ==
 
== Schedule ==
Show a simple table or figures that show your scheduled as planned before you started working on the project.  Then in another table column, write down the actual schedule so that readers can see the planned vs. actual goals.  The point of the schedule is for readers to assess how to pace themselves if they are doing a similar project.
 
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 92: Line 87:
  
 
Finish PCB, and submit PCB layout to fab.
 
Finish PCB, and submit PCB layout to fab.
| Completed? Problems Encountered?
+
| Game logic completed. Bluetooth logic working, and basic gloves put together.
 
|-
 
|-
 
! scope="row"| 7
 
! scope="row"| 7
Line 103: Line 98:
  
 
Receive PCB from fab, populate and test PCB.
 
Receive PCB from fab, populate and test PCB.
| Completed?  Problems Encountered?
+
| Completed.
 +
 
 +
PCB received.
 +
 
 +
Glove drivers working and glove is physically put together.
 +
 
 
|-
 
|-
 
! scope="row"| 8
 
! scope="row"| 8
Line 114: Line 114:
  
 
Re-work PCB as needed based on testing.
 
Re-work PCB as needed based on testing.
| Completed?  Problems Encountered?
+
| Completed
 
|-
 
|-
 
! scope="row"| 9
 
! scope="row"| 9
Line 122: Line 122:
 
Prepare project for demo.
 
Prepare project for demo.
  
| Completed?  Problems Encountered?
+
| Completed.
 
|-
 
|-
 
! scope="row"| 10
 
! scope="row"| 10
 
| 05/22
 
| 05/22
 
| Finish debugging project for demo.
 
| Finish debugging project for demo.
| Completed?  Problems Encountered?
+
| Completed.
 
|-
 
|-
 
|}
 
|}
Line 142: Line 142:
 
! scope="row" | 32x32 RGB LED Matrix
 
! scope="row" | 32x32 RGB LED Matrix
 
| 32x32 Matrix to display game
 
| 32x32 Matrix to display game
| https://www.adafruit.com/product/2026
+
| [https://www.adafruit.com/product/2026 Link]
 
| 1
 
| 1
 
| $44.95
 
| $44.95
Line 148: Line 148:
 
! scope="row"| Bluetooth HC-05 Module
 
! scope="row"| Bluetooth HC-05 Module
 
| Bluetooth module with master mode capability
 
| Bluetooth module with master mode capability
| https://www.velleman.eu/products/view/?id=435518
+
| [https://www.velleman.eu/products/view/?id=435518 Link]
 +
| 2
 +
| $29.90
 +
|-
 +
! scope="row"| XBee Bluetooth HC-06 Module
 +
| Gikfun HC-06 Bluetooth Bee V2.0 Slave Module
 +
| [https://www.amazon.com/Gikfun-Bluetooth-Module-Arduino-EK1225/dp/B00RCTWPEQ?SubscriptionId=AKIAILSHYYTFIVPWUY6Q&tag=duckduckgo-d-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=B00RCTWPEQ Link]
 +
| 2
 +
| $23.96
 +
|-
 +
! scope="row"| SJOne Boards
 +
| Microcontrollers
 +
| N/A
 +
| 4
 +
| $320
 +
|-
 +
! scope="row" | Flex Sensors
 +
| Flex sensors to measure finger flex
 +
| [https://www.amazon.com/SPECTRA-SYMBOL-FS-L-0112-103-ST-SYMBOLFLEX-SENSOR/dp/B005T8743E/ref=sr_1_4?ie=UTF8&qid=1523123017&sr=8-4&keywords=flex+sensor Link]
 +
| 4
 +
| $26
 +
|-
 +
! scope="row" | Parallax 28821 Vibration motors
 +
| Feedback motors
 +
| [https://www.amazon.com/VNDEFUL-Vibration-Adhesive-Models-Repair/dp/B073Y51PQN/ref=sr_1_1?ie=UTF8&qid=1523116368&sr=8-1&keywords=vibration+motor+flat+coin Link]
 +
| 2
 +
| $11
 +
|-
 +
! scope="row" | Gloves
 +
| Gloves to record gestures
 +
| [https://www.amazon.com/VNDEFUL-Vibration-Adhesive-Models-Repair/dp/B073Y51PQN/ref=sr_1_1?ie=UTF8&qid=1523116368&sr=8-1&keywords=vibration+motor+flat+coin Link]
 
| 2
 
| 2
| $14.95
+
| $20
 +
|-
 +
! scope="row" | PCB
 +
| PCB to route power
 +
| [http://www.4pcb.com/ Link]
 +
| 3
 +
| $140
 +
|-
 
|}
 
|}
 
  
 
== Design & Implementation ==
 
== Design & Implementation ==
The design section can go over your hardware and software design.  Organize this section using sub-sections that go over your design and implementation.
 
  
 
=== Hardware Design ===
 
=== Hardware Design ===
Discuss your hardware design hereShow detailed schematics, and the interface here.
+
Smart Rock Paper Scissors used a 32x32 LED Display, four Bluetooth modules, and four SJOne boards as the primary hardware devices in the system.  Two SJOne boards were used as Bluetooth receivers, and two SJOne boards were used as Bluetooth transmitters.  One Bluetooth receiver board was also responsible for driving the LED Display and handling the Rock, Paper, Scissors logic.
 +
 
 +
A PCB was created in order to route power from a 5V, 4A AC/DC power supply to the LED display, and both Bluetooth receiver SJOne boards.  The SJOne boards attached to the glove controllers each use USB power banks to power the board.
 +
 
 +
[[File:CMPE244 S18 RPS BlockDiagram.png]]
 +
 
 +
'''Figure 2: Block diagram of the Smart Rock, Paper, Scissors hardware components.'''
 +
 
 +
==== Bluetooth Gloves ====
 +
 
 +
{| style="width:800px;" |
 +
|
 +
[[File:CMPE244 S18 Cumulonimbus FlexSensor.jpg|thumb|120px|right|Flex Sensors]]
 +
[[File:CMPE244 S18 Cumulonimbus BT.jpg|thumb|120px|right|Bluetooth HC-05]]
 +
[[File:CMPE244_S18_Cumulonimbus_VibMotor.jpg|thumb|120px|right|Vibtration Motor]]
 +
 
 +
'''Flex Sensors'''
 +
Flex sensors are used to detect the hand gesture of the user. Two flex sensors are used on fingers 2 and 4, if paper both are not bent, if scissors one is bent and if rock both are bent.
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
<BR/>
 +
''' Bluetooth HC-05 '''
 +
The gloves are connected over bluetooth. Two HC-05, are used on the gloves (one each) and they serve as master modules to the 2 HC-06 xBee modules which serve as slave modules on the board.
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
<BR/>
 +
''' Vibrator Motor '''
 +
Vibration sensors are used as feedback to the user. One vibration sensor on each glove.
 +
 
 +
|}
 +
 
 +
==== LED Matrix ====
 +
 
 +
The LED Matrix is composed of 1024 RGB LEDs arranged in a 32x32 matrix. Each RGB LED contains separate red, green, and blue LED chips within a single package. The display is split horizontally into two equal halves with 16 rows and 32 columns each. The display's columns are driven by one set of drivers and its rows by another set. In order for a particular LED to be lit, both its column and row driver have to be on. To manipulate the color of a specific LED, the red, green, and blue chips in the column driver are controlled separately. The column driver devolves into 3 32-bit shift registers as well as 3 32-bit parallel data registers. On each rising edge of the shift clock, whatever data is present at the 6 data inputs gets shifted in. Once the shift registers are filled, the latch signal is asserted to latch these register contents into the parallel data output register. The overall schematic for the LED Matrix is provided below.
 +
 
 +
[[File:CmpE244_S18_Cumulonimbus_MatrixDiagram.PNG]]
 +
 
 +
'''Figure 3: LED Matrix Hardware Configuration (Source: http://bikerglen.com/projects/lighting/led-panel-1up/)'''
 +
 
 +
====Printed Circuit Board (PCB) Design====
 +
 
 +
[[File:CmpE244 S18 Cumulonimbus AnnotatedPCB.PNG]]
 +
 
 +
'''Figure 4: PCB with Annotations'''
 +
 
 +
The PCB is used to distribute power between the SJOnes an the LED display from the DC Power Jack.  The PCB also routes the LED control and data signals to the SJOne.  The PCB uses a common ground plane across all of the LED signals in order to reduce Electromagnetic Interference between the signals.  The PCB also includes switches to allow the user to switch between game modes (1 player vs 2 player), as well as to start the game.
 +
 
 +
Holes for standoffs were also built into the PCB design for a convenient way to package the Smart Rock Paper Scissors systemBy screwing the SJOne's to the PCB, the wires were much neater and the hardware was able to be tightly integrated.
  
 
=== Hardware Interface ===
 
=== Hardware Interface ===
 
In this section, you can describe how your hardware communicates, such as which BUSes used.  You can discuss your driver implementation here, such that the '''Software Design''' section is isolated to talk about high level workings rather than inner working of your project.
 
In this section, you can describe how your hardware communicates, such as which BUSes used.  You can discuss your driver implementation here, such that the '''Software Design''' section is isolated to talk about high level workings rather than inner working of your project.
 +
 +
==== Bluetooth Gloves ====
 +
The hardware interface to the Flex sensors uses ADC for input and ground to complete connection.
 +
Vibration motor uses GPIO.
 +
Bluetooth HC-05 uses UART connection to the SJOne board.
 +
 +
{| class="wikitable"
 +
! SJ One Board Pin
 +
! Description
 +
|-
 +
| P0.0
 +
| Vibration motor Vcc
 +
|-
 +
| P0.1
 +
| Flex Sensor1 Vcc
 +
|-
 +
| P2.7
 +
| Flex Sensor2 Vcc
 +
|-
 +
| P0.26
 +
| Flex Sensor1 ADCin
 +
|-
 +
| P1.31
 +
| Flex Sensor2 ADCin
 +
|-
 +
| P RxD2
 +
| Bluetooth HC05 Tx
 +
|-
 +
| P TxD2
 +
| Bluetooth HC05 Rx
 +
|-
 +
| PTxD2
 +
| Bluetooth HC05 Vcc
 +
|-
 +
| P GND
 +
| Common to Flex sensors
 +
|-
 +
| P GND
 +
| Bluetooth HC05 GND
 +
|-
 +
| P GND
 +
| Vibration motor GND
 +
|}
 +
 +
[[File:CMPE244 S18 RPS glove sch labeled.jpg|thumb|600px|center|frame|Glove Connection]]
 +
 +
==== LED Matrix ====
 +
 +
The hardware interface to the LED Matrix uses GPIO for all data and control lines. A GPIO clock is toggled to shift in data sitting at the Matrix's 6 data ports.
 +
 +
{| class="wikitable"
 +
! SJ One Board Pin
 +
! Name
 +
! Description
 +
|-
 +
| P0.0
 +
| addrA
 +
| Address Input A
 +
|-
 +
| P0.1
 +
| addrB
 +
| Address Input B
 +
|-
 +
| P2.6
 +
| addrC
 +
| Address Input C
 +
|-
 +
| P2.7
 +
| addrD
 +
| Address Input D
 +
|-
 +
| P1.29
 +
| Latch
 +
| Shift in row data/Active High
 +
|-
 +
| P1.28
 +
| Output Enable
 +
| Turn on selected rows/Active Low
 +
|-
 +
| P1.19
 +
| Clock
 +
| Shift clock
 +
|-
 +
| P1.22
 +
| R1
 +
| Top half red data
 +
|-
 +
| P0.26
 +
| G1
 +
| Top half green data
 +
|-
 +
| P1.23
 +
| B1
 +
| Top half blue data
 +
|-
 +
| P1.30
 +
| R2
 +
| Bottom half red data
 +
|-
 +
| P1.31
 +
| G2
 +
| Bottom half green data
 +
|-
 +
| P2.5
 +
| B2
 +
| Bottom half blue data
 +
|}
 +
[[File: CmpE244 S18 Cumulonimbus MatrixInterface.png]]
 +
 +
'''Figure 5: Matrix Hardware Interface Block Diagram'''
 +
 +
====PCB Interface====
 +
The PCB has a 2x8 header to connect to the LED display, a 2x10 header to connect to the SJOne, a 2 pin header to connect to the UART of SJOne Board 1, and a 3 pin header to connect to the Ground and  UART of SJOne Board 2.
 +
 +
The header to the LED Display and SJOne are defined as in the above tables. 
 +
 +
{| class="wikitable"
 +
! PCB Pin
 +
! Description
 +
|-
 +
| P2.0
 +
| Start button
 +
|-
 +
| P2.1
 +
| Mode button
 +
|-
 +
| TX2
 +
| Connected between SJOne Board 1 & 2
 +
|-
 +
| RX2
 +
| Connected between SJOne Board 1 & 2
 +
|-
 +
|}
  
 
=== Software Design ===
 
=== Software Design ===
Show your software designFor example, if you are designing an MP3 Player, show the tasks that you are using, and what they are doing at a high level. Do not show the details of the code. For example, do not show exact code, but you may show psuedocode and fragments of code.  Keep in mind that you are showing DESIGN of your software, not the inner workings of it.
+
The Rock Paper Scissors software is designed with several modules in mind: game control logic, LED display, and the Bluetooth gloves.
 +
 
 +
===== Game Control Logic =====
 +
Smart Rock Paper Scissors was designed using classes to support a generic game flowRock, Paper, Scissors is a type of game, and each game has up to two players and one display.  Each player has a score and a controller.  The display and controller classes use initialization methods in order to create tasks to handle the display and controller drivers. 
 +
 
 +
The game parent class creates a task that handles the flow of each game in stages.  A game is started by calling "start_game" on an instance of the game you wish to start.  Start game will create a task for the game logic, and then launch the task scheduler. 
 +
 
 +
[[File:CMPE244 S18 RPS ClassDiagram.png]]
 +
 
 +
'''Figure 6: Game and Rock, Paper, Scissors Classes'''
 +
 
 +
The game task operates on a private "stage" variable, where each stage is an abstraction of levels for different types of games.  The game task will continually call the virtual function get_stage().  get_stage() must be defined by the subclass game, in this case Rock, Paper, Scissors.  The Rock, Paper, Scissors get_stage() will determine which stage to call based on the current stage value, and will handle switching between stages as makes sense in the flow of the game.  The game task will run forever unless the developer implements logic for the "stop" variable.  The game class maintains this "stop" variable in case the developer wants to quit the current game type.
 +
 
 +
[[File:CMPE244 S18 RPS PlayerDiagram.png]]
 +
 
 +
'''Figure 7: Player Class'''
 +
 
 +
The game class also has "start", "mode" and "reset" variables that help to control the flow of the game.  For Rock, Paper, Scissors, the start variable corresponds to an external start switch.  When the start switch is pressed, the game "start" is set to high and the Rock, Paper, Scissors will begin the rounds of the game.
 +
 
 +
[[File:CmpE244 S18 Cumulonimbus SWFlow.PNG]]
 +
 
 +
'''Figure 8: Overall Software Flow of Smart Rock, Paper Scissors'''
 +
 
 +
===== Bluetooth Gloves =====
 +
 
 +
A very important part of this project is gesture recognition. Flex sensors are used to control and recognize gestures made by users for rock, paper and scissors. The flex sensor is nothing but a resistor whose resistor increases when it is bent. Thus, voltage across the flex sensor decreases.
 +
*The voltage is read by the controller using ADC driver and threshold is set whether the sensor is OFF or ON.
 +
*One sensor of the 2 is programmed as a key. which state will impact whether the current state is Rock or Paper or Scissor. If the FlexKey is OFF i.e. voltage read at ADC < threshold set, and other flex is ON i.e. voltage at other Flex > threshold, then Scissors. IF key is OFF and Flex is also OFF, Rock and Key as well as flex are ON, then Paper is considered as current state of user's gesture.
 +
 
 +
<syntaxhighlight lang="C">
 +
 
 +
            if(flexKey < thresholdLow && flex < thresholdHigh)
 +
            {
 +
                strncpy(status, "R", 1);
 +
                xQueueSend(q, status, 0);
 +
            }
 +
            else if(flexKey > thresholdLow && flex > thresholdHigh)
 +
            {
 +
                strncpy(status, "P", 1);
 +
                xQueueSend(q, status, 0);
 +
            }
 +
            else if(flexKey < thresholdLow && flex > thresholdHigh)
 +
            {
 +
                strncpy(status, "S", 1);
 +
                xQueueSend(q, status, 0);
 +
            }
 +
 
 +
</syntaxhighlight>
 +
 
 +
This gesture from the user is to be sent to the controller by wireless communication. Bluetooth is used for this communication.
 +
*Value obtained at the ADC is written to the Bluetooth master(HC05 connected on UART2). Bluetooth transmits this value to the slave Bluetooth connected on controller.
 +
 
 +
<syntaxhighlight lang="C">
 +
 
 +
        if (xQueueReceive(q, &buffer, portMAX_DELAY))
 +
        {
 +
            //send over bluetooth to game
 +
 
 +
            buffer[9] = '\0';
 +
            uartObj.putline(buffer);
 +
        }
 +
 
 +
</syntaxhighlight>
 +
 
 +
The user needs to know when to gesture and create a rock or paper or scissor. This needs some indication from controller side.
 +
*Controller sends signal using Bluetooth to the glove side Bluetooth when round starts.
 +
*GPIO inerface is used for vibration motors that vibrates on this indication.
 +
 
 +
<syntaxhighlight lang="C">
 +
 
 +
        if(uartObj.getChar(buff,0))
 +
        {
 +
            buff[2] = '\0';
 +
            // if feedback opcode
 +
            if(buff[0] == 'A')
 +
            {
 +
                //vibration sensors
 +
                activateVibr();
 +
                delay_ms(1000);
 +
                clearVibr();
 +
            }
 +
        }
 +
 
 +
</syntaxhighlight>
 +
 
 +
 
 +
===== LED Matrix =====
 +
 
 +
An LEDMatrixGPIO class was built for the purposes of providing a common singleton instance to allow our state machine to easily draw images on our LED Matrix. A uint32_t 2-D array object called frameBuffer stored as a private member of this class is used to store the current 'image frame' that should be displayed on the matrix. The 32-bit pixel data for each row and each color is stored within this singular frameBuffer. The class provides API functions that will be used by our state machine such as drawStartScreen(), drawScoreboard(), drawRoundEnd(), drawGameOver(), and updateDisplay() are described in detail below.
 +
 
 +
====== updateDisplay() ======
 +
 
 +
The core of the LED Matrix API is an updateDisplay function that is called by a Repetitive Interrupt Timer at a rate of 30-50 MHz. This essentially acts as our refresh clock frequency. The updateDisplay code fragment is shown below.  
 +
 
 +
<syntaxhighlight lang="C">
 +
 
 +
//designed to loop through frameBuffer and shift data for paired rows
 +
void updateDisplay()
 +
{
 +
    //only need to iterate up to 16 because we're doing 2 rows at a time
 +
    for(int i = 0; i < 16; i++)
 +
    {
 +
        setEn(true); //turn off currently displayed row
 +
        setLatch(false); //about to shift in data so make sure that data isn't latched until whole row is shifted in
 +
        selRow(i); //will select i and i+16 rows at same time, this function sets 4 address lines to select correct rows
 +
        //this inner loop will shift MSB of frameBuffer[i] first into corresponding shift register
 +
        for(int j = 31; j >= 0; j--)
 +
        {
 +
            ((frameBuffer[i][0] >> j) & 1) ? r1->setHigh() : r1->setLow(); //shift in r1 data with msb getting shifted in first
 +
            ((frameBuffer[i][1] >> j) & 1) ? g1->setHigh() : g1->setLow(); //g1 data
 +
            ((frameBuffer[i][2] >> j) & 1) ? b1->setHigh() : b1->setLow(); //b1 data
 +
 
 +
            //bottom half for corresponding row; rows 0/16, 1/17 etc
 +
            ((frameBuffer[i + 16][0] >> j) & 1) ? r2->setHigh() : r2->setLow(); //r2
 +
            ((frameBuffer[i + 16][1] >> j) & 1) ? g2->setHigh() : g2->setLow(); //g2
 +
            ((frameBuffer[i + 16][2] >> j) & 1) ? b2->setHigh() : b2->setLow(); //b2
 +
            clk->setHigh(); clk->setLow(); //shift in all 6 bits at once
 +
        }
 +
 
 +
        //at this point, all 3 shift registers should be filled with corresponding row data in frameBuffer
 +
        setLatch(true); //latch shift register contents down to parallel output register
 +
        setEn(false); //turn on two rows i and i+16
 +
        delay_us(250); //very small delay to reduce flickering
 +
    }
 +
}
 +
 
 +
</syntaxhighlight>
 +
 
 +
With this updateDisplay running in the background, drawing images simply entails updating frameBuffer with the bit representations of desired images and letting updateDisplay handle the overhead of toggling control signals such as latch and output enable. For each 'image' that we're using, such as words, numbers and graphics, the bitmap representations had to be painstakingly obtained so that information about which pixels are lit could be procured for each row and each color.
 +
 
 +
====== drawStartScreen() ======
 +
 
 +
drawStartScreen() works by copying a pre-built startScreenImg 2-D array into the common frameBuffer instance. The refresh function running in the background continually calling updateDisplay() will handle the actual overhead of drawing the image. An animation is used within drawStartScreen to switch between a 'START' idle image as well as a 'MODE' image indicating the current mode that the game is set to (1P or 2P). When the appropriate switch is pressed, the state machine clears the screen and calls drawScoreboard().
 +
 
 +
[[File: CMPE244 S18 RPS StartScreen.jpeg]]
 +
 
 +
'''Figure 9: Start Screen'''
 +
 
 +
====== drawScoreboard(uint8_t roundNum, uint8_t leftScore, uint8_t rightScore) ======
 +
 
 +
This function is used to draw the entire pre-round animation as well as re-draw the in-round score. First, it continually updates the frameBuffer to allow an animation of 'Round' from the bottom to the top of the screen. Once there, it displays roundNum and flashes the round before disappearing for the scoreboard. Regardless of 1P or 2P mode, the players are the same with P1 and P2/CPU. This function calls a drawCountdown() that counts down from 3 to synchronize the recorded player gestures. Afterwards, a drawAction(Action left, Action right) call is made to draw the gestures of each player. A new Action enumeration was made with Rock, Paper, and Scissors as possible options. A variation of the scoreBoard design is shown below. While drawStartScreen simply modifies the entire frameBuffer, drawScoreboard and its subfunctions like drawCountdown, drawScore etc, simply clear their respective sections of the screen and shift pre-drawn bit images to their correct positions.
 +
 
 +
[[File: CMPE244_S18_RPS_RoundScreen.jpeg]]
 +
 
 +
====== drawRoundEnd(Player winner) ======
 +
 
 +
This function is called after a drawScoreboard call once a winner has been determined. A new Player enumeration was created so the state machine can indicate whether Player1, Player2 or Neither was the winner. This function clears all pixels under the scoreboard and displays a winner animation.
 +
 
 +
====== drawGameOver(Player matchWinner) ======
 +
 
 +
This function is called when a player reaches the end condition of 3 wins. This function clears the screen and then displays the winner. Slight alterations are made on this screen depending on mode.
  
 
=== Implementation ===
 
=== Implementation ===
This section includes implementation, but again, not the details, just the high level.  For example, you can list the steps it takes to communicate over a sensor, or the steps needed to write a page of memory onto SPI Flash. You can include sub-sections for each of your component implementation.
+
==== Game logic ====
 +
# Create tasks for Controllers, Display and Game
 +
# Start task scheduler
 +
# Call run_stage() virtual function, which calls Rock, Paper, Scissors run_stage()
 +
# Beginning at stage 0, run Rock, Paper, Scissors state machine based on stages
 +
# Stage 3 will reset stage to 0, where user can press "start" to play again
 +
 
 +
==== Bluetooth Gloves ====
 +
 
 +
Steps to acquire data from gloves and send it via Bluetooth to controller
 +
# Initialize ADC for 2 channels (power to Flex sensors)
 +
# Initialize UART2
 +
# Read ADC input and write it to UART2
 +
# If the data is read successfully and received feedback from the controller, vibrate the indicator motor
 +
# Repeat this process for every round
 +
 
 +
[[File:CmpE244 S18 Cumulonimbus Glove Flow.PNG]]
 +
 
 +
==== LED Matrix ====
 +
 
 +
Steps to write to LED Matrix:
 +
# Set output enable HIGH to disable LED output for selected row
 +
# Set latch LOW to block shift register from pushing to parallel data output register
 +
# Shift in 32 bits of data for R1, G1, B1, R2, G2, B2 for rows 0/16
 +
# Set addr lines A, B, C, D to select rows 0/16
 +
# Set latch HIGH to move shift register data into parallel data output register
 +
# Set output enable LOW to display currently selected row
 +
# Repeat this process for the other rows 100 to 200 times per second
 +
 
 +
The overall flow is also shown below.
 +
 
 +
[[File: CMPE244_S18_RPS_MatrixFlowChart.jpg]]
  
 
== Testing & Technical Challenges ==
 
== Testing & Technical Challenges ==
Describe the challenges of your projectWhat advise would you give yourself or someone else if your project can be started from scratch again?
+
In order to get the Smart Rock Paper Scissors working well, lots of testing was required.  We faced several difficult technical challenges, as detailed below.
Make a smooth transition to testing section and described what it took to test your project.
+
 
 +
=== Game Logic ===
 +
 
 +
===== Using tasks within game control logic =====
 +
In order to implement the game logic in an abstract way, each class was to create its own task and handle its own private membersHowever, in order for a task to access a previously instantiated class, the class must be in memory that is accessible to the task.  When initially creating the classes, they were created in stack space in the main function was not visible to the individual class tasks (for example, the player task could not access its member controller class because the controller class was not pointing to the right space in memory).  In order to solve this issue, the classes were initialized in global memory and pointers to the global memory space was passed to various tasks required to run the game.
 +
 
 +
=== Bluetooth Gloves ===
 +
 
 +
Some technical challenges such as ADC interface, shortage of power source pins and Bluetooth connectivity were faced in the initial phases. They were rectified in the later phases and described below:
 +
 
 +
===== Flex Sensor-SJOne board ADC interface =====
 +
*The number of Vcc pins on the board are less than required. Thus GPIO pins P0.1 P0.0 are used in High state to use as Vcc source for Flex sensors.
 +
*Flex sensor requires voltage divider circuit to pull down to ground. 10K resistor is used with each Flex sensor.
 +
*Ground for 2 flex sensors on one glove is made coommon on a separate GPB due to shortage of ground pins on SJOne board.
 +
 
 +
===== Bluetooth pairing =====
 +
*Bluetooth module XBee HC06 can not work in master as well as slave modes. Bluetooth module HC05, which supports master as well as slave modes does not conme in Xbee package, which attachment is provided on SJOne board. However, Controller side boards i.e. board connected to RGB led matrix did not have vacant UART pins since used for the matrix. So, usage of 4 HC05 modules is limited. As a solution, 2 HC06 modules on controller side boards(slaves) and 2 HC05 modules, 1 on each glove(masters), are used.
 +
*Bluetooth module HC05 by default comes in slave mode. It has to be converted to master mode. This conversion is done using AT commands. Then the module is paired with HC06(slave) using its mac address.
 +
 
 +
=== LED Matrix ===
 +
 
 +
The RGB LED Matrix presented several challenges from both a hardware and software standpoint.
 +
 
 +
Several of these problems are described below:
  
Include sub-sections that list out a problem and solution, such as:
+
===== SPI vs GPIO =====
 +
The LED Matrix works by shifting in data available at its six data ports: R1, G1, B1, R2, G2, B2. Early experimentation with an SPI interface involved testing data transfer through a MOSI to B1 and SPI_CLK to SCLK connection. While this worked fine for top half control with one color, more thought had to go into being able to transfer data to the 2 other top-half color data ports as well as the bottom half color data ports. An approach that was tried was connecting MOSI0 to one of the three top-half data ports and MOSI1 to one of the three bottom-half ports to at least have one color control over both halves. Because the matrix takes in a single SCLK input to shift in data for both halves, a high-speed multiplexer was connected to let the SJOne board select the SPI clock that would be shifting in data depending on which half would be receiving data. Because MOSI is not low when idle, different approaches were tried to make sure no junk data was being shifted in when data for the other half was being written. Setting the un-used MOSI port to a GPIO port and then driving it low during the brief interval when the other half was shifting in SPI data proved unsuccessful. Another approach was to feed both MOSI ports into two separate multiplexers so our software could choose either MOSI or ground also proved unsuccessful. Even if either approach worked, a solution would also have to be discovered for how to write data into the 4 other color data ports. A theory was to use the matrix output port that would usually be connected to the input of another matrix display. It was thought that the output of any particular color shift register would go towards the corresponding pin in the output port and that a physical wire could be used to "daisy-chain" the data back into the input of another color register. If this worked, data for all three colors could be sent as three 32-bit words where data could be shifted through the R, G, and B shift registers. However, this ended up not being fruitful. In the end, GPIO was used to build a working API.
  
=== <Bug/issue name> ===
+
With GPIO, the issue of having to write data to 6 data lines with only two MOSI lines was solved by just using 6 GPIO ports and a GPIO clock. Multiplexers to switch between two clocks and other data lines were no longer needed.
Discuss the issue and resolution.
+
 
 +
===== Sensitivity =====
 +
The LED Matrix proves to be extremely sensitive to input and other factors. Early use of the matrix showed flickering and other visual glitches to slight movements of the data lines as well as the power supply line. An initial theory was that this was a software issue and adjustments would have to be made to the control line sequence such as when address lines are changed and when latch is asserted or perhaps timing. Data lines were taped in a particular way as to minimize flickering. However, the flickering always returned after a short amount of time. The cause of this flickering was eventually traced to an unstable power supply. Replacing the unstable supply with a better-built 5V 4A power adapter virtually eliminated all of the display problems that we encountered.
 +
 
 +
=====Bitmapping=====
 +
The images of rock paper and scissors along with other graphics and animations were made using bitmapping. Initially, creating a direct bitmap was tough. The images were drawn using a bunch of drawPixel function calls. The API takes in the coordinates x and y along with the color as parameters. Once the image could be made using this API, the drawPixel function calls were replaced by bitmaps in c. This helped in cleaning the part of the code.
 +
 
 +
=== Testing ===
 +
 
 +
==== Bluetooth Gloves ====
 +
 
 +
Bluetooth glove testing includes testing of 3 individual components and then testing of glove as a whole.
 +
*Flex sensors are tested for 45 degree to 90 degree bend of palm for gesture recognition.
 +
*The resistance and voltage is monitored continuously while checking states.
 +
*Bluetooth modules were tested by sending and reception of smallest data as single character to Large strings.
 +
*Bluetooth module pairing was tested recursively by powering on different modules and checked if it gets paired. It should not get paired to any other Bluetooth module other than set mac address.
 +
*Vibration motor was tested by sending right and wrong indications with small and large time intervals. It should vibrate only on right signal indication.
 +
*Glove as a whole was tested by capturing Flex values, sending them to slave Bluetooth and vibrate the motor if it gives right indication.
 +
 
 +
==== LED Matrix ====
 +
 
 +
Testing the LED Matrix independently simply involves an eye inspection of which pixels are lit and their color. Images that were used were drawn out by hand on a paper grid and as such porting them to the matrix display itself involved left shifting these patterns into the specific rows. Typical errors such as off-by-one could be quickly corrected through trial and error inspection. Many errors such as the additional LEDs lighting up could be detected. After testing each and every part, cables and also replacing the board, we could find the error was in the power adapter which was then replaced with a better working one.
 +
 
 +
==== System ====
 +
 
 +
The system integration was tested by adding one module at a time.  The first step was to create the game classes and game control logic.  Once the game control logic and classes were in place, the Bluetooth glove was plugged into the abstract controller class to ensure that the Rock Paper Scissors values could be read from the glove.  The controller also needed to receive the feedback from the vibration motor, so that was tested in parallel with the Bluetooth connection.  The tests we ran into while testing is detailed in the above Technical Challenges section.
 +
 
 +
After verifying that one Bluetooth glove worked with the system, we integrated the Display logic into the game control code.  The key to successful integration of these modules was to understand the interfaces between each module - and by abstracting the classes of the game, it was easy to integrate into the controller and display logic to the game logic.
 +
 
 +
==== PCB ====
 +
 
 +
The PCB was tested in steps:
 +
 
 +
1. Ensure none of the components are shorted after assembly (short test with the multimeter).
 +
 
 +
2. Verify connections on PCB and resistance values with the multimeter.
 +
 
 +
3. Plug in SJOne to power on PCB and verify board can power on with DC power jack.
 +
 
 +
4. Test push buttons on PCB to GPIO on SJOne.
 +
 
 +
5. Connect all display wires and run Rock, Paper, Scissors to verify display works.
 +
 
 +
6. Connect Bluetooth Glove to SJOne on PCB and verify Bluetooth works with PCB connections.
 +
 
 +
== Lessons Learned ==
 +
 
 +
*PCB components need to be carefully selected and verified.
 +
 
 +
*Creating class abstractions can help future-proof code, but make short term implementations more complicated.
 +
 
 +
*Bad or noisy power supplies can create enough EMI and signal noise to create significant fluctuations on nearby wires.
 +
 
 +
*Some Bluetooth modules only support Slave or Master mode in the firmware. Re-flashing the firmware of Bluetooth modules is a very difficult process, it is better to buy modules that have support for master/slave mode.
 +
 
 +
*Using the Repetitive Interrupt Timer on a board with. many tasks can cause strange behavior when the RIT is expected to operate very quickly.  If the RIT has too much to do in too little time, the function run by the RIT can operate slightly differently on various iterations.
  
 
== Conclusion ==
 
== Conclusion ==
Conclude your project hereYou can recap your testing and problemsYou should address the "so what" part here to indicate what you ultimately learnt from this projectHow has this project increased your knowledge?
+
This project provided a broad application of knowledge learned over the course of this lecture with ADC, GPIO, SPI, UART and RIT used at various points of this project's development. When confronted with many technical challenges during component development as well as integration, several important problem-solving skills were applied to resolve these issuesThe Smart Rock Paper Scissors project works fully as intended - both gloves are able to send accurate Rock Paper Scissors commands to the SJOne receiver boards, and the game logic takes these inputs and runs the Rock Paper Scissors game smoothly.  The LED display makes creative use of animation and prompts to enhance the user experience of the Rock Paper Scissors gameThe tactic feedback of the Bluetooth Gloves helps the user know that their input was received correctly as well as making the game more exciting. 
 +
 
 +
Overall, we are proud to have created a tightly integrated Smart Rock Paper Scissors gameBy using a PCB to integrate the receiver boards with the LED matrix, and using gloves, velcro and USB power banks to create wireless gloves, the experience of Rock Paper Scissors has never been more fun.
  
 
=== Project Video ===
 
=== Project Video ===
Upload a video of your project and post the link here.
+
[https://youtu.be/ApM4fAIpozc Smart Rock Paper Scissors Video]
  
 
=== Project Source Code ===
 
=== Project Source Code ===
[https://sourceforge.net/projects/sjsu/files/CmpE_S2016/ Sourceforge Source Code Link]
+
[https://gitlab.com/jskow/Cumulonimbus-GloveGame Source Code]
 +
[http://socialledge.com/sjsu/images/4/49/CmpE244_F18_RPS_source_code.zip Zip File]
  
 
== References ==
 
== References ==
 
=== Acknowledgement ===
 
=== Acknowledgement ===
Any acknowledgement that you may wish to provide can be included here.
+
We would like to express our gratitude to Professor Preetpal Kang for generously sharing his time and knowledge with us and guiding us through the completion of this project. We would also like to thank the ISA team for their valuable advice and constructive feedback.
  
 
=== References Used ===
 
=== References Used ===
List any references used in project.
 
  
=== Appendix ===
+
[https://learn.sparkfun.com/tutorials/flex-sensor-hookup-guide Flex Sensor guide]<br>
You can list the references you used.
+
[http://www.electronicaestudio.com/docs/istd016A.pdf Bluetooth HC 05 guide]<br>
 +
[https://cdn-learn.adafruit.com/downloads/pdf/32x16-32x32-rgb-led-matrix.pdf  RGB LED Matrix Guide]<br>
 +
[http://bikerglen.com/projects/lighting/led-panel-1up/ RGB LED Matrix Tutorial]

Latest revision as of 14:07, 27 May 2018

Smart Rock Paper Scissors

Hand gesture controlled rock paper scissors game.

CmpE244 S18 Cumulonimbus SystemPicture.PNG

Figure 1: The Smart Rock Paper Scissors System

Abstract

This project recreates the classic Rock,Paper, Scissors game using a LED Matrix to display and keep track of scoring, and wireless gloves to record the user input. The LED Matrix helps users to know what round they are on, what the score is, as well as provide prompts of when to give the Rock, Paper or Scissors signal. The Smart aspect of the system also allows users to play against a computer player. The wireless gloves utilize Bluetooth modules and flex sensors to detect the player input and feed that input into the game logic and game controller.

Objectives & Introduction

  • Create LED display to guide players through the game
  • Create wireless gloves to allow players to give input
  • Create feedback to the gloves to enhance the game experience
  • Make the game platform modular, so other games could be easily built with the system

Team Members & Responsibilities

  • Josh Skow
    • System Integration
    • PCB
  • Kaustubh Jawalekar
    • Bluetooth
    • Glove, Sensors
  • Kevin Gadek
    • LED Display
  • Akinfemi Aluko
    • Bluetooth
    • Glove, Sensors
  • Sarvpreet Singh
    • LED Display

Schedule

Week# Date Task Actual
1 03/20 Submit Project Proposal Completed.

Had to modify project proposal from original submission.

2 03/27 Determine project architecture Completed.
3 04/03 Order Parts Completed.
4 04/10 Begin work on Display, Bluetooth

Begin work on PCB.

Begin testing glove sensors.

Completed.
5 04/17 Write multiple lines on Display Successfully

Set Bluetooth device to master mode

Read valid data from flex sensor

Completed.

Had to purchase a Bluetooth module that supports master mode.

6 04/24 Draw full image on Display.

Complete Bluetooth logic.

Begin working on game logic and game state machine.

Begin assembling glove and sensors to mount on wrist.

Finish PCB, and submit PCB layout to fab.

Game logic completed. Bluetooth logic working, and basic gloves put together.
7 05/01 Finalize glove design, and wrist mounting.

Finish Display logic.

Finalize game logic.

Receive PCB from fab, populate and test PCB.

Completed.

PCB received.

Glove drivers working and glove is physically put together.

8 05/08 Test for edge cases with game logic.

Fine tune display images for game.

Add game features if time permits.

Re-work PCB as needed based on testing.

Completed
9 05/15 Complete final project documentation.

Prepare project for demo.

Completed.
10 05/22 Finish debugging project for demo. Completed.

Parts List & Cost

Part Name Description Link Quantity Cost
32x32 RGB LED Matrix 32x32 Matrix to display game Link 1 $44.95
Bluetooth HC-05 Module Bluetooth module with master mode capability Link 2 $29.90
XBee Bluetooth HC-06 Module Gikfun HC-06 Bluetooth Bee V2.0 Slave Module Link 2 $23.96
SJOne Boards Microcontrollers N/A 4 $320
Flex Sensors Flex sensors to measure finger flex Link 4 $26
Parallax 28821 Vibration motors Feedback motors Link 2 $11
Gloves Gloves to record gestures Link 2 $20
PCB PCB to route power Link 3 $140

Design & Implementation

Hardware Design

Smart Rock Paper Scissors used a 32x32 LED Display, four Bluetooth modules, and four SJOne boards as the primary hardware devices in the system. Two SJOne boards were used as Bluetooth receivers, and two SJOne boards were used as Bluetooth transmitters. One Bluetooth receiver board was also responsible for driving the LED Display and handling the Rock, Paper, Scissors logic.

A PCB was created in order to route power from a 5V, 4A AC/DC power supply to the LED display, and both Bluetooth receiver SJOne boards. The SJOne boards attached to the glove controllers each use USB power banks to power the board.

CMPE244 S18 RPS BlockDiagram.png

Figure 2: Block diagram of the Smart Rock, Paper, Scissors hardware components.

Bluetooth Gloves

Flex Sensors
Bluetooth HC-05
Vibtration Motor

Flex Sensors Flex sensors are used to detect the hand gesture of the user. Two flex sensors are used on fingers 2 and 4, if paper both are not bent, if scissors one is bent and if rock both are bent.







Bluetooth HC-05 The gloves are connected over bluetooth. Two HC-05, are used on the gloves (one each) and they serve as master modules to the 2 HC-06 xBee modules which serve as slave modules on the board.







Vibrator Motor Vibration sensors are used as feedback to the user. One vibration sensor on each glove.

LED Matrix

The LED Matrix is composed of 1024 RGB LEDs arranged in a 32x32 matrix. Each RGB LED contains separate red, green, and blue LED chips within a single package. The display is split horizontally into two equal halves with 16 rows and 32 columns each. The display's columns are driven by one set of drivers and its rows by another set. In order for a particular LED to be lit, both its column and row driver have to be on. To manipulate the color of a specific LED, the red, green, and blue chips in the column driver are controlled separately. The column driver devolves into 3 32-bit shift registers as well as 3 32-bit parallel data registers. On each rising edge of the shift clock, whatever data is present at the 6 data inputs gets shifted in. Once the shift registers are filled, the latch signal is asserted to latch these register contents into the parallel data output register. The overall schematic for the LED Matrix is provided below.

CmpE244 S18 Cumulonimbus MatrixDiagram.PNG

Figure 3: LED Matrix Hardware Configuration (Source: http://bikerglen.com/projects/lighting/led-panel-1up/)

Printed Circuit Board (PCB) Design

CmpE244 S18 Cumulonimbus AnnotatedPCB.PNG

Figure 4: PCB with Annotations

The PCB is used to distribute power between the SJOnes an the LED display from the DC Power Jack. The PCB also routes the LED control and data signals to the SJOne. The PCB uses a common ground plane across all of the LED signals in order to reduce Electromagnetic Interference between the signals. The PCB also includes switches to allow the user to switch between game modes (1 player vs 2 player), as well as to start the game.

Holes for standoffs were also built into the PCB design for a convenient way to package the Smart Rock Paper Scissors system. By screwing the SJOne's to the PCB, the wires were much neater and the hardware was able to be tightly integrated.

Hardware Interface

In this section, you can describe how your hardware communicates, such as which BUSes used. You can discuss your driver implementation here, such that the Software Design section is isolated to talk about high level workings rather than inner working of your project.

Bluetooth Gloves

The hardware interface to the Flex sensors uses ADC for input and ground to complete connection. Vibration motor uses GPIO. Bluetooth HC-05 uses UART connection to the SJOne board.

SJ One Board Pin Description
P0.0 Vibration motor Vcc
P0.1 Flex Sensor1 Vcc
P2.7 Flex Sensor2 Vcc
P0.26 Flex Sensor1 ADCin
P1.31 Flex Sensor2 ADCin
P RxD2 Bluetooth HC05 Tx
P TxD2 Bluetooth HC05 Rx
PTxD2 Bluetooth HC05 Vcc
P GND Common to Flex sensors
P GND Bluetooth HC05 GND
P GND Vibration motor GND
Glove Connection

LED Matrix

The hardware interface to the LED Matrix uses GPIO for all data and control lines. A GPIO clock is toggled to shift in data sitting at the Matrix's 6 data ports.

SJ One Board Pin Name Description
P0.0 addrA Address Input A
P0.1 addrB Address Input B
P2.6 addrC Address Input C
P2.7 addrD Address Input D
P1.29 Latch Shift in row data/Active High
P1.28 Output Enable Turn on selected rows/Active Low
P1.19 Clock Shift clock
P1.22 R1 Top half red data
P0.26 G1 Top half green data
P1.23 B1 Top half blue data
P1.30 R2 Bottom half red data
P1.31 G2 Bottom half green data
P2.5 B2 Bottom half blue data

CmpE244 S18 Cumulonimbus MatrixInterface.png

Figure 5: Matrix Hardware Interface Block Diagram

PCB Interface

The PCB has a 2x8 header to connect to the LED display, a 2x10 header to connect to the SJOne, a 2 pin header to connect to the UART of SJOne Board 1, and a 3 pin header to connect to the Ground and UART of SJOne Board 2.

The header to the LED Display and SJOne are defined as in the above tables.

PCB Pin Description
P2.0 Start button
P2.1 Mode button
TX2 Connected between SJOne Board 1 & 2
RX2 Connected between SJOne Board 1 & 2

Software Design

The Rock Paper Scissors software is designed with several modules in mind: game control logic, LED display, and the Bluetooth gloves.

Game Control Logic

Smart Rock Paper Scissors was designed using classes to support a generic game flow. Rock, Paper, Scissors is a type of game, and each game has up to two players and one display. Each player has a score and a controller. The display and controller classes use initialization methods in order to create tasks to handle the display and controller drivers.

The game parent class creates a task that handles the flow of each game in stages. A game is started by calling "start_game" on an instance of the game you wish to start. Start game will create a task for the game logic, and then launch the task scheduler.

CMPE244 S18 RPS ClassDiagram.png

Figure 6: Game and Rock, Paper, Scissors Classes

The game task operates on a private "stage" variable, where each stage is an abstraction of levels for different types of games. The game task will continually call the virtual function get_stage(). get_stage() must be defined by the subclass game, in this case Rock, Paper, Scissors. The Rock, Paper, Scissors get_stage() will determine which stage to call based on the current stage value, and will handle switching between stages as makes sense in the flow of the game. The game task will run forever unless the developer implements logic for the "stop" variable. The game class maintains this "stop" variable in case the developer wants to quit the current game type.

CMPE244 S18 RPS PlayerDiagram.png

Figure 7: Player Class

The game class also has "start", "mode" and "reset" variables that help to control the flow of the game. For Rock, Paper, Scissors, the start variable corresponds to an external start switch. When the start switch is pressed, the game "start" is set to high and the Rock, Paper, Scissors will begin the rounds of the game.

CmpE244 S18 Cumulonimbus SWFlow.PNG

Figure 8: Overall Software Flow of Smart Rock, Paper Scissors

Bluetooth Gloves

A very important part of this project is gesture recognition. Flex sensors are used to control and recognize gestures made by users for rock, paper and scissors. The flex sensor is nothing but a resistor whose resistor increases when it is bent. Thus, voltage across the flex sensor decreases.

  • The voltage is read by the controller using ADC driver and threshold is set whether the sensor is OFF or ON.
  • One sensor of the 2 is programmed as a key. which state will impact whether the current state is Rock or Paper or Scissor. If the FlexKey is OFF i.e. voltage read at ADC < threshold set, and other flex is ON i.e. voltage at other Flex > threshold, then Scissors. IF key is OFF and Flex is also OFF, Rock and Key as well as flex are ON, then Paper is considered as current state of user's gesture.
            if(flexKey < thresholdLow && flex < thresholdHigh)
            {
                strncpy(status, "R", 1);
                xQueueSend(q, status, 0);
            }
            else if(flexKey > thresholdLow && flex > thresholdHigh)
            {
                strncpy(status, "P", 1);
                xQueueSend(q, status, 0);
            }
            else if(flexKey < thresholdLow && flex > thresholdHigh)
            {
                strncpy(status, "S", 1);
                xQueueSend(q, status, 0);
            }

This gesture from the user is to be sent to the controller by wireless communication. Bluetooth is used for this communication.

  • Value obtained at the ADC is written to the Bluetooth master(HC05 connected on UART2). Bluetooth transmits this value to the slave Bluetooth connected on controller.
        if (xQueueReceive(q, &buffer, portMAX_DELAY))
        {
            //send over bluetooth to game

            buffer[9] = '\0';
            uartObj.putline(buffer);
        }

The user needs to know when to gesture and create a rock or paper or scissor. This needs some indication from controller side.

  • Controller sends signal using Bluetooth to the glove side Bluetooth when round starts.
  • GPIO inerface is used for vibration motors that vibrates on this indication.
        if(uartObj.getChar(buff,0))
        {
            buff[2] = '\0';
            // if feedback opcode
            if(buff[0] == 'A')
            {
                //vibration sensors
                activateVibr();
                delay_ms(1000);
                clearVibr();
            }
        }


LED Matrix

An LEDMatrixGPIO class was built for the purposes of providing a common singleton instance to allow our state machine to easily draw images on our LED Matrix. A uint32_t 2-D array object called frameBuffer stored as a private member of this class is used to store the current 'image frame' that should be displayed on the matrix. The 32-bit pixel data for each row and each color is stored within this singular frameBuffer. The class provides API functions that will be used by our state machine such as drawStartScreen(), drawScoreboard(), drawRoundEnd(), drawGameOver(), and updateDisplay() are described in detail below.

updateDisplay()

The core of the LED Matrix API is an updateDisplay function that is called by a Repetitive Interrupt Timer at a rate of 30-50 MHz. This essentially acts as our refresh clock frequency. The updateDisplay code fragment is shown below.

//designed to loop through frameBuffer and shift data for paired rows
void updateDisplay()
{
    //only need to iterate up to 16 because we're doing 2 rows at a time
    for(int i = 0; i < 16; i++)
    {
        setEn(true); //turn off currently displayed row
        setLatch(false); //about to shift in data so make sure that data isn't latched until whole row is shifted in
        selRow(i); //will select i and i+16 rows at same time, this function sets 4 address lines to select correct rows
        //this inner loop will shift MSB of frameBuffer[i] first into corresponding shift register
        for(int j = 31; j >= 0; j--)
        {
            ((frameBuffer[i][0] >> j) & 1) ? r1->setHigh() : r1->setLow(); //shift in r1 data with msb getting shifted in first
            ((frameBuffer[i][1] >> j) & 1) ? g1->setHigh() : g1->setLow(); //g1 data
            ((frameBuffer[i][2] >> j) & 1) ? b1->setHigh() : b1->setLow(); //b1 data

            //bottom half for corresponding row; rows 0/16, 1/17 etc
            ((frameBuffer[i + 16][0] >> j) & 1) ? r2->setHigh() : r2->setLow(); //r2
            ((frameBuffer[i + 16][1] >> j) & 1) ? g2->setHigh() : g2->setLow(); //g2
            ((frameBuffer[i + 16][2] >> j) & 1) ? b2->setHigh() : b2->setLow(); //b2
            clk->setHigh(); clk->setLow();  //shift in all 6 bits at once
        }

        //at this point, all 3 shift registers should be filled with corresponding row data in frameBuffer
        setLatch(true); //latch shift register contents down to parallel output register
        setEn(false); //turn on two rows i and i+16
        delay_us(250); //very small delay to reduce flickering
    }
}

With this updateDisplay running in the background, drawing images simply entails updating frameBuffer with the bit representations of desired images and letting updateDisplay handle the overhead of toggling control signals such as latch and output enable. For each 'image' that we're using, such as words, numbers and graphics, the bitmap representations had to be painstakingly obtained so that information about which pixels are lit could be procured for each row and each color.

drawStartScreen()

drawStartScreen() works by copying a pre-built startScreenImg 2-D array into the common frameBuffer instance. The refresh function running in the background continually calling updateDisplay() will handle the actual overhead of drawing the image. An animation is used within drawStartScreen to switch between a 'START' idle image as well as a 'MODE' image indicating the current mode that the game is set to (1P or 2P). When the appropriate switch is pressed, the state machine clears the screen and calls drawScoreboard().

CMPE244 S18 RPS StartScreen.jpeg

Figure 9: Start Screen

drawScoreboard(uint8_t roundNum, uint8_t leftScore, uint8_t rightScore)

This function is used to draw the entire pre-round animation as well as re-draw the in-round score. First, it continually updates the frameBuffer to allow an animation of 'Round' from the bottom to the top of the screen. Once there, it displays roundNum and flashes the round before disappearing for the scoreboard. Regardless of 1P or 2P mode, the players are the same with P1 and P2/CPU. This function calls a drawCountdown() that counts down from 3 to synchronize the recorded player gestures. Afterwards, a drawAction(Action left, Action right) call is made to draw the gestures of each player. A new Action enumeration was made with Rock, Paper, and Scissors as possible options. A variation of the scoreBoard design is shown below. While drawStartScreen simply modifies the entire frameBuffer, drawScoreboard and its subfunctions like drawCountdown, drawScore etc, simply clear their respective sections of the screen and shift pre-drawn bit images to their correct positions.

CMPE244 S18 RPS RoundScreen.jpeg

drawRoundEnd(Player winner)

This function is called after a drawScoreboard call once a winner has been determined. A new Player enumeration was created so the state machine can indicate whether Player1, Player2 or Neither was the winner. This function clears all pixels under the scoreboard and displays a winner animation.

drawGameOver(Player matchWinner)

This function is called when a player reaches the end condition of 3 wins. This function clears the screen and then displays the winner. Slight alterations are made on this screen depending on mode.

Implementation

Game logic

  1. Create tasks for Controllers, Display and Game
  2. Start task scheduler
  3. Call run_stage() virtual function, which calls Rock, Paper, Scissors run_stage()
  4. Beginning at stage 0, run Rock, Paper, Scissors state machine based on stages
  5. Stage 3 will reset stage to 0, where user can press "start" to play again

Bluetooth Gloves

Steps to acquire data from gloves and send it via Bluetooth to controller

  1. Initialize ADC for 2 channels (power to Flex sensors)
  2. Initialize UART2
  3. Read ADC input and write it to UART2
  4. If the data is read successfully and received feedback from the controller, vibrate the indicator motor
  5. Repeat this process for every round

CmpE244 S18 Cumulonimbus Glove Flow.PNG

LED Matrix

Steps to write to LED Matrix:

  1. Set output enable HIGH to disable LED output for selected row
  2. Set latch LOW to block shift register from pushing to parallel data output register
  3. Shift in 32 bits of data for R1, G1, B1, R2, G2, B2 for rows 0/16
  4. Set addr lines A, B, C, D to select rows 0/16
  5. Set latch HIGH to move shift register data into parallel data output register
  6. Set output enable LOW to display currently selected row
  7. Repeat this process for the other rows 100 to 200 times per second

The overall flow is also shown below.

CMPE244 S18 RPS MatrixFlowChart.jpg

Testing & Technical Challenges

In order to get the Smart Rock Paper Scissors working well, lots of testing was required. We faced several difficult technical challenges, as detailed below.

Game Logic

Using tasks within game control logic

In order to implement the game logic in an abstract way, each class was to create its own task and handle its own private members. However, in order for a task to access a previously instantiated class, the class must be in memory that is accessible to the task. When initially creating the classes, they were created in stack space in the main function was not visible to the individual class tasks (for example, the player task could not access its member controller class because the controller class was not pointing to the right space in memory). In order to solve this issue, the classes were initialized in global memory and pointers to the global memory space was passed to various tasks required to run the game.

Bluetooth Gloves

Some technical challenges such as ADC interface, shortage of power source pins and Bluetooth connectivity were faced in the initial phases. They were rectified in the later phases and described below:

Flex Sensor-SJOne board ADC interface
  • The number of Vcc pins on the board are less than required. Thus GPIO pins P0.1 P0.0 are used in High state to use as Vcc source for Flex sensors.
  • Flex sensor requires voltage divider circuit to pull down to ground. 10K resistor is used with each Flex sensor.
  • Ground for 2 flex sensors on one glove is made coommon on a separate GPB due to shortage of ground pins on SJOne board.
Bluetooth pairing
  • Bluetooth module XBee HC06 can not work in master as well as slave modes. Bluetooth module HC05, which supports master as well as slave modes does not conme in Xbee package, which attachment is provided on SJOne board. However, Controller side boards i.e. board connected to RGB led matrix did not have vacant UART pins since used for the matrix. So, usage of 4 HC05 modules is limited. As a solution, 2 HC06 modules on controller side boards(slaves) and 2 HC05 modules, 1 on each glove(masters), are used.
  • Bluetooth module HC05 by default comes in slave mode. It has to be converted to master mode. This conversion is done using AT commands. Then the module is paired with HC06(slave) using its mac address.

LED Matrix

The RGB LED Matrix presented several challenges from both a hardware and software standpoint.

Several of these problems are described below:

SPI vs GPIO

The LED Matrix works by shifting in data available at its six data ports: R1, G1, B1, R2, G2, B2. Early experimentation with an SPI interface involved testing data transfer through a MOSI to B1 and SPI_CLK to SCLK connection. While this worked fine for top half control with one color, more thought had to go into being able to transfer data to the 2 other top-half color data ports as well as the bottom half color data ports. An approach that was tried was connecting MOSI0 to one of the three top-half data ports and MOSI1 to one of the three bottom-half ports to at least have one color control over both halves. Because the matrix takes in a single SCLK input to shift in data for both halves, a high-speed multiplexer was connected to let the SJOne board select the SPI clock that would be shifting in data depending on which half would be receiving data. Because MOSI is not low when idle, different approaches were tried to make sure no junk data was being shifted in when data for the other half was being written. Setting the un-used MOSI port to a GPIO port and then driving it low during the brief interval when the other half was shifting in SPI data proved unsuccessful. Another approach was to feed both MOSI ports into two separate multiplexers so our software could choose either MOSI or ground also proved unsuccessful. Even if either approach worked, a solution would also have to be discovered for how to write data into the 4 other color data ports. A theory was to use the matrix output port that would usually be connected to the input of another matrix display. It was thought that the output of any particular color shift register would go towards the corresponding pin in the output port and that a physical wire could be used to "daisy-chain" the data back into the input of another color register. If this worked, data for all three colors could be sent as three 32-bit words where data could be shifted through the R, G, and B shift registers. However, this ended up not being fruitful. In the end, GPIO was used to build a working API.

With GPIO, the issue of having to write data to 6 data lines with only two MOSI lines was solved by just using 6 GPIO ports and a GPIO clock. Multiplexers to switch between two clocks and other data lines were no longer needed.

Sensitivity

The LED Matrix proves to be extremely sensitive to input and other factors. Early use of the matrix showed flickering and other visual glitches to slight movements of the data lines as well as the power supply line. An initial theory was that this was a software issue and adjustments would have to be made to the control line sequence such as when address lines are changed and when latch is asserted or perhaps timing. Data lines were taped in a particular way as to minimize flickering. However, the flickering always returned after a short amount of time. The cause of this flickering was eventually traced to an unstable power supply. Replacing the unstable supply with a better-built 5V 4A power adapter virtually eliminated all of the display problems that we encountered.

Bitmapping

The images of rock paper and scissors along with other graphics and animations were made using bitmapping. Initially, creating a direct bitmap was tough. The images were drawn using a bunch of drawPixel function calls. The API takes in the coordinates x and y along with the color as parameters. Once the image could be made using this API, the drawPixel function calls were replaced by bitmaps in c. This helped in cleaning the part of the code.

Testing

Bluetooth Gloves

Bluetooth glove testing includes testing of 3 individual components and then testing of glove as a whole.

  • Flex sensors are tested for 45 degree to 90 degree bend of palm for gesture recognition.
  • The resistance and voltage is monitored continuously while checking states.
  • Bluetooth modules were tested by sending and reception of smallest data as single character to Large strings.
  • Bluetooth module pairing was tested recursively by powering on different modules and checked if it gets paired. It should not get paired to any other Bluetooth module other than set mac address.
  • Vibration motor was tested by sending right and wrong indications with small and large time intervals. It should vibrate only on right signal indication.
  • Glove as a whole was tested by capturing Flex values, sending them to slave Bluetooth and vibrate the motor if it gives right indication.

LED Matrix

Testing the LED Matrix independently simply involves an eye inspection of which pixels are lit and their color. Images that were used were drawn out by hand on a paper grid and as such porting them to the matrix display itself involved left shifting these patterns into the specific rows. Typical errors such as off-by-one could be quickly corrected through trial and error inspection. Many errors such as the additional LEDs lighting up could be detected. After testing each and every part, cables and also replacing the board, we could find the error was in the power adapter which was then replaced with a better working one.

System

The system integration was tested by adding one module at a time. The first step was to create the game classes and game control logic. Once the game control logic and classes were in place, the Bluetooth glove was plugged into the abstract controller class to ensure that the Rock Paper Scissors values could be read from the glove. The controller also needed to receive the feedback from the vibration motor, so that was tested in parallel with the Bluetooth connection. The tests we ran into while testing is detailed in the above Technical Challenges section.

After verifying that one Bluetooth glove worked with the system, we integrated the Display logic into the game control code. The key to successful integration of these modules was to understand the interfaces between each module - and by abstracting the classes of the game, it was easy to integrate into the controller and display logic to the game logic.

PCB

The PCB was tested in steps:

1. Ensure none of the components are shorted after assembly (short test with the multimeter).

2. Verify connections on PCB and resistance values with the multimeter.

3. Plug in SJOne to power on PCB and verify board can power on with DC power jack.

4. Test push buttons on PCB to GPIO on SJOne.

5. Connect all display wires and run Rock, Paper, Scissors to verify display works.

6. Connect Bluetooth Glove to SJOne on PCB and verify Bluetooth works with PCB connections.

Lessons Learned

  • PCB components need to be carefully selected and verified.
  • Creating class abstractions can help future-proof code, but make short term implementations more complicated.
  • Bad or noisy power supplies can create enough EMI and signal noise to create significant fluctuations on nearby wires.
  • Some Bluetooth modules only support Slave or Master mode in the firmware. Re-flashing the firmware of Bluetooth modules is a very difficult process, it is better to buy modules that have support for master/slave mode.
  • Using the Repetitive Interrupt Timer on a board with. many tasks can cause strange behavior when the RIT is expected to operate very quickly. If the RIT has too much to do in too little time, the function run by the RIT can operate slightly differently on various iterations.

Conclusion

This project provided a broad application of knowledge learned over the course of this lecture with ADC, GPIO, SPI, UART and RIT used at various points of this project's development. When confronted with many technical challenges during component development as well as integration, several important problem-solving skills were applied to resolve these issues. The Smart Rock Paper Scissors project works fully as intended - both gloves are able to send accurate Rock Paper Scissors commands to the SJOne receiver boards, and the game logic takes these inputs and runs the Rock Paper Scissors game smoothly. The LED display makes creative use of animation and prompts to enhance the user experience of the Rock Paper Scissors game. The tactic feedback of the Bluetooth Gloves helps the user know that their input was received correctly as well as making the game more exciting.

Overall, we are proud to have created a tightly integrated Smart Rock Paper Scissors game. By using a PCB to integrate the receiver boards with the LED matrix, and using gloves, velcro and USB power banks to create wireless gloves, the experience of Rock Paper Scissors has never been more fun.

Project Video

Smart Rock Paper Scissors Video

Project Source Code

Source Code Zip File

References

Acknowledgement

We would like to express our gratitude to Professor Preetpal Kang for generously sharing his time and knowledge with us and guiding us through the completion of this project. We would also like to thank the ISA team for their valuable advice and constructive feedback.

References Used

Flex Sensor guide
Bluetooth HC 05 guide
RGB LED Matrix Guide
RGB LED Matrix Tutorial