Difference between revisions of "F20: Jubeat"
Proj user9 (talk | contribs) (→Capsense Touch Task) |
Proj user9 (talk | contribs) (→12 Square Tasks) |
||
Line 464: | Line 464: | ||
* Does nothing | * Does nothing | ||
− | == 12 Square Tasks == | + | === 12 Square Tasks === |
Each of the 12 squares in the 4x3 grid uses its own task to easily handle multiple animations simultaneously. Upon receiving a signal from the main game loop task, the task will perform its animation utilizing another semaphore. This second semaphore will be sent by the Capsense Touch Task to interrupt the animation and display a judgement (Perfect/OK). If the animation is not interrupted, this is a miss. | Each of the 12 squares in the 4x3 grid uses its own task to easily handle multiple animations simultaneously. Upon receiving a signal from the main game loop task, the task will perform its animation utilizing another semaphore. This second semaphore will be sent by the Capsense Touch Task to interrupt the animation and display a judgement (Perfect/OK). If the animation is not interrupted, this is a miss. | ||
[Square State diagram] | [Square State diagram] |
Revision as of 13:12, 17 December 2020
Contents
- 1 Game photo
- 2 Introduction
- 3 Objectives & Introduction
- 4 Schedule
- 5 Parts List & Cost
- 6 Design & Implementation
- 7 Hardware Design
- 8 Schematic
- 9 Hardware Interface
- 10 Game Software Design
- 11 Testing & Technical Challenges
- 12 Conclusion
- 13 References
Game photo
Introduction
Jubeat is a music game uses an arrangement of a 3x4 grid for the game play. An animation pops up on each grid according to the beat of the song. The player has to use his fingers to tap on the screen and catch the animation in time. The scoreboard locates on the top row of the LED display. Different scores will be given depending on when the player catch the animation. Tapes can be judged as PERFECT, GOOD, and MISS.
Objectives & Introduction
This system:
1. Recognizes finger touch as the input to record the score.
2. Supports real time capacitive touch detection on a LED matrix display.
3. Update corresponding score on the LED display.
4. Outputs the audio via a mp3 module and the soundtracks are stored in an SD card.
Team objectives
1. Understand how each sensor/module works by studying the datasheets to develop an actual product.
2. Understand the proper use of freeRTOS semaphores and event groups in order to send/receive the data between multiple tasks.
3. Integrate modules and develop smooth game logic. Identify each game state and its transition to next state.
4. Learn how to cooperate codes and logic written by team members through git and branch control.
Team Members & Responsibilities
- Ka In Ng
- Git Admin
- Tyler Tran
- Srikar Narapureddy
Schedule
Week# | Date | Task | Status |
---|---|---|---|
0 | 10/13/2020 |
|
|
1 | 10/21/2020 |
|
|
2 | 10/28/2020 |
|
|
3 | 11/4/2020 |
|
|
4 | 11/11/2020 |
|
|
5 | 11/18/2020 |
|
|
6 | 11/24/2020 |
|
|
7 | 12/1/2020 |
|
|
8 | 12/8/2020 |
|
|
Parts List & Cost
Part Name | Part Model & link | Quantity | Cost Per Unit |
---|---|---|---|
Evaluation Board | SJ2 board (NXP LPC4078 MCU) | 1 | $50 |
64x64 RGB LED Matrix | 3mm Pitch - 192mm x 192mm | 1 | $59.95 |
5V Power Supply | Aclorol 5V 5A 25W AC-DC | 1 | $13.19 |
12-Key Capacitive Touch Sensor | Adafruit MPR121 | 1 | $7.95 |
ITO (Indium Tin Oxide) Coated PET Plastic | 100mm x 200mm | 2 | $9.95 |
DFPlayer Mini | MakerHawk DFPlayer Mini | 2 | $3.99 |
Speaker | 3" Diameter - 4 Ohm 3 Watt | 2 | $1.95 |
Design & Implementation
The design section can go over your hardware and software design. Organize this section using sub-sections that go over your design and implementation.
Enclosure Design
The enclosure is made to divide the The enclosure is made with 4 goals in mind.
- Divide the LED display into a display area and 4x3 grid
- Route wires cleanly
- Hold ITO plastic squares in place
- House hardware
After taking measurements, the first goal is easily met by drawing out a simple grid that conforms to the size of the LED matrix. Clean wire routing is done by creating grooves in the sides and on the grid of the enclosure, where 30-gauge wire is glued in place. Extrusions are placed all around the top part of the enclosure, at the same thickness of the ITO squares, allowing for a friction fit. Lastly, the base of the enclosure contains a large open area to contain all hardware.
The enclosure represented a large portion of the issues faced during design, namely the connections from the wire to the ITO. The initial aim was to friction fit the wires against the ITO. While this was successful most of the time, 100% reliability is a necessity. Some less-reliable connections are supplemented by copious amounts of superglue and some tape.
Hardware Design
//Discuss your hardware design here. Show detailed schematics, and the interface here.
LED Matrix
A 64x64 RGB led matrix display is used in this project, The display has a scan rate of 1:32 and uses six 64-bit shift registers.
- The register shift inputs R1,G1,B1 and R2,G2,B2 are used to store the Red ,Green and Blue states for the two rows .i.e 128 LEDs from the top and bottom halves of the display.
- The rows are multiplexed using the 5 row pins A,B,C,D,E.
- The scanning is controlled by Output enable(OE) and Latch(LAT) pins.
- OE is used to disable the leds when changing the row during scanning.
- LAT is used to latch in the row data after the shifting is complete.
In total, 14 GPIO pins are used to drive the display.
Technical details
1. Brightness: 2800cd/square meter.
2. Size: 192mm x 192mm.
3. Pitch: 3M.
4. 5 Addressable Pins.
5. Scan: 1/32.
6. Refresh Frequency: >=400HZ.
7. Weight: 235g.
Power: The led matrix needs an external power supply capable of supplying at least 8A current at 5V.
Capacitive touch breakout
The capacitive touch sensor input pins are wired to 12 pieces of transparent ITO films which are placed on top of each section of the LED matrix display. When the game starts, the song stores in the SD card of the mp3 module starts to play. Animation shows up on any of the 12 LED sections and player has to catch the animation on time to receive scores. The breakout board consists of MPR121 IC which uses I2C to communicate. It supports four I2C addresses and each address can be used to detect 12 individual touches.
mp3 module
Schematic
The following schematic shows the wiring of all components.
Hardware Interface
In this section, you can describe how your hardware communicates, such as which BUSes used. You can discuss your driver implementation here, such that the Software Design section is isolated to talk about high level workings rather than inner working of your project.
LED matrix Driver
Since, it is very inefficient to light up all the 4096 leds at the same time, The images/frames are shown on the display through a process called scanning where only a few rows expressed by its scan rate are driven at the same time. For the scan rate of 1:32 in a 64x64 led matrix, each row from the top half and the bottom half are displayed serially.
For each row of pixels, we repeat the following cycle of steps:
- Clock in the data for the current row one bit at a time
- Pull the latch and output enable pins high. This enables the latch, allowing the row data to reach the output driver but it also disables the output so that no LEDs are lit while we're switching rows.
- Switch rows by driving the appropriate row select lines.
- Pull the latch and output enable pins low again, enabling the output and closing the latch so we can clock in the next row of data.
The pixel data is put in a global buffer in RAM, this buffer is periodically accessed by the display task to refresh the display.
Code Snippet:
while (1) {
for (int row = 0; row < 32; row++) {
// left half
for (int x = 31; x >= 0; x--) {
logic = ((led_matrix[0][row][0] >> x) & 1);
gpio__set_logic(R1, logic);
logic = ((led_matrix[1][row][0] >> x) & 1);
gpio__set_logic(G1, logic);
logic = ((led_matrix[2][row][0] >> x) & 1);
gpio__set_logic(B1, logic);
logic = ((led_matrix[0][row + 32][0] >> x) & 1);
gpio__set_logic(R2, logic);
logic = ((led_matrix[1][row + 32][0] >> x) & 1);
gpio__set_logic(G2, logic);
logic = ((led_matrix[2][row + 32][0] >> x) & 1);
gpio__set_logic(B2, logic);
gpio__set(CLK);
gpio__reset(CLK);
}
// right half
for (int x = 31; x >= 0; x--) {
logic = ((led_matrix[0][row][1] >> x) & 1);
gpio__set_logic(R1, logic);
logic = ((led_matrix[1][row][1] >> x) & 1);
gpio__set_logic(G1, logic);
logic = ((led_matrix[2][row][1] >> x) & 1);
gpio__set_logic(B1, logic);
logic = ((led_matrix[0][row + 32][1] >> x) & 1);
gpio__set_logic(R2, logic);
logic = ((led_matrix[1][row + 32][1] >> x) & 1);
gpio__set_logic(G2, logic);
logic = ((led_matrix[2][row + 32][1] >> x) & 1);
gpio__set_logic(B2, logic);
gpio__set(CLK);
gpio__reset(CLK);
}
gpio__set(OE);
gpio__set(LAT);
gpio__set_logic(A, (row & (1 << 0)) >> 0);
gpio__set_logic(B, (row & (1 << 1)) >> 1);
gpio__set_logic(C, (row & (1 << 2)) >> 2);
gpio__set_logic(D, (row & (1 << 3)) >> 3);
gpio__set_logic(E, (row & (1 << 4)) >> 4);
gpio__reset(LAT);
gpio__reset(OE);
}
vTaskDelayUntil(&xLastWakeTime, xFrequency);
}
}
Audio Driver
The audio driver is used to play the sound tracks that stored on a SD card of the DFPlayer mp3 module.
1. The DFPlayer mp3 module is connected to the SJ2 board via I2C. Each command takes 10 UART data transmissions.
2. When the board is turned on, the function audio_init will be called and get ready.
3. The audio_specify_track function is called after the player choose which track to play using the capacitive touch game buttons.
4. Then the audio_start_play function is being called right after.
5. Each audio command contains a 2 byte checksum to maintain the data integrity.
Code Snippet:
void send_audio_command(uart_e uart, uint8_t audio_command, uint8_t param1, uint8_t param2) {
uint16_t checksum_2byte = audio_calculate_2byte_checksum(audio_command, param1, param2);
uint8_t checksum_lowbyte = checksum_2byte;
uint8_t checksum_highbyte = (checksum_2byte >> 8);
fprintf(stderr, "checksum = 0x%x, checksum_lowbyte = 0x%x, checksum_highbyte = 0x%x\n", checksum_2byte,
checksum_lowbyte, checksum_highbyte);
uint8_t audio_full_command[10] = {start_byte, version_byte, command_length, audio_command, feedback,
param1, param2, checksum_highbyte, checksum_lowbyte, end_byte};
for (int i = 0; i < 10; i++) {
uart__polled_put(uart, audio_full_command[i]);
}
}
Capacitive touch logic
The capacitive touch sensor is used to identify the finger touch on the ITO button which is transparent and conductive.
1. Each capacitive touch sensor input pin is connected to a ITO film that acts as the game buttons. There are 12 of them throughout the game.
2. When any of the 12 button is being pressed, an interrupt is triggered and we find out the pin that caused the interrupt.
Code Snippet:
int8_t get_touching_pin(void) {
uint8_t i; // i = touching pin number, pin released when i = -1;
currently_touch = (i2c__read_single(I2C__2, 0xB4, 0x01) << 8); // the MSB 4 input pins (pin 11 - 8)
currently_touch += i2c__read_single(I2C__2, 0xB4, 0x00); // add the LSB 8 input pins (pin 7 - 0)
currently_touch &= 0x0FFF; // remove 4 MSB for slave register 0x01
for (i = 0; i < 12; i++) {
if ((currently_touch & (1 << i))) {
fprintf(stderr, "touching pin %d \n", i);
return i;
}
}
return -1;
}
Game Software Design
The game’s software design consists of:
- Main game loop task
- Equalizer display task
- Score display task
- Capsense touch task
- 12 Tasks for each square in the 4x3 grid.
Main Game Loop Task
The responsibilities of the main game loop are to handle all menus, the state of the game and loading in the gameplay events (timing and position of the gameplay touch events). [Need State Diagrams here]
Start Screen
Simply uses a semaphore triggered by any button to move to the next state
Song Selection
Utilizes an event group to distinguish between button presses. Actions vary depending on which button is pressed.
- Increment/Decrement Song Selection
- Start Song
Gameplay
Once gameplay has begun, the task sends signals the spectrogram to start and loads in song events from a specified text file, as shown in the following diagram: [Gameplay state diagram]
End Screen
Score is displayed at the end with a count-up animation. High score is also displayed in different colors depending on the final scores compared to the high score. High score is written to a file if a higher one is met. Tapping any square will return to the start screen.
Equalizer Display Task
The equalizer displays only a single range of frequencies, roughly 0-350Hz, representing the bass. The values displayed are not evaluated in real time, rather, using a python script, the values are sampled from the audio file at a 30ms interval and are saved to a file and placed on the SD card. The file is then read by the SJ2 board at a 30ms interval to display an equalizer at the top of the game while playing. [Equalizer state diagram]
Score Display Task
The score display task simply updates the score at the top of the screen during gameplay.
Capsense Touch Task
This task handles determining which capsense input is detected, and sends semaphore signals or event group signals based on the current game state.
State = Start Screen
- Sends a semaphore
State = Song Selection
- Sets bits on event group based on which squares are pressed
State = Gameplay
- Compares timing of press to gameplay events currently being played
- Sends semaphore signal or not depending on previous evaluation
- Based on timing, will judge press as perfect or not and update scores.
State = End Screen
- Sends a semaphore
State = Other (Mid-transition animations)
- Does nothing
12 Square Tasks
Each of the 12 squares in the 4x3 grid uses its own task to easily handle multiple animations simultaneously. Upon receiving a signal from the main game loop task, the task will perform its animation utilizing another semaphore. This second semaphore will be sent by the Capsense Touch Task to interrupt the animation and display a judgement (Perfect/OK). If the animation is not interrupted, this is a miss. [Square State diagram]
Implementation
The 64x64 matrix display is divided into 4 rows, the top row (16x64) is used as a scoreboard, and the remaining rows are divided into 12 square regions, each 16x16 that connects to the ITO buttons that connects to the capacitive sensor for the game input.
Animation Design
The group put a heavy focus on animations and animation quality, notably during gameplay. For this reason, each group member had a hand in designing and deploying animation with a wide variety of approaches and methodologies. [I don’t know how we want to talk about this]
Testing & Technical Challenges
Describe the challenges of your project. What advise would you give yourself or someone else if your project can be started from scratch again? Make a smooth transition to testing section and described what it took to test your project.
Include sub-sections that list out a problem and solution, such as:
Wiring
While this isn't a software bug, it's a physical one with software implications. The connections from the wire to the ITO coated plastic are somewhat inconsistent, resulting in both misfires and missed inputs. This hardware bug would have been easily resolved with conductive adhesive, and if given the opportunity, we would use that instead of friction fitting. However, these issues became highly present too late into development to consider the use of conductive adhesive, so steps were taken to resolve the issue in software.
The capsense peripheral has the ability to automatically adjust and tune itself to whatever is connected to its input pins. This causes a major issue in the case of inconsistent wiring as an accidental touch to an exposed wire, or a poor connection can potentially render an input square near unusable. These issues from automatic calibration additionally may not happen immediately, or it may take some time to cause major issues. We noticed that some inputs would cease functioning after playing multiple songs in a row, but would be fine after simply resetting the SJ2 board. Our resolution is to reset the capsense peripheral after the end of every song as a preventative measure against poor automatic calibration.
<Bug/issue name>
Discuss the issue and resolution.
Conclusion
Conclude your project here. You can recap your testing and problems. You should address the "so what" part here to indicate what you ultimately learnt from this project. How has this project increased your knowledge?
Project Video
//youtube video link
Project Source Code
References
Acknowledgement
Any acknowledgement that you may wish to provide can be included here.