F16: Piano Glove

Grading Criteria

• How well is Software & Hardware Design described?
• How well can this report be used to reproduce this project?
• Code Quality
• Overall Report Quality:
  • Software Block Diagrams
  • Hardware Block Diagrams

    Schematic Quality

  • Quality of technical challenges and solutions adopted.

Project Title

OVS: Piano Glove

Abstract

Music is in our everyday lives. Unfortunately, it is expensive to learn how to play an instrument. Therefore, we have decided to create Piano Gloves which can stand as a cheaper alternative to purchasing a Piano. This is a device that generates a realistic piano-key note based on which finger is flexed.

Objectives & Introduction

The objective of this project is to create a glove which can produce a specific piano-key note based which finger is flexed. To accomplish this task, we had to do the following:

• Purchase all necessary parts (as seen in the next section below)
• Simulate piano-key notes in MATLAB using the Karplus Strong Algorithm
• Port MATLAB Code to C using RTOS
• Sew Flex Sensors onto Gloves
• Create our own ADC circuit using OP-AMPS to handle 10 Analog inputs (1 per each finger)
• Use RTOS to write to the onboard DAC on the SJOne Board
• Set up Bluetooth Communication for SJSUOne board and Unity3D
• Create a Real Time visualization based on which finger is flexed
• Perform a full system integration for a finalized product

Team Members & Responsibilities

Melissa Ortiz

• Full System Design
• Digital Signal Processing in MATLAB
• Device Driver Implementation
• Algorithm Development
• System Integration, Verification, and Validation Testing

Benjamin Vuong

• Algorithm Development
• Software Development
• Research on Music Frequencies and Design of Simulated Soundwaves
• Device Driver Implementation
• System Integration, Verification, and Validation Testing

Schedule Overview

Show a simple table or figures that show your scheduled as planned before you started working on the project. Then in another table column, write down the actual schedule so that readers can see the planned vs. actual goals. The point of the schedule is for readers to assess how to pace themselves if they are doing a similar project.

Week #	Start Date	End Date	Team Task Description	Status	Actual End Date
1	10/14/16	10/21/16	Research Design Approach and Assign Tasks	Complete	10/21/16
2	10/21/16	10/28/16	Set up Version Control and Research Parts	Complete	10/28/16
3	10/28/16	11/4/16	Discuss MCU Responsibilities and Communication	Complete	11/4/16
4	11/4/16	11/11/16	Purchase Parts and Begin Prototyping	Complete	11/11/16
5	11/11/16	11/18/16	Status Update: Demo First Round of Prototyping	In Progress
6	11/18/16	11/25/16	Interface Modules and Perform System Design Verification/Validation Testing	N/A
7	11/25/16	12/02/16	Continue Validation and Optimize System	N/A
8	12/02/16	12/20/16	Finalize Design and Prepare for Demo	N/A

Parts List & Cost

Give a simple list of the cost of your project broken down by components. Do not write long stories here.

Item #	Part Description	Vendor	QTY	Cost (USD)
1	SJSUOne Board	Preet	1	$80
2	Flex Sensors	Mouser	10	$6.45
3	Intel Compute Stick	Intel	1	N/A
4	MCP6242-E/P OPAMP	Mouser	10	$0.36
5	3.5mm Audio Jack	Adafruit	1	$0.95
6	Gloves	Northface	1	N/A
7	Misc Parts (Wires, Resistors, Capacitors etc)	Personal	N/A	N/A

Hardware Design & Interface

The design section can go over your hardware and software design. Organize this section using sub-sections that go over your design and implementation.

Hardware Schedule

Discuss your hardware design here. Show detailed schematics, and the interface here.

Week #	Start Date	End Date	Description	Status	Actual End Date	Comments
1	10/21/16	11/4/16	Research Design Implementation and Brainstorm PCB Schematic Layout Determine and Purchase Parts	Completed	11/4/16
2	11/4/16	11/11/16	Design and Interface Schmitt Trigger Circutry w/SJSUOne Board Create PCB Schematic	Completed	11/11/16	We determined that creating a PCB for this project was not necessary given the time constraints.
3	11/11/16	11/18/16	Sew Flex Sensors onto Gloves and Interface w/Development Board Send PCB out for fabrication Status Update: Demo First Round of Prototyping	Complete	11/18/16	Sewing the sensors onto the gloves was tedious. The sensors are very fragile, so they had to be handled with caution.
4	11/18/16	11/25/16	Integrate Hardware and Software, Perform System Design Verification/Validation Testing Implement Bluetooth Module and Interface with additional MCU	Complete
5	11/25/16	12/02/16	Continue Validation and Optimize System	In Progress
6	12/02/16	12/20/16	Finalize Design and Prepare for Demo	N/A

Hardware Development

System Block Diagram

The following diagram is a high-level overview of our system. It is depicts the general structure of our design. In addition to that, it also demonstrates a generalization of the major system components that were used as well as their relationship among each other.

==== Flex Sensors ==== -- replace with current schematic

File:FlexSensorSchematic.png

Figure 1. Flex Sensor Schematic

• How they work

• How we used them

For this project, 10 Flex Sensors were used. Each flex sensor was individually hand-sewn onto a finger of the gloves. These sensors were used to detect finger movement. They accomplish this task because as you flex your finger, the resistance in the flex sensor changes variabley. Once we determined this variable "flex" resistance per each finger, we were able to [to be continued]

Schmitt and Trigger

The Schmitt and Trigger circuit is used to convert the Analog Output of the Flex Sensors into a Digital Signal. This Analog to Digital Conversion (ADC) is done so that the SJSUOne MCU can interpret the input signal using a GPIO port.

• Design Choice: We decided to create our own ADC circuit because of the limited availability on the SJSUOne board. There are simply not enough ADC pins to perform our task, as we would need 10 ADC pins.

Figure 2. General Schmitt Trigger Schematic

Figure 3. Detailed on-chip Schmitt Trigger Schematic (This image shows two Schmitt Trigger Circuits implemented on 1 MCP6242-E/P OP AMP

• How it works --- still editing

The Schmitt and Trigger circuit is a comparator with hysteresis. In general terms, hysteresis is the "dependence of the state of a system based on its history". For this circuit in particular, it aids in preventing unwanted and frequent switching by cancelling out the noise. This is accomplished this by applying a positive feedback into the non-inverting terminal of the differential amplifier. By doing this, we are able to see if the circuit has "triggered" or hit a certain threshold.

SJSUOne Board

Pin Connectivity Table for Glove Sensor Network and MCU -- still editing

OPAMP Pin	SJSUOne Board Pin	SJSUOne Board Pin	HC-05 Pin	Description
TXD3	RXD	Transmission through UART3 to HC-05
RXD3	TXD	Reception through UART3 from HC-05
3V3	3.3	3.3V voltage supply
GND	GND	Ground

Digital to Analog Converter (DAC)

• Design Choice Trade-Off: Due to time constraints and limited funding, we used the on-board 10-Bit DAC instead of purchasing a 12-Bit module. Since the on-board DAC is 10-Bits, we were only able to get 1K distinct values, as opposed to 4K with a 12-Bit. Therefore the Analog Signal that was output to the Audio Speaker was not as precise as we hoped for.

Bluetooth Modules

HC-O5 Bluetooth Module

Pin Connectivity Table

SJSUOne Board Pin	HC-05 Pin	Description
TXD3	RXD	Transmission through UART3 to HC-05
RXD3	TXD	Reception through UART3 from HC-05
3V3	3.3	3.3V voltage supply
GND	GND	Ground

Datasheets

• MCP6242-E/P Operational Amplifier
• Spectra Flex Sensor

System Interface

In this section, you can describe how your hardware communicates, such as which BUSes used. You can discuss your driver implementation here, such that the Software Design section is isolated to talk about high level workings rather than inner working of your project.

Software Design & Implementation

Software Schedule

Show your software design. For example, if you are designing an MP3 Player, show the tasks that you are using, and what they are doing at a high level. Do not show the details of the code. For example, do not show exact code, but you may show psuedocode and fragments of code. Keep in mind that you are showing DESIGN of your software, not the inner workings of it.

Week #	Start Date	End Date	Description	Status	Actual End Date
1	10/21/16	10/28/16	Research the following: Algorithm Design/Implementation and Digital Signal Processing in MATLAB	Complete	10/28/16
2	10/28/16	11/11/16	Develop Karplus Strong Algorithm by incorporating RTOS, Create Device Drivers, and Implement Visualization in Unity	Complete	11/11/16
3	11/11/16	11/16/16	Implement algorithm with DAC for simulation purposes	Complete	11/16/16
4	11/11/16	11/18/16	Status Update: Demo First Round of Prototyping	Complete	11/25/16
5	11/18/16	11/25/16	Implement Code w/Integrated Unit and Begin Debugging As Necessary	Complete	12/02/16
6	11/25/16	12/02/16	Continue System Integration Testing for Validation and Optimization	Complete	12/07/16
7	12/02/16	12/20/16	Finalize Design and Prepare for Demo	In Progress

Audio Algorithm Development

To discuss the audio algorithms, one must understand the notes that are being simulated. Audio frequencies correspond to certain notes. For example, a C4(commonly known as middle C on a piano) has a frequency of 261.63Hz. Sound waves are longitudinal and, simply put, create segments of compressed medium, such as air. A speaker converts an analog value to a sound by vibrating a membrane at a certain frequency, compressing and decompressing air(usually). Therefore, to simulate the note, we have to make the speaker play at that note's specific frequency. For that purpose, the DAC needs to send something similar to a sine wave function into the speaker to get sections of high and low compression.

This Project's Method to Simulating the Frequency

The initial plan of this project was to simulate a complete sine wave with 16 different values in a circular buffer. We would then use the vTaskDelay() function to simulate the space between the sine wave rise and fall increments. However, we found that with the possible tick rates, this led to too little precision in playback frequency. Beyond increasing the tick rate of RTOS, which we ran into issues with, we changed the design choice of using a sine wave. Due to the fact that the speaker needed sections of high and low voltage to create a sound, it was decided that we could send a square wave through the DAC and then use a hardware solution to create the curve of the sine wave, which was a low pass filter. <Insert pictures of sine wave>

Algorithm Flowchart

Karplus Strong

We also decided to use a Karplus strong algorithm, which simulates the fading of a note after is is played. Notes don't go on forever in real pianos, they fade. The algorithm decreases the amplitude by using a circular buffer, containing the wave, and averaging each element to a central value. Eventually, all the elements of the circular buffer would reach that central value and thus the amplitude (or difference between the highest and lowest value) would decrease. Therefore, to make the sound more real we used this algorithm.

Algorithm Diagram

Figure 5. Diagram of Karplus Strong Algorithm

Communication Protocols & Device Drivers

For our implementation, we are using GPIO ports and the onboard Digital-to-Analog Converter (DAC) of the SJOne Board. We also are using Bluetooth for communicating to another board for our visualization. These are described below.

GPIO

The GPIO pins are configured to read the inputs of each finger of the gloves. If a finger is bent, the corresponding GPIO pin changes voltage levels. Upon reading this a key will be played.

Digital-to-Analog Converter (DAC)

The DAC we are using for this project is onboard the SJOne board. The DAC is used to communicate an analog signal with the speakers. The DAC we used has a resolution of 10 bits. It is modified by writing into DACR, which is the register for the DAC. By writing 10 single-bit values into this register, it converts the binary value to an analog value on the output pin of the DAC. This DAC has two sample frequency options, once every 1 microsecond or once every 2.5 microseconds. This is determined by a control bit in the DACR. For the purpose of precision, we decided to set this bit to 0, corresponding to a sample every microsecond.

Implementation

This section includes implementation, but again, not the details, just the high level. For example, you can list the steps it takes to communicate over a sensor, or the steps needed to write a page of memory onto SPI Flash. You can include sub-sections for each of your component implementation.

The steps required to initialize the GPIO pins were:

1. Set the pin configuration to a GPIO
2. Prepare the pin to be an input by setting it to a known value

The steps required to initialize the DAC were:

1. Set the pin configuration to an Analog output
2. Enable the clock for the DAC
3. Write values into the DAC register

Integration Testing, Results, & Technical Challenges

Describe the challenges of your project. What advise would you give yourself or someone else if your project can be started from scratch again? Make a smooth transition to testing section and described what it took to test your project.

Include sub-sections that list out a problem and solution, such as:

Hardware

Problems Encountered #1

• Initially we had ordered the wrong parts. Due to time constraints, we rushed selecting parts and were not careful on selecting the components that had the right packaging. Therefore, we ended up purchasing MSOPs (Mini-Small-Outlined-Package) instead of PDIPs (Plastic Dual Inlined Package). Using MSOPs would not work for us because the IC was simply too small to work with. It would not be able to fit on our breadboard for prototyping purposes.

For more information on IC Packaging and Dimensions, please refer to this link: IC Package Dimensions

Validation

Breadboard Prototype

File:Breadboardtest.PNG

Figure 1. Initial Prototype

• Schmitt and Trigger Output Testing

Before the system was fully integrated, the basic design was implemented on a breadboard. The purpose of this test was to ensure that the Schmitt and Trigger circuit was functioning properly. In other words, we wanted to validate that the output voltage changed according to how much each flex sensor was flexed.

File:Breaboardtest2.PNG

Figure 2. Flex Test

As seen in Figures 1 and 2, we have two Flex Sensors connected to an MC6242 OP AMP. Initially the outputs, VOUTA and VOUTB, are both output 0v which causes the output LEDs to be on. The output LEDs remain on until the Flex Sensors are flexed to a certain degree. When this happens, the resistance on the Flex Sensor increases and causes the input to the V+ terminal to increase. Once the terminal V+ passes the set threshold voltage compared to terminal V-, VOUTA or VOUTB (depending on which sensor is flexed), will produce 3.3v. At this time, the output LED will turn off.

• NOTE: Since each flex sensor was different, the degree of flexing was different for each sensor. Therefore, each Flex Sensor had it own specialized resistor in order to achieve the proper voltage level through the voltage divider that we created.

• Oscilloscope Readings

Partial Integration

• GPIO Testing

In addition to testing the output voltages, the circuitry on the breadboard was interfaced with the SJSUOne MCU. Once the above output testing was complete, we wanted to make sure that the ADC we made was also functioning properly. Therefore, we connected the output terminals, VOUTA and VOUTB, to a GPIO Pin on the SJSUOne MCU to make sure that it could now read a Digital value. After we had configured the ports to be GPIO Inputs, we began the test. During this test, we read continuously the Logic Level on the GPIO Input Pins. Once the the Logic Level switched from 0 to 1, an on board LED would light up to indicate that a finger had been flexed. Thus, when the on board LED 1 was on, the meant that the output voltage was at Logic 0. When the on board LED 3 was on, that meant that the Logic value read at the GPIO input pin was a Logic 1.

• Bluetooth Integration

Problems Encountered #2

Discuss the issue and resolution.

Software

DSP Simulations

In order to test the playing of notes through the DAC, we had to connect a speaker (or earphones). Therefore, we created a simple low-pass filter circuit that connected to a pair of Apple earphones. The image below shows how we connected it. The pin of a standard Apple headphone has mic, ground, right, and left. In the figure below, we have the green wire connecting to ground and the red wire connecting to right and left. The electrical tape is present to prevent shorts.

File:PianoGloveInitialPrototype.jpg

Figure 5: Circuit initially used to test sound generation.

Problems Encountered #1

• Initially, we had to figure out how a speaker worked and how to simulate a specific note on a speaker. As mentioned in the previous section, a speaker works by vibrating a membrane according to an analog voltage value. We figured that we needed to oscillate the speaker membrane at a specific frequency per note (as shown below). At that point, we needed to figure out which notes corresponded to which frequencies. After this, we had to figure out how to make the membrane move in a fashion is similar to a note's specific oscillation frequency. Realizing that this could be simulated by a sine wave, we wrote a similar function into the DAC to simulate each note. Several notes and frequencies are listed below.

File:NotesFrequencies.png

Figure 6. Sample of notes and frequencies in Hz. Taken from this site

Problems Encountered #2

• The precision of the frequencies was a significant limiting factor. If we were to use to RTOS default tick rate of 1ms (which in terms of frequency would be 1kHz), the number of audible notes we could simulate would have been very limited. Therefore, the tick rate had to be increased. We tried to implement a 100kHz tick rate, which would increase precision by a factor of 100. However, this seemed to crash the board. Therefore, it was decided that about 44kHz was a sufficient tick rate to simulate the frequencies we required. A discovery made during this experimentation was that the changing of tick rates also requires the disabling of the periodic scheduler task, which resets the microcontroller anytime the tick rate is not 1kHz.

Conclusion

Conclude your project here. You can recap your testing and problems. You should address the "so what" part here to indicate what you ultimately learnt from this project. How has this project increased your knowledge?

Hardware & Software Tradeoffs

• DAC Precision (10-bit vs 12-bit)
• Sampling Frequency (11KHz vs 44KHz)
• Wave (Sine Approximation vs Square)
  • Sine offered smooth transition
  • Square wave offered precision and allowed for filter to do the curving of the wave

Project Video

Upload a video of your project and post the link here.

Project Source Code

• Sourceforge Source Code Link

References

Acknowledgement

Any acknowledgement that you may wish to provide can be included here.

References Used

List any references used in project.

• Digital Synthesis of Plucked-String and Drum Timbres
• Flex Sensor Basics
• Commuted Piano Synthesis
• Piano Sounds Synthesis
• The Gloves Project
• PHL Gloves
• University of Texas on DAC and sound
• Notes and Frequencies
• Karplus Strong

Appendix

You can list the references you used.