Embedded Systems Learning Academy - User contributions [en]

S14: FaceTime Robo

2014-12-25T08:44:37Z

Himanshu: /* Project Video */

== Project Title ==
FaceTime Robo

== Abstract ==
FaceTime Robo (FTB) is a robot which helps in video chat. With the help of FTB, the user can do video chat without holding cell phone in hand. Now user can do multitask, while chatting with their loved ones. FTB works on face detection algorithm.

== Objectives & Introduction ==
FTB has three different modes.
The first mode is Normal mode. In normal mode, FTB just tracks user in the room. If user suddenly disappears from FTB range of view, then it would not be able to follow up with user. Therefore, second mode is developed. The second mode is called autonomous search mode. When FTB is set in autonomous search mode, then it keeps searching for user in the room. On finding the user, FTB automatically resumes normal mode. FTB also has voice recognition feature built in. FTB wakes up/sleep on user vocal command. The Third mode is sound localization mode. In this mode, user just calls FTB name and FTB turns into that direction and starts searching for user. The FTB uses It’s artificial intelligence to determine the direction of sound and turns into that direction.

[[File:Cmpe244_S14_FaceTimeRobo_CompleteSystemFlow.jpg]]

=== Team Members & Responsibilities ===
* Team Member
** Himanshu Saini
*** Hardware Design
*** Driver Development
*** Software Design
*** Integration and Test
*** Project Documentation

== Schedule ==

{| class="wikitable"
|-
! scope="col"| Week#
! scope="col"| Date
! scope="col"| Task
! scope="col"| Actual
|-
! scope="row"| 1
| 02/27/2014
| Rough Estimation of parts required in project
| Done
|-
! scope="row"| 2
| 03/06/2014
| Parts Ordered
| Done
|-
! scope="row"| 3
| 03/13/2014
| PWM and ADC Driver
| Done
|-
! scope="row"| 4
| 03/20/2014
| Individual Testing of each parts
| Done
|-
! scope="row"| 5
| 04/03/2014
| Implementation of face recognition algorithm
| Done
|-
! scope="row"| 6
| 04/10/2014
| Windows driver development
| Done
|-
! scope="row"| 7
| 04/17/2014
| Integration of project with RTOS
| Done
|-
! scope="row"| 8
| 04/24/2014
| Sound localization testing
| Done
|-
! scope="row"| 9
| 05/01/2014
| Complete testing of system starts
| Done
|}

== Parts List & Cost ==
The table below summarizes the parts used and the cost for the project.

{| class="wikitable"
|-
! scope="col"| Ref Number
! scope="col"| Part Number
! scope="col"| Description
! scope="col"| Qyt
! scope="col"| Cost
|-
! scope="row"| 1
| SJ one Board
| SJSU development board, LPC1758
| 1
| $75
|-
! scope="row"| 2
| Servos
| For Pan/Tilt Mechanism
| 2
| $40
|-
! scope="row"| 3
| Microphones
| For Sound Localization
| 2
| $25
|-
! scope="row"| 4
| Webcam
| Microsoft Webcam
| 1
| $25
|-
! scope="row"| 5
| Wires
| Length: 0.5', 0.3', and 1.00' feet
| 25
| $35
|-
! scope="row"| 6
| Acrylic sheets
| 1 X 1 feet
| 1
| $30
|-
! scope="row"| 7
| USB to Serial Converter
| Converter
| 1
| $20
|-
! scope="row"| 8
| Other Miscellaneous Stuffs
| Stands and Screws
| 15 Screws and 8 Stands
| $30
|-
! scope="row"| 9
| Brackets
| For Pan/Tilt
| 2
| $10
|-
! scope="row"| 10
| Velcro
| Size 1 X 1 feet
| 1
| $20
|}

== Design & Implementation ==

=== Hardware Design ===

[[File:Cmpe244_S14_FaceTimeRobo_CompleteProject.jpg]]

The hardware design of FTB is consists of three layers. The very top layer of FTB has two servo motors, which are placed in such a way so that pan/tilt mechanism can easily be obtained. Bottom servo rotates from left to right. Top servo rotates up and down. The top servo has a bracket attached which has a Velcro on it. The reason of putting Velcro on bracket is to easily place or remove cell phone from FTB. The top layer of FTB also has a camera attach to the bracket. The camera also has inbuilt microphone. There are two more separate microphones on the top layer for sound localization.

The second layer of FTB is consists of breadboards, switches, power supply.

The third layer of FTB is consists of SJ One Board.

=== Hardware Interface ===
The FTB robot uses following interfaces:
* A/D Converter
** Analog to Digital converter from sound sensors.
* Windows USB to Serial Driver:
** To send the data from PC to SJ One Board.
* Pulse Width Modulation (PWM)
** PWM interface to move servos preciously
* GPIO
** Switches to enable/disable Autonomous Search Mode and Sound Localization Mode

[[File:CMPE244_SP14_FaceTimeRobo_Hardware_Interface.jpg]]

=== Software Design ===
The software is divided into two parts. The first part of the software is running at the PC end and the second part of the software is running at the SJ One Board.

The first part of the software running at the PC requires the face detection algorithm. The software calculates frame per second and the size of the picture frame. On calculating the X and Y axis of the picture frame, the data is send to the SJ One Board.

[[File:Cmpe244_S14_FaceTimeRobo_SoftwareFlowInPC.jpg]]

The second part of the software running at SJ One Board requires initialization of PWM and ADC driver. The general flow of software is shown in figure. The software has four tasks. The first task controls “Normal Mode” of the robo. The second task controls “Autonomous Search Mode” of the robo. The third task controls “Voice recognition” of the robo. The fourth and final task controls “Sound Localization Mode” of the robo.

The first task controls “Normal Mode”. As the task begins executing, it waits for the serial data from the PC. On receiving the data from PC, LPC 1758 calculates the values for servos and updates the servos.
The second task controls “Autonomous Search mode”. The task two executes when the autonomous search mode switch is turned on. The task two keeps updating servos with new position until unless it finds out the user. The moment tasks two find the user, it automatically switches to “Normal Mode”.

The third task controls voice recognition feature. Whenever the voice command matches to the pre-stored voice commands, then the PC sends data to third task and it takes corresponding actions.

The fourth and final task controls “Sound Localization Mode”. The fourth task executes when the sound localization switch is turned on. The microphone starts receiving data and determines the direction of the sound. Once the direction of the sound detected, the LPC 1758 updates the new position of the servos.

[[File:Cmpe244_S14_FaceTimeRobo_SoftwareFlow.jpg]]

=== Implementation ===

'''Microcontroller to Servo'''
The pulse width modulation driver was written to make the communication between SJ1 board and serovs. When SJ1 board boots up, it initializes PWM driver. Then after, the servo position task keeps updating the position of the servo as instructed by microcontroller.

'''Microcontroller to Sound Sensors'''
The analog to digital converter driver was written to make the communication between SJ1 board and sound sensors. The sound sensors are analog sensors. When SJ1 boots up, it initializes A/D driver. When “sound localization” mode is selected, then A/D task keeps collecting the data from sound sensors and updates the microcontroller by sensors. Once the sound sensors cross threshold level, then microcontroller compares the reading of all sensors and takes appropriate actions.

'''Computer to Microcontroller'''
The face detection program running at PC calculates the position of user and sends the data to microcontroller through an interface. The interface is designed using c++. The interface is a windows driver which forwards the data from face detection program to serial COM port. On the other hand, the SJ1 microcontroller receives data through serial communication, UART.

== Testing & Technical Challenges ==
There were several testing and technical challenges, I faced while building this project. There are some challenges which can be avoided very easily by future students, such as
* Have a good soldering iron. It is very important to have a good soldering iron. If wires are not soldered properly, then it would be hard to debug the hardware. Good connections between microcontroller and peripheral will save a lot of hardware debugging time.
* Have a good Oscilloscope. An oscilloscope is very important tool, if you want to build a project like FTB. A good quality of oscilloscope will help you to debug in Pulse Width Modulation. Please do not buy cheap oscilloscope just to save couple hundred dollars. It will help you in a long run.
* Please join a workshop like a “Techshop” to get the right cutting edge tools. If you are a hobbyist and you want to build project from scratch, do not use “Dremel” and other cutting tools at home until unless you are expert. If you do not know how to use cutting tools, then you will waste lot of time and money.

=== My Issue #1 ===
The biggest issue was designing the system. It took me approx 7 to 10 days to draw the system on paper. If your design has some loop hole, then it would be hard to implement and it will consume your lot of time. For example, I have three layers in my prototype, such as top layer has servos and camera, middle layer has all connections and last layer has SJ1 board.
I wanted to design system in such a way that it would be easier to separate layer from each other, if something goes wrong in any layer.

=== My Issue #2 ===
Another biggest issue was designing interface between computer/laptop to FTB. The interface is responsible to collect the data from laptop and send to SJ1 board. It was a main key of the project. If I would be unable to take out the data from laptop and feed it into SJ1 board, then it would be tough to give life to FTB.
Fortunately, I learned windows driver programming in the past. It helped me to design an interface between the computer and SJ1 board.

=== My Issue #3 ===
The another issue was the huge amount of memory consumption by face detection program running on laptop. Face detection program was consuming memory more than 3GB due to some program errors and then it crashes. Therefore, I increased RAM from 4GB to 12GB. However, the problem was same. The consumption of RAM was still 3GB and then crash. Later on, I found out there was some problem in memory allocation in the program. It took a lot of time to figure out the problem.

== Conclusion ==

The FaceTImeRobo was fun project to work on. Through this project, I achieved lot of knowledge about face detection technology. I also learned how to develop windows driver, PWM driver, A/D driver, and GPIO. This project also helped me to learn different ways of testing project, so that it will work flawlessly in real time environment. It was a great experience to write C++ code for ARM Cortex-M3 and later on integrate with real time operating system (FreeRTOS)

If there was more time, I would like to implement the following additions:

* Facial recognition technology so that robot can lock itself to a particular user.
* I wanted to add speaker, so that robot can talk to user.

=== Project Video ===
https://www.youtube.com/watch?v=sDsf6LbdYqA&list=UUmTKr2RNmUsWPyTO_T6kgNg

=== Project Source Code ===
* [https://sourceforge.net/projects/sjsu/files/CmpE244_SJSU_S2014/ Sourceforge source code link]

== References ==
=== Acknowledgement ===
I am very thankful to Professor Preetpal Kang for
* Teaching advance micro-processor.
* Teaching Real Time Operating System (FreeRTOS)
* Helping me in building FaceTime Robo : Darwin

=== References Used ===
* [http://www.nxp.com/documents/user_manual/UM10360.pdf LPC1758 User Manual]
* [http://www.freertos.org/ FreeRTOS]
* [http://opencv.org/ Opencv ]

PING))) Ultrasonic Sensor

2012-06-30T06:04:35Z

Himanshu:

== Overview ==
The PING))) Ultrasonic works by sending sound waves and then the sensor signal emits a PWM that needs to be timed to detect the distance. The PWM width depends on the time it takes for the sound waves to bounce back from an object.

== Interface Details ==
The interface of this sensor is simple, just Power, Ground, and PWM Signal, but this signal wire is multiplexed between input and output. Here are the steps needed:

* Connect to your CPU Pin, such as P0.X
* Configure P0.X as output, and give a 20uS Pulse of HIGH then LOW
* Immediately configure P0.X as input
* Take a snapshot of a hardware timer or start a hardware timer
* Enable P0.X as falling edge interrupt
* --> After some time <--
* P0.X will go HIGH, and then LOW and the width of this HIGH wave corresponds to the object distance
* Interrupt on falling edge will now trigger, so compute the delta of the hardware timer
* Disable falling edge interrupt otherwise when you trigger this sensor again (bullet 2 above), the interrupt will fire unintentionally.

== LPC2148 FreeRTOS Hints ==
* Since FreeRTOS uses Timer0, simply use Timer1
* The FreeRTOS sample code provided already starts TIMER1, so:
* When you want to compute the delta, save the timer value when you trigger the PING))) Sensor by:
*: <code>unsigned int timerValueAtTrigger = portGET_RUN_TIME_COUNTER_VALUE();</code>
* When the ISR occurs, compute the delta:
*: <code>unsigned int timerValueAtIsr = portGET_RUN_TIME_COUNTER_VALUE();</code>
*: <code>unsigned int timerDelta = (timerValueAtIsr - timerValueAtTrigger);</code>
* At TIMER1_PC=32 (default), each tick = 1/(48Mhz/32) = 666ns, so:
*: <code>unsigned int deltaInMicroSec = timerDelta * 0.666;
* Now use the PING))) datasheet to convert Micro-seconds to inches.

== '''Timer/Capture Based PING sensor:''' ==

=== '''Things need to be done before start interfacing PING sensor with LPC2148:''' ===
: 1. Read chapter 15, Timer/Counter, from the datasheet of LPC2148.
: 2. Make sure the PING sensor is digital one.
: 3. Make sure do not use TIMER0 because it is used by FreeRTOS.
: 4. Select pin of LPC2148 that can be set for Capture 1, for example Pin 10.
=== '''Pin description of PING sensor''' ===
There are three pins in digital PING sensor.
a) 5v pin b) Signal Pin c) Ground
=== '''Points to be noted:''' ===
: 1. FreeRTOS starts TIMER1 and TIMER0 by itself.
: 2. OR you can also initiate TIMER1 by yourself.
: 3. OR you can set only T1CCR register (Atleast).
=== '''How to Write TIMER1 driver?''' ===
: 1. First enable the timer reset by setting bit 1 as ‘1’ in TITCR.
: 2. Then set the Prescale Register (T1PR) according to number of PCLK you want to count before incrementing Timer Counter (T1TC). Once TITC increments T1PC (Prescale Counter Register) is reset. You can set T1PR to zero so that T1TC will increment on every PCLK.
: 3. The third register need to set is Capture Control Register (T1CCR). This register can be set for one of the following functions:
a. Capture on Rising edge b. Capture on Falling edge c. Interrupt on Capture pin CAPn.0
: 4. Finally, enable the TIMER by enabling bit 0 in T1TCR
=== '''Working of PING sensor''' ===
In order to activate the sensor, the host microcontroller (LPC2148) will send a short pulse of duration 20 microseconds on signal pin. Just after the pulse will end, set the signal pin as input. Then, at that moment take the snapshot of the PCLK through T1TC register. Snapshot means note down the time at point X. Initialize the register T1CCR so that when falling edge will come then T1CCR register store the time in T1TC register, which will be at point Y. Now, subtract point X from Y.
Find out the HOLDOFF time from the datasheet of PING sensor. Mostly, it is around 750us. HOLDOFF is the time between the PING sensor receives activation signal from host microcontroller and PING sensor emits ultrasonic wave to detect object. Suppose, you get Z by subtracting X from Y. Then, subtract 750us from Z.
Z = Y – X
A = Z – 750us
A’ is the time taken by wave to go and come back; therefore the one side distance will be ‘A’/2 .
B = A/2
Now, divide ‘B’ by 73.746 to get the distance in inches. The value 73.746 is taken according to the Parallax digital PING sensor. It could vary for another PING sensor; therefore, please check the datasheet of your PING sensor.
[[File:Timer_Based_PING_Sensor1.JPG|700px|thumb|center|alt=Figure 1: Working of Ping Sensor |Figure 1: Working of Ping Sensor]]

File:Timer Based PING Sensor1.JPG

2012-06-30T06:00:25Z

Himanshu: Working of PING Sensor

Working of PING Sensor

File:Timer Based PING Sensor.JPG

2012-06-30T05:55:38Z

Himanshu: Working of PING Sensor

Working of PING Sensor