Difference between revisions of "S16: Helios"

From Embedded Systems Learning Academy
Jump to: navigation, search
(→‎Team Members & Responsibilities)
(→‎Software Design)
 
(254 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
=== Grading Criteria ===
+
== Automated Assembly Line ==
−
<font color="green">
+
This project is an automated assembly line capable to defect detection, rejection and sorting with the help of wireless communication and image processing.
−
*  How well is Software & Hardware Design described?
 
−
*  How well can this report be used to reproduce this project?
 
−
*  Code Quality
 
−
*  Overall Report Quality:
 
−
**  Software Block Diagrams
 
−
**  Hardware Block Diagrams
 
−
**:  Schematic Quality
 
−
**  Quality of technical challenges and solutions adopted.
 
−
</font>
 
−
 
 
−
== Project Title ==
 
  
 
== Abstract ==
 
== Abstract ==
−
This section should be a couple lines to describe what your project does.
+
The project is an automated assembly line capable of object defect detection, object rejection and sorting with the help of image processing algorithms and wireless communication. The entire system comprises of three micro controllers (2 SJOne boards and one Raspberry pi 2). A wired interface exists between raspberry pi 2 and SJOne transmitter board, while the two SJOne boards communicate via nordic wireless. Raspberry pi 2 is equipped with a linux operating system with Open CV support to run image processing algorithms for color and shape detection. All three micro controllers are mounted to a miniature assembly line with object moving on it. The SJOne transmitter board is capable of object rejection using servo motor and IR object detector module, while the SJOne receiver board is capable of simple sorting using servo motor based on the result of object color and shape detection algorithms running on raspberry pi 2.
  
 
== Objectives & Introduction ==
 
== Objectives & Introduction ==
−
Show list of your objectives. This section includes the high level details of your project.  You can write about the various sensors or peripherals you used to get your project completed.
+
*To design a automated assembly line with motor drive.
 +
*To implement nordic wireless communication on SJOne board for communication of control signals.
 +
*To design and implement color and shape detection algorithms using Open CV.
 +
*To use servo motors with mechanical parts to perform object rejection and sorting.
 +
*To be able to process images of objects on the belt and decide the acceptance of object for future processing on the assembly line.
  
 +
There is a need for Assembly lines to incorporate intelligence to identify and reject defective products. Image processing is one of the foremost technologies responsible for achieving this intelligence. An assembly line with wireless and object detection, sorting capabilities is the prime goal of this project. The need for a wireless assembly line arises due to large wiring overhead in wired systems.
 +
 +
A camera captures the pictures of items that move on an Assembly line. The raspberry pi 2 board processes these pictures and assesses color and shape information and communicates to boards further down the aseembly line to take appropriate action. The SJOne transmitter board receives the information and rejects defective items from the assembly line while retaining fit items for the future stages of the assembly line. The SJOne receiver board uses servo motor to bifurcate the products into two separate belts based on color for future actions.
 +
 
=== Team Members & Responsibilities ===
 
=== Team Members & Responsibilities ===
  
−
'''Karthik Parameswaran'''<br>
+
'''[https://www.linkedin.com/in/kparameswaran Kartick Parameswaran]'''<br>
−
*  OpenCV, Rasberry Pi
+
*  OpenCV installation, Rasberry Pi environment setup, Hardware support
−
'''Anush Shankar'''<br>
+
'''[https://www.linkedin.com/in/anush-shankar-b9bb3376 Anush Shankar]'''<br>
−
*  Open CV, Nordic, Motor
+
*  Open CV installation support, Nordic wireless communication, Stepper and servo control, Conveyor and Hardware
 
'''[https://www.linkedin.com/in/sindhuja-gopalakrishnan-elango-b97b16106 Sindhuja Gopalakrishnan Elango]'''<br>
 
'''[https://www.linkedin.com/in/sindhuja-gopalakrishnan-elango-b97b16106 Sindhuja Gopalakrishnan Elango]'''<br>
−
*  Open CV, Nordic, Sensor
+
*  Open CV algorithm design, Nordic wireless communication, Sensor implmentation, Hardware support
−
'''Akshay Dabholkar'''<br>
+
'''[https://www.linkedin.com/in/akshaydabholkar Akshay Dabholkar]'''<br>
−
*  Open CV
+
*  Open CV algorithm design,implementation and verification, Hardware support
  
 
== Schedule ==
 
== Schedule ==
Line 50: Line 47:
 
*Ordering components required.
 
*Ordering components required.
 
*Research on prospective image processing platforms and their feasibility.
 
*Research on prospective image processing platforms and their feasibility.
−
| In Progress
+
| Completed
−
|  
+
| 03/29/2016
 
|-
 
|-
 
! scope="row"| 2
 
! scope="row"| 2
Line 60: Line 57:
 
*Establish speed and direction control for Motors.
 
*Establish speed and direction control for Motors.
 
*To start building assembly line model for future integration.
 
*To start building assembly line model for future integration.
−
| Incomplete
+
| Completed
−
|  
+
| 04/05/2016
 
|-
 
|-
 
! scope="row"| 3
 
! scope="row"| 3
Line 69: Line 66:
 
*Decide on image acquisition, enhancement and basic color detection algorithm, start implementation.
 
*Decide on image acquisition, enhancement and basic color detection algorithm, start implementation.
 
*Complete motor controls and begin wireless communication design and implementation.
 
*Complete motor controls and begin wireless communication design and implementation.
−
| Incomplete
+
| Completed
−
|  
+
| 04/12/2016
 
|-
 
|-
 
! scope="row"| 4
 
! scope="row"| 4
Line 78: Line 75:
 
*Design Image morphology detection scheme and start implementation.
 
*Design Image morphology detection scheme and start implementation.
 
*Implementation of object sorting and rejection system.
 
*Implementation of object sorting and rejection system.
−
| Incomplete
+
| Completed
−
|  
+
| 04/25/2016
 
|-
 
|-
 
! scope="row"| 5
 
! scope="row"| 5
Line 87: Line 84:
 
*Verification of image processing algorithm on different objects based on color and morphology.
 
*Verification of image processing algorithm on different objects based on color and morphology.
 
*Mounting of motor controls on assembly line system and verification of design.
 
*Mounting of motor controls on assembly line system and verification of design.
−
| Incomplete
+
| Completed
−
|  
+
| 05/05/2016
 
|-
 
|-
 
! scope="row"| 6
 
! scope="row"| 6
Line 97: Line 94:
 
*Verification of wireless communication across individual boards.
 
*Verification of wireless communication across individual boards.
 
*Decision on safety controls for the system.
 
*Decision on safety controls for the system.
−
| Incomplete
+
| Completed
−
|  
+
| 05/12/2016
 
|-
 
|-
 
! scope="row"| 7
 
! scope="row"| 7
Line 105: Line 102:
 
|  
 
|  
 
*Extensive Testing of the design with varied control signals and perform necessary calibration.
 
*Extensive Testing of the design with varied control signals and perform necessary calibration.
−
| Incomplete
+
| Completed
−
|  
+
| 05/15/2016
 
|-
 
|-
 
! scope="row"| 8
 
! scope="row"| 8
Line 113: Line 110:
 
|  
 
|  
 
*Final testing, checks and enhancements in design.  
 
*Final testing, checks and enhancements in design.  
−
| Incomplete
+
| Completed
−
|  
+
| 05/21/2016
 
|-
 
|-
 
|}
 
|}
  
 
== Parts List & Cost ==
 
== Parts List & Cost ==
−
Give a simple list of the cost of your project broken down by components.  Do not write long stories here.
+
{| class="wikitable"
 +
|-
 +
! scope="col"| Item#
 +
! scope="col"| Part Description
 +
! scope="col"| Vendor
 +
! scope="col"| Qty
 +
! scope="col"| Cost
 +
|-
 +
! scope="row"| 1
 +
| Conveyer system
 +
| [http://www.amazon.com/Bruder-Toys-02031-Conveyor-Belt/dp/B001EHEH9S?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_detailpage_o01_s00 Amazon]
 +
| 1
 +
| $34.62
 +
|-
 +
! scope="row"| 2
 +
| IR Object detector
 +
| [http://www.google.com/shopping/product/1603653807359657272?lsf=seller:6136318,store:2824197551625125398&prds=oid:8396077592678156451&q=frys+ir+detector&hl=en&ei=YGAmV4bIE9PGjwPg4LZY Fry's]
 +
| 2
 +
| $21.78
 +
|-
 +
! scope="row"| 3
 +
| Antenna - Wireless communication
 +
| [http://www.amazon.com/Wireless-Routers-Extend-Antenna-Linksys/dp/B000SJV9C6?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_detailpage_o00_s00 Amazon]
 +
| 2
 +
| $10.9
 +
|-
 +
! scope="row"| 4
 +
| 5V Stepper Motor with driver break out board
 +
| [http://www.amazon.com/4-phase-Stepper-28byj-48-5v-Uln2003-Arduino/dp/B00CCXHGG4?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_detailpage_o01_s00 Amazon]
 +
| 2
 +
| $11.98
 +
|-
 +
! scope="row"| 5
 +
| Microsoft USB HD camera
 +
| [http://www.amazon.com/Microsoft-LifeCam-HD-3000-Webcam-T3H-00011/dp/B008ZVRAQS?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_detailpage_o00_s00 Amazon]
 +
| 1
 +
| $24.24
 +
|-
 +
! scope="row"| 7
 +
| Wire Cutter
 +
| [https://www.amazon.com/Hakko-CHP-170-Stand-off-Construction-21-Degree/dp/B00FZPDG1K Amazon]
 +
| 1
 +
| $2.99
 +
|-
 +
! scope="row"| 8
 +
| Hardware Items
 +
| [http://www.homedepot.com/ Home Depot]
 +
| 1
 +
| $15.49
 +
|-
 +
! scope="row"| 9
 +
| SJOne Board
 +
| Purchased from Preet
 +
| 3
 +
| $240
 +
|-
 +
! scope="row"| 10
 +
| Raspberry Pi 2 Board
 +
|
 +
| 1
 +
| $30
 +
|-
 +
! scope="row"|
 +
| Total Cost
 +
|  
 +
|
 +
| $383.87
 +
|-
 +
|}
 +
 
 +
== System Block Diagram==
 +
[[File:CMPE_244_S16_Helios_block_diagram.png|centre|800px|thumb|System Block Diagram]]
  
 
== Design & Implementation ==
 
== Design & Implementation ==
−
The design section can go over your hardware and software design. Organize this section using sub-sections that go over your design and implementation.
+
This section of the report provide the Hardware and Software design of the project. Details of all design considerations and implementation are provided in this section.
  
 
=== Hardware Design ===
 
=== Hardware Design ===
−
Discuss your hardware design here.  Show detailed schematics, and the interface here.
+
----
 +
This section describes the various hardware design parameters involved in the project. It involves conveyor system design, object rejection hardware, object sorting hardware,object detection hardware and conveyor drive hardware.
 +
==== SJOne Board ====
 +
----
 +
[[File:CMPE_244_S16_Helios_SJOne_board.png‎|left|200px|thumb|SJOne board]]
 +
 
 +
 
 +
The SJOne board is one of the development boards used for the project. Some of the features offered by SJOne board are as follows:
 +
 
 +
 
 +
*Built-in Nordic Wireless (Board-to-Board communication).
 +
*Software Stack for Mesh Network.
 +
*4 Switches and 4 LEDs (both hard-wired).
 +
*Many GPIOs with two SPI, Multiple UARTs, and I2C availability.
 +
*Power from USB or External Power.
 +
*PWM support.
 +
 
 +
 
 +
The SJOne board offers whole lot of features out of which wireless communication (Nordic), GPIO, I2C and PWM timers are utilized for the project. The I2C feature was used to communicate with the Raspberry pi, GPIO's used to drive stepper motors, PWM's to drive servo motors, Nordic for wireless communicaton between SJOne boards.
 +
 
 +
==== Raspberry Pi ====
 +
----
 +
Raspberry Pi is a  linux based embedded computer created by the Raspberry Pi Foundation. The Foundation aims at providing cheap and affordable computing solutions.
 +
=====Raspberry Pi features=====
 +
Microcontroller: The Raspberry Pi contains a microcontroller manufactured by Broadcom - BCM2836. The processor within this chip is a quad-core Cortex-A7.
 +
Operating System: Raspberry pi has many compatible operating systems available. We chose the Raspbian Jessie operating system as it popular. The underlying core for this operating system is a debian linux kernel.
 +
Peripherals:
 +
The various hardware available on board can be found [https://www.raspberrypi.org/documentation/hardware/raspberrypi/README.md here].
 +
 
 +
=====Installing OpenCV libraries on Raspberry Pi=====
 +
 
 +
OpenCV is a collection of libraries that provide various C, C++ and Python functions that can be directly used for Image processing. The advantage is that it provides data structures to hold RGB images and complex image processing algorithms, such as edge detection, is already present and just have to be called within our program.
 +
OpenCV installation can be done using various methods:
 +
# Cross compiling on a general purpose computer: This is a common method for compiling OpenCV. This method involves compiling OpenCV on a general purpose computer which has a cross compiler installed intended for your target platform, such as the arm-gcc. The advantage of cross compiling is that the compiled libraries can be now used on both the target and host computers. It would now be possible to compile program by using the arm-gcc on your host platform and then copying the generated executable into the target platform and directly run it.
 +
#Installing by building a new kernel image for your target platform: Another popular method is to use tools such as Buildroot which compile a kernel image for your target platform. This is useful if you are installing a new OS and want the OpenCV libraries installed along with this new OS. This tool automatically searches and compiles any dependencies that would be needed to install OpenCV, which is the reason this is popular.
 +
#Installing directly on the target platform: This is a less preferred method because of the large amount of time it can take and because it needs an . Compiling libraries is a resource intensive process and thus takes a large amount of time (more than 2-3 hours) on a slower processor.
 +
For our project we chose method three for the following reasons:
 +
#Raspberry Pi 2 with its quad-core processor is capable of handling these loads.
 +
#It was easier to compile on the native system than cross-compiling.
 +
#With an internet connectivity adding dependencies was easy using “sudo apt-get install”
 +
After searching the internet for methods to compile the libraries we came across [https://www.raspberrypi.org/forums/viewtopic.php?f=32&t=122555 this script] to install the libraries and its dependencies.
 +
[[File:Helios RP2 Pinout.png|500px|center|thumb|Raspberry Pi Pin connections]]
 +
 
 +
==== Conveyor Belt ====
 +
----
 +
Conveyor belt is a mechanical system with which materials can be transferred from one location to another.  Conveyor systems are used in many industrial applications for safely transferring materials where several processing steps happen on the material when it  travels on the belt.  As the conveyor system is fully automated, it can be ensured to a have a uniform throughput  of the deliverable. Industries thus use the conveyors for increasing their production rate.  Conveyor systems are very popular in material handling, packaging, automotive, food processing and many other industries.<br>
 +
[[File:CMPE_244_S16_Helios_Conveyor_basic.png|centre|800px|thumb|Basic conveyor structure]]
 +
 
 +
====  Webcam ====
 +
----
 +
[[File:CMPE_244_S16_Helios_USB_webcam.PNG‎|right|140px|thumb|USB webcam]]
 +
The USB webcam is an integral part of the Image processing system running as part of the automated assembly line. The camera acts as eyes of the system and is used to capture images, that may be processed later using open CV algorithms. Choice of camera is crucial to the success of the project. A camera with high definition image quality and wide angle view provides better image processing control and lower pixel corruption due to external light related noise. A microsoft HD lifecam 3000 was chosen due to the various high end features it offers and comparatively low cost. Some of the features of this camera are as follows:
 +
 
 +
 
 +
*Widescreen - 16:9 format offers cinematic video recording.
 +
*TrueColor technology  - TrueColor technology automatically delivers bright and colorful video, in virtually all lighting conditions.
 +
*Universal attachment base - Works on your desktop, laptop, or notebook.
 +
*Small size and ease of mounting.
 +
*2.0 USB connectivity.
 +
 
 +
==== Nordic wireless ====
 +
----
 +
 
 +
[[File:CMPE_244_S16_Helios_Nordicblock.png|centre|800px|thumb|Nordic communication]]
 +
SJone board has  a nordic wireless chip nRF24L01+ manufactured by the Nordic Semiconductors interfaced to it. Through a Serial peripheral Interface it is possible to operate and configure the Nordic chip. The air data rate, operating modes and RF channel frequency need to be set by configuring the registers in the  nRF24L01+.
 +
 
 +
Air data rate is the modulating signalling rate that is used by nRF24L01+ for transmitting and receiving data. It can be 250 Kbps, 1 Mbps and 2 Mbps. Lower air data rate gives better receiver sensitivity while higher data rate gives a reduced current consumption. The RF channel frequency determines the centre of the channel frequency in which nRF24L01+ operates. Typical operating frequencies are from 2.400GHz to 2.525GHz
 +
 
 +
For communication to happen,
 +
* Both nordic transmitter and receiver must be set at the same air data rate. We have set the air data rate at 250 Kbps.
 +
* Both nordic transmitter and receiver must be operated at the same RF channel frequency.We have set the channel number to 2.499 GHz.
 +
 
 +
==== IR Detector ====
 +
----
 +
 
 +
[[File:CMPE_244_S16_Helios_IRDetector.png|left|150px|thumb|IR Detector]]
 +
[[File:CMPE_244_S16_Helios_IRDetector_circuit.png|150px|thumb|IR Detector Calibration]]
 +
[[File:CMPE_244_S16_Helios_IRDetector_back.png|none|150px|thumb|IR Detector]]
 +
 
 +
 
 +
 
 +
IR Detector OSEPP IRDET-01 detects obstacles when they are in the calibrated range. The module has 3 pins which are the ground(G), VCC(V) and Signal(S). The signal wire line is digital. It can be either high or low. High if the detected range is further than the calibrated distance, low if the detected range is less than the calibrated distance. The pins are connected to the GND, VCC and GPIO pin of the SJone board,
 +
The pin direction is set as input. When no object is detected the pin will read 0 and when object is detected the pin will read 1.
 +
=====Calibration=====
 +
 
 +
[[File:CMPE_244_S16_Helios_NoObject.png|left|300px|thumb]]
 +
[[File:CMPE_244_S16_Helios_object.png|right|220px|thumb]]
 +
IR detector module device has two potentiometers that can be used to calibrate the device. One for frequency (F) and one for distance (D). Much like an IR Remote, the IR light is oscillated at a defined frequency and detected at that same frequency; this enables the sensor to work in noisy environments. The frequency of the detector module can be adjusted to avoid interference with other detectors or other IR sources.
 +
 
 +
 
 +
Turning the frequency potentiometer clockwise increases the frequency and turning it anti clockwise decreases the frequency.The distance at which the sensor is triggered is determined by adjusting the Distance potentiometer. When the sensor is triggered the LED will light up. Turning this pot clockwise increases the triggering distance, turning it counter-clockwise decreases the distance.
 +
 
 +
==== Servo Motor ====
 +
----
 +
SG90 is a tiny and lightweight servo motor with high output power. Servo can rotate approximately 180 degrees (90 in each direction), and is driven by a PWM wave. It fits in small places and offers good torque. Using servo motors provides precise control of angular movement using relevant duty cycle of a constant frequency PWM wave. Servos consist of an internal motor with a feedback system for position.
 +
 
 +
 
 +
[[File:CMPE_244_S16_Helios_Servo.png|350px|left|thumb|SG90 Servo Motor]]
 +
 
 +
[[File:CMPE_244_S16_Helios_Servo_PWM.png|390px|none|thumb|Servo PWM Waves]]
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
The wiring information for SG90 is as follows:
 +
 
 +
{| class="wikitable"
 +
|-
 +
! scope="col"| Sl. No
 +
! scope="col"| Servo Pin Color
 +
! scope="col"| Purpose
 +
|-
 +
|-
 +
|1
 +
|Brown
 +
|Ground
 +
|-
 +
|-
 +
|2
 +
|Red
 +
|VCC (max 4.8 Volts)
 +
|-
 +
|-
 +
|3
 +
|Yellow
 +
|PWM input to motor
 +
|}
 +
 
 +
 
 +
Some of the specifications if SG90 servo are as follows:
 +
 
 +
*Weight: 9 g.
 +
 
 +
*Dimension: 22.2 x 11.8 x 31 mm approx.
 +
 
 +
*Stall torque: 1.8 kgf·cm.
 +
 
 +
*Operating speed: 0.1 s/60 degree.
 +
 
 +
*Operating voltage: 4.8 V (~5V).
 +
 
 +
 
 +
A PWM wave of 50 Hz or 20 ms is needed to drive the servo. The servo works with duty cycle between 5-10%. Provided a wave with duty cycle 5% the motor rotates 90 degree to the left from neutral position (extreme left position), with 10% it rotates 90 degree to the right from neutral position(extreme right position) and with 7.5% duty cycle moves to neutral position.
 +
 
 +
==== Stepper Motor ====
 +
----
 +
[[File:CMPE_244_S16_Helios_steppermotor.gif|240px|left|thumb|Stepper motor working]]
 +
 
 +
A stepper motor is a DC motor which moves in discreet steps. The motor has poles which when energizedFor driving the conveyer system 28BYJ-48 stepper motors were used with ULN2003 darlington array module. The ULN2003 is a high voltage, high current darlington array with 7 NPN darlington pairs for providing high current required by the stepper motor.
 +
[[File:CMPE_244_S16_Helios_stepper.jpg|180px|right|thumb|28BYJ-48 Stepper motor]]
 +
 
 +
 
 +
Some of the features of the stepper are :
 +
 
 +
*Voltage: 5V.
 +
*Diameter: 28mm.
 +
*Step angle: 5.625 x 1 / 64.
 +
*Reduction ratio: 1 / 64
 +
*5 Line 4 phase can be driven by ordinary ULN2003 chip.
 +
*Driven using PWM or GPIO.
 +
*Simple connections with easy to plug connectors and switch for power.
 +
*A, B, C, D four-phase LED indicates the status of the stepper motor work.
 +
 
  
 
=== Hardware Interface ===
 
=== Hardware Interface ===
−
In this section, you can describe how your hardware communicates, such as which BUSes used. You can discuss your driver implementation here, such that the '''Software Design''' section is isolated to talk about high level workings rather than inner working of your project.
+
----
 +
The hardware interface is divided into two parts. Components that were interfaced with SJOne Master board and components that were interfaced with the SJOne Slave board.
 +
Both the SJone boards had to be mounted one on each end of the conveyor system. Decsions made by the SJone master board will be implemented by the SJOne slave board.
 +
====Building our own conveyor system====
 +
----
 +
Initially it was decided to buy or rent conveyor systems and interface components to it. First option was to buy the conveyor system. On enquiring with many dealers who supplied industrial mini conveyor systems to small scale and large scale industries , it turned out that a simple system was around 1200 US dollars. Second option was to rent the conveyor belt. Renting it for a day would cost around 200 dollars. But the requirement for the project was definitely more than a day. There were development and testing phases of each component that had to be mounted with the conveyor belt and tested frequently. So the options of buying and renting were eliminated. Having an objective of building an economical automated conveyor system, the decision to self build the conveyor system was made by the team.<br>
 +
Nevertheless, it wasn't that easy to build a conveyor system. The conveyor system required the following<br>
 +
*A rigid frame on both sides to hold the conveyor belt <br>
 +
*Rotating cylinders that had to be mounted to either ends of the frame.<br>
 +
*A conveyor belt that could slide on the cylinders bordered by the rigid frame<br>
 +
 
 +
====Interface with SJOne Master Board====
 +
----
 +
The SJOne master board had the following components to it:
 +
* I2C connection with Raspberry Pi: Raspberry pi passes information about the object detected to the SJOne Master board through I2C. The SJOne board is configured as a slave so that it can receive the information of the color of the object.
 +
*Nordic transmitter:The information received through I2C is relayed to the slave SJOne board for further sorting of the objects.
 +
*IR sensor: The IR sensor detects the presence of an object. This is needed because the camera can detect only the valid colors, but there is no way for the camera to know whether there is an object that is of the wrong color or if there are no objects.
 +
*Servo motor for rejection: The servo motor requires has three pins: VCC, GND and PWM signal. The PWM signal is provided at port 2 pin 1. This servo motor is responsible for rejecting any object that is not blue or green in color. The decision to swing the servo motor arm is based on the information received on the I2C interface and the IR sensor input.
 +
 
 +
The pin connections for the master SJOne board are as listed in the table below:
 +
{| class="wikitable"
 +
|-
 +
! scope="col"| Connections
 +
! scope="col"| SJ One Pin
 +
! scope="col"| Peripheral Name
 +
! scope="col"| Peripheral Pin name
 +
! scope="col"| Description
 +
 
 +
|-
 +
! scope="row"| 1
 +
| SDA
 +
| RaspberryPi
 +
|SDA
 +
|SDA pin for I2C communication
 +
|-
 +
|-
 +
! scope="row"| 2
 +
| SCL
 +
| RaspberryPi
 +
|SCL
 +
|SCL pin for I2C communication
 +
|-
 +
! scope="row"| 3
 +
| GND
 +
| RaspberryPi
 +
|GND
 +
|Common GND for I2C communication
 +
|-
 +
! scope="row"| 4
 +
|Port 2 Pin 1
 +
|Servo Motor
 +
|Signal
 +
|PWM for servo motor
 +
|-
 +
! scope="row"| 5
 +
|Port 0 Pin 1
 +
|IR sensor
 +
|Signal
 +
|Input from IR sensor
 +
|}
 +
 
 +
====Interface with SJOne Slave Board====
 +
----
 +
The SJOne slave board has three interfaces:
 +
*Nordic wireless receiver:The wireless receiver receives information of the object on the conveyor belt and will take the decision to sort it based on the color of the object
 +
*IR sensor to detect object: The IR sensor detects the presence of an object. This is needed because the camera can detect only the valid colors, but there is no way for the camera to know whether there is an object that is of the wrong color or if there are no objects.
 +
*Servo motor for sorting: The servo motor connected to the slave is responsible for sorting the green object, but allow the blue object to pass through. The decision to swing the servo motor arm is based on the information received on the Nordic wireless interface and the IR sensor input.
 +
 
 +
The pin connections for the slave SJOne board are as listed in the table below:
 +
{| class="wikitable"
 +
|-
 +
! scope="col"| Connections
 +
! scope="col"| SJ One Pin
 +
! scope="col"| Peripheral Name
 +
! scope="col"| Peripheral Pin name
 +
! scope="col"| Description
 +
 
 +
|-
 +
! scope="row"| 1
 +
|Port 2 Pin 1
 +
|Servo Motor
 +
|Signal
 +
|PWM for servo motor
 +
|-
 +
! scope="row"| 2
 +
|Port 0 Pin 1
 +
|IR sensor
 +
|Signal
 +
|Input from IR sensor
 +
 
 +
|-
 +
! scope="row"| 3
 +
|VCC
 +
|IR sensor and servo motor
 +
|VCC
 +
|3.3 V supply for IR sensor and servo motor
 +
|-
 +
! scope="row"| 4
 +
|GND
 +
|IR sensor and servo motor
 +
|GND
 +
|GNDfor IR sensor and servo motor
 +
|-
 +
|}
  
 
=== Software Design ===
 
=== Software Design ===
−
Show your software design.  For example, if you are designing an MP3 Player, show the tasks that you are using, and what they are doing at a high level. Do not show the details of the code. For example, do not show exact code, but you may show psuedocode and fragments of code. Keep in mind that you are showing DESIGN of your software, not the inner workings of it.
+
==== Assembly line Mechanism ====
 +
[[File:CMPE_244_S16_Helios_SJOneMaster.png|center|700px|thumb|Master SJOne board Flow Sequence]]
 +
[[File:CMPE_244_S16_Helios_SJOneSlave.png|center|700px|thumb|Slave SJOne board Flow Sequence]]
 +
 
 +
==== Object Detection and Analysis with OpenCv and image processing ====
 +
----
 +
Image processing is the process of manipulating image data in order to make it suitable for computer vision applications. Computer vision helps to obtain relevant information from images and make decisions based on that information. In our project, we have used OpenCV with C++ to develop our computer vision application. OpenCV is an open source C++ library for image processing and computer vision, originally developed by Intel and now supported by Willow Garage. OpenCV has many inbuilt functions for image processing and computer vision that are mainly aimed at real-time processing.
 +
   
 +
=====Digital Image and pixels=====
 +
A digital image is a collection of pixels. A pixel is the smallest element in a digital image. In an image, each pixel can have a particular value to represent that image. For example, consider a grayscale image where each pixel can be represented by 8-bit value. So, each pixel can have a range of 0 to 255 where 0 is the complete black and 255 is a completely white pixel.
 +
 
 +
[[File:CMPE_244_S16_Helios_Matrix_representation_of_GRAYSCALE.png|center|700px|thumb|Matrix Representation of GRAYSCALE]]
 +
 
 +
A color image consists of 3 planes Red, Green, and Blue. Any color can be formed with the combination these three planes. These planes are called channels of the image. So, an RGB i.e. a color image has 3 channels. Each pixel in RGB has a combination of three values. (255, 0, 0) represent pure red. (0, 255, 0) represent pure green. (255, 0, 255) represents pure violate. Thus, each pixel is represented by 24-bits in an RGB.
 +
 
 +
[[File:CMPE_244_S16_Helios_RGB_Planes.png|center|700px|thumb|RGB Planes]]
 +
 
 +
[[File:CMPE_244_S16_Helios_RGB_Matrix.png|center|700px|thumb|RGB Planes]]
 +
 
 +
===== Color Detection =====
 +
Object detection and segmentation is the most important task of computer vision. The easiest way to detect and segment the object is through color detection.
 +
 
 +
The method used for color detection was to first convert the RGB image into HSV and then by thresholding the HSV image, the binary version (Black and White) of the HSV can be obtained. In this binary image, the white space would represent the color which wants to detect. This binary image can then be used to detect the object we are looking using color detection and also to find the shape of the detected object using the available OpenCv library functions.
 +
 
 +
'''Need to convert from RGB to HSV'''
 +
 
 +
RGB color space describes colors in terms of the amount of red, green, and blue present. HSV color space describes colors in terms of the Hue, Saturation, and Value. The HSV image separates luma (intensity component), from chroma (color component). So that we can now perform histological equalization of the image by adjusting the minimum and maximum values of H, S, and V to such levels that only the pixel with the desired color will be illuminated (white) and the rest of the pixels will be dark.
 +
void inRange(InputArray src, InputArray lowerb, InputArray upperb, OutputArray dst) library function was used to threshold the HSV image.
 +
'''Src''' is the source image.
 +
'''Dst''' is the thresholded image.
 +
The '''upperb''' and '''lowerb''' boundary parameters are scalars (minimum and maximum values of HSV) which can be determined using trackbars.
 +
 
 +
[[File:CMPE_244_S16_Helios_HSVandOriginal.JPG|center|500px|thumb|HSV and Original Image]]
 +
 
 +
 
 +
[[File:CMPE_244_S16_Helios_Shape_Original&thresholded.JPG|center|500px|thumb|Thresholded and Original Image]]
 +
 
 +
'''Trackbars'''
 +
 
 +
'''Trackbar''' is a very useful tool which allows us to change various parameters while our OpenCv application is running. In our project, we made extensive use of trackbars to determine upper and lower boundary parameters for the particular color which we want to detect.
 +
Show the image of trackbars and thresholded image.
 +
As shown in the fig. we can change the minimum values of H, S, and V in such a manner that only the desired color will remain as a whitespace and rest of the image should be dark. Once the minimum and maximum values are determined we can lock down these values with inRange() function to get the thresholded image.
 +
 
 +
  int createTrackbar(const string& trackbar name, const string& winname, int* value, int count, TrackbarCallback) 
 +
is the library function which was used to create the trackbars.
 +
 
 +
[[File:CMPE_244_S16_Helios_Trackbars.JPG|center|500px|thumb|RGB Planes]]
 +
 
 +
 
 +
'''Trackbarname''' is the name of the trackbar.
 +
'''Winname''' is the window name to which trackbar is attached. Integer, pointed by this pointer, holds the value associated with the position of the trackbar.
 +
'''count''' - The maximum value of the trackbar. The minimum value is always zero.
 +
 
 +
'''findContours library function'''
 +
 
 +
Using find contours we can retrieve the contours from the binary image that we obtained through thresholding the HSV image.
 +
void findContours(InputOutputArray image, OutputArrayOfArrays contours, OutputArray hierarchy, int mode, int method, Point offset=Point())
 +
We can pass appropriate modes to retrieve the different contour information from the image. The contour information is retrieved in the contour is a vector of points.
 +
From the contours information, we can find the area of the whitespaces using moments method which is actually the target color and can detect the color.
 +
 
 +
'''Algorithm for Color Detection
 +
'''<br>
 +
'''Step1''': Initialize the camera.<br>
 +
'''Step2''': Capture the image frame.<br>
 +
'''Step3''': Convert the image from RGB to HSV.<br>
 +
'''Step4''': Threshold the image for the required color.<br>
 +
'''Step5''': Apply morphological operations to eliminate noise.<br>
 +
'''Step6''': Find contours of the binary image.<br>
 +
'''Step7''': Find the number of whitespace regions using the contour information.<br>
 +
'''Step8''': If no whitespaces found go to step 2, else proceed to step9.<br>
 +
'''Step9''': Check if the area of whitespace is greater than the minimum area. If area is less than the minimum area set, it may be a noise then go to step 2, else proceed to step 10.<br>
 +
'''Step10''': Set object found flag to true. Continue scanning go to step2.<br>
 +
 
 +
'''Flowchart for Color Detection'''
 +
[[File:CMPE_244_S16_Helios_Color_Detection.png|center|700px|thumb|Color Detection Flowchart]]
 +
 
 +
===== Morphological Operations =====
 +
Filtering is an important part of computer vision. In most of the applications, filtering should be applied to the images before processing on them. In our project, we applied morphological operations to remove noise and to do proper segmentation of the images. The two morphological operations we performed were: Dilation and Erosion.
 +
 
 +
'''Dilation''': Dilation increases the white pixels along the boundaries of the foreground object. Consider the following example.
 +
 
 +
[[File:CMPE_244_S16_Helios_Dilate.png|center|700px|thumb|Image Dilation]]
 +
 
 +
As we can see that dilation increases white pixels along the boundary of the image.
 +
 
 +
'''Erosion''': Erosion is opposite to dilation. It increases the dark pixels along the boundaries of the foreground object. The example of the erosion is as shown in the figure.
 +
[[File:CMPE_244_S16_Helios_Erode.png|center|700px|thumb|Image Erosion]]
 +
 
 +
As we can clearly see the erosion operation increased the black pixels along the boundary of the image. In our project, we applied erosion followed by dilation because the erosion sinks our object, so we dilate it.
  
 
=== Implementation ===
 
=== Implementation ===
−
This section includes implementation, but again, not the details, just the high level. For example, you can list the steps it takes to communicate over a sensor, or the steps needed to write a page of memory onto SPI Flash. You can include sub-sections for each of your component implementation.
+
----
 +
The implementation of the object color identification was done through image-processing. We used Raspberry pi and OpenCV to perform the image-processing. The image frames were captured from a USB camera connected to the Raspberry pi. Each frame was processed with our OpenCV algorithm to detect the desired object. The information about the detected object was then passed to SJOne board interfaced to the Raspberry pi through I2C. The values for the object present in front of the camera could have three values:
 +
#Blue
 +
#Green
 +
#Invalid
 +
The value invalid is for both when there is no object in front of the camera or an object that does not have the desired color. Once this information is passed to the SJOne Master board the IR sensor connected to this board identifies whether the invalid classification was for an invalid colored object or for an absence of any object. If the decision made computes as an invalid object, the servo motor activates and pushes the object out of the conveyor system else the servo motor remains in its idle position.
 +
A similar logic is employed at the slave SjOne board. The information for the color of the object is relayed to this slave board via the Nordic wireless interface. If the object is green, then the sorting servo pushes it away and if it is blue, it lets it pass through.
 +
 
 +
The flow chart below explains this process graphically.
 +
 
 +
[[File:Helios_Implementation_Flowchart.png|500px|thumb|center]]<br/>
  
 
== Testing & Technical Challenges ==
 
== Testing & Technical Challenges ==
−
Describe the challenges of your project.  What advise would you give yourself or someone else if your project can be started from scratch again?
+
===Testing===
−
Make a smooth transition to testing section and described what it took to test your project.
+
----
 +
Testing for this project was carried out in phases. This was a conscious decision on part of all the team members as previous experiences have taught is that testing everything all at once can lead to problems that could otherwise have been identified earlier. The testing involved the following phases:
 +
====Phase 1====
 +
Phase once consisted of unit testing each part. The image processing algorithm was first tested on a desktop computer. This was then ported to the Raspberry Pi which resulted in an instant success on the Raspberry Pi. Parallel to this the motor systems, I2C, IR sensor calibration and Nordic wireless messaging were tested and the code was fixed.
 +
====Phase 2====
 +
Phase 2 testing was to combine the smaller parts of the system. The code for image processing and I2C was merged and tested. At the same time the IR and servo motor was tested so that they worked in conjunction with the messages received over I2C(for the SJOne Master board) and Nordic wireless(for the SJOne slave board). At this point we also ensured the conveyor system was in place and working smoothly.
 +
 
 +
====Final testing====
 +
Final testing consisted of combining all the parts and securing them in the housing and testing the rejection system first and then the rejection system was tested. Phased testing enabled us to quickly identify any changes that would be needed when the entire system was in place. This resulted in reducing the time spent on integrating the system.
 +
 
 +
===Technical Challenges===
 +
----
 +
This project was a challenging one due to the varied components involved: from choosing the correct board for image processing to building our own conveyor system. There were a lot of difficult areas in this project. The most significant being the entire housing for all the components and the conveyor system and a lot of the ideas we had at the start had to be changed in order to have the desired result.
 +
The most significant challenges are listed below:
 +
 
 +
==== Installing OpenCV on ARM 11 (FriendlyARM Tiny6410 SDK) ====
  
−
Include sub-sections that list out a problem and solution, such as:
+
The Raspberry Pi was not our first choice for a board that can perform image processing, rather it was another board: The FriendlyARM Tiny6410 SDK, it was readily available with us. This board has a Samsung ARM 11 based microcontroller with good memory and decent speed. The board comes with a preinstalled Linux image. The problem was, the documentation for this board is not very comprehensive and the support community is not very active. This meant a lot of trial and error. We tried to install OpenCV but were not successful. The cross compilation kept failing for this board, and eventually we had to give up the idea of using this board due to time constraints in favor of the highly popular Raspberry Pi.
  
−
=== My Issue #1 ===
+
==== Using a pre-built conveyor system ====
−
Discuss the issue and resolution.
+
 
 +
At the outset, we tried to minimize the amount of custom build required for this project. This meant needed a readymade conveyor system. We found and ordered a conveyor system that could be manually cranked. We thought we could attach motors and drive the conveyor leading to a hassle free conveyor system. The result, however, was the opposite. The conveyor could not be dismantled and so the correct area for the motor drive could not be accessed. We managed to attach a motor at one end, but, by this time the structural integrity of the rubber conveyor belt was compromised and so we could not get the motor to drive the conveyor successfully. Eventually we had to build our own conveyor system and housing for all the components. This was a great build exercise as we decided to choose recyclable material for this and made the assembly using cartons and paper.
 +
 
 +
==== Lighting Problem ====
 +
 
 +
Lighting was the most critical problem faced during image-processing. In our project, we converted the RGB matrix frames to HSV for the thresholding. The HSV frame extracts the lighting information ‘Luminosity’ and ‘Chroma’ from the RGB planes, so any slight change in the light source angles would alter the thresholding values for HSV frames. Also, we faced a lot of problems due to distortions arising from the light source. We tried to handle the problem by applying morphological operations and implemented the color detection Algorithm. But, still due to slight distortions still present we faced problem in implementing our shape detection Algorithm where even slight distortions would cause problems in detecting the vertices of the detected object.
  
 
== Conclusion ==
 
== Conclusion ==
−
Conclude your project here. You can recap your testing and problems. You should address the "so what" part here to indicate what you ultimately learnt from this project. How has this project increased your knowledge?
+
The learning experience our team wanted to have was, to learn about things we had not done before. This is reflected in the major components in the project, such as the use of image processing. This further led to the realization that SJOne cannot be used for image processing and hence we had to to look for other resources. The use of an embedded linux computer enhanced our understanding of an RTOS. The use of two different Operating Systems: FreeRTOS and Raspbian Jessie helped us in understanding the different use cases for them and how the software complements the hardware.
 +
Another takeaway from this project is the build. Building the conveyor system was challenging and needed a lot of original thinking. Cross compiling of OpenCV was another major part of this project and it helped us understand how libraries and compilers work together and help in writing programs easily. This aspect of programming is often neglected in application programming and this project gave us a deep insight into this.In conclusion, it can be said that the project was a success and it has been successful in making us understand Embedded Software better.
  
 
=== Project Video ===
 
=== Project Video ===
−
Upload a video of your project and post the link here.
+
[https://www.youtube.com/watch?v=eW2aHOTU_IU YouTube]
  
 
=== Project Source Code ===
 
=== Project Source Code ===
−
*  [https://sourceforge.net/projects/sjsu/files/CmpE_S2016/ Sourceforge Source Code Link]
+
[https://gitlab.com/Team_Helios/AssemblyLine.git Sourceforge Source Code Link]
  
 
== References ==
 
== References ==
 
=== Acknowledgement ===
 
=== Acknowledgement ===
−
Any acknowledgement that you may wish to provide can be included here.
+
We would like to thank our advisor Preetpal Kang for encouraging us to take on a challenging project.
  
 
=== References Used ===
 
=== References Used ===
−
List any references used in project.
+
 
 +
*[http://osepp.com/products/sensors-arduino-compatible/ir-detector/ IR detector]
 +
*[https://www.sparkfun.com/datasheets/Components/SMD/nRF24L01Pluss_Preliminary_Product_Specification_v1_0.pdf Nordic wireless]
 +
*[https://www.raspberrypi.org/ Raspberry Pi]
 +
*[http://opencv.org/ OpenCV]
 +
*[http://www.datasheetspdf.com/datasheet/SG90.html/ Servo motor]
 +
*[http://42bots.com/tutorials/28byj-48-stepper-motor-with-uln2003-driver-and-arduino-uno/ Stepper motor]
 +
*[http://www.ti.com/product/ULN2003A/datasheet/ Stepper driver]
  
 
=== Appendix ===
 
=== Appendix ===
−
You can list the references you used.
+
*[http://opencv.org/quickstart.html OpenCV cross compilation]
 +
*[http://docs.opencv.org/2.4/modules/refman.html OpenCV API references]

Latest revision as of 13:58, 24 May 2016

Automated Assembly Line

This project is an automated assembly line capable to defect detection, rejection and sorting with the help of wireless communication and image processing.

Abstract

The project is an automated assembly line capable of object defect detection, object rejection and sorting with the help of image processing algorithms and wireless communication. The entire system comprises of three micro controllers (2 SJOne boards and one Raspberry pi 2). A wired interface exists between raspberry pi 2 and SJOne transmitter board, while the two SJOne boards communicate via nordic wireless. Raspberry pi 2 is equipped with a linux operating system with Open CV support to run image processing algorithms for color and shape detection. All three micro controllers are mounted to a miniature assembly line with object moving on it. The SJOne transmitter board is capable of object rejection using servo motor and IR object detector module, while the SJOne receiver board is capable of simple sorting using servo motor based on the result of object color and shape detection algorithms running on raspberry pi 2.

Objectives & Introduction

  • To design a automated assembly line with motor drive.
  • To implement nordic wireless communication on SJOne board for communication of control signals.
  • To design and implement color and shape detection algorithms using Open CV.
  • To use servo motors with mechanical parts to perform object rejection and sorting.
  • To be able to process images of objects on the belt and decide the acceptance of object for future processing on the assembly line.

There is a need for Assembly lines to incorporate intelligence to identify and reject defective products. Image processing is one of the foremost technologies responsible for achieving this intelligence. An assembly line with wireless and object detection, sorting capabilities is the prime goal of this project. The need for a wireless assembly line arises due to large wiring overhead in wired systems.

A camera captures the pictures of items that move on an Assembly line. The raspberry pi 2 board processes these pictures and assesses color and shape information and communicates to boards further down the aseembly line to take appropriate action. The SJOne transmitter board receives the information and rejects defective items from the assembly line while retaining fit items for the future stages of the assembly line. The SJOne receiver board uses servo motor to bifurcate the products into two separate belts based on color for future actions.

Team Members & Responsibilities

Kartick Parameswaran

  • OpenCV installation, Rasberry Pi environment setup, Hardware support

Anush Shankar

  • Open CV installation support, Nordic wireless communication, Stepper and servo control, Conveyor and Hardware

Sindhuja Gopalakrishnan Elango

  • Open CV algorithm design, Nordic wireless communication, Sensor implmentation, Hardware support

Akshay Dabholkar

  • Open CV algorithm design,implementation and verification, Hardware support

Schedule

This section of the report provides the team schedule for the Assembly line project, indicating the milestones to be achieved during the course of the project.

SI No. Start Date End Date Task Status Actual Completion Date
1 03/20/2016 03/29/2016
  • Discussions on overall system design and implementation.
  • Identification of parts and components required for the project.
  • Ordering components required.
  • Research on prospective image processing platforms and their feasibility.
 Completed 03/29/2016
2 03/30/2016 04/05/2016
  • Familiarization with image processing tools and software(Open CV).
  • Establish speed and direction control for Motors.
  • To start building assembly line model for future integration.
 Completed 04/05/2016
3 04/06/2016 04/12/2016
  • Decide on image acquisition, enhancement and basic color detection algorithm, start implementation.
  • Complete motor controls and begin wireless communication design and implementation.
 Completed 04/12/2016
4 04/13/2016 04/19/2016
  • Design Image morphology detection scheme and start implementation.
  • Implementation of object sorting and rejection system.
 Completed 04/25/2016
5 04/20/2016 04/26/2016
  • Verification of image processing algorithm on different objects based on color and morphology.
  • Mounting of motor controls on assembly line system and verification of design.
 Completed 05/05/2016
6 04/27/2016 05/03/2016
  • Integrating image processing and other systems (sorting and rejection system, and assembly line).
  • Verification of wireless communication across individual boards.
  • Decision on safety controls for the system.
 Completed 05/12/2016
7 05/04/2016 05/10/2016
  • Extensive Testing of the design with varied control signals and perform necessary calibration.
 Completed 05/15/2016
8 05/11/2016 05/17/2016
  • Final testing, checks and enhancements in design.
Completed 05/21/2016

Parts List & Cost

Item# Part Description Vendor Qty Cost
1 Conveyer system Amazon 1 $34.62
2 IR Object detector Fry's 2 $21.78
3 Antenna - Wireless communication Amazon 2 $10.9
4 5V Stepper Motor with driver break out board Amazon 2 $11.98
5 Microsoft USB HD camera Amazon 1 $24.24
7 Wire Cutter Amazon 1 $2.99
8 Hardware Items Home Depot 1 $15.49
9 SJOne Board Purchased from Preet 3 $240
10 Raspberry Pi 2 Board 1 $30
Total Cost $383.87

System Block Diagram

System Block Diagram

Design & Implementation

This section of the report provide the Hardware and Software design of the project. Details of all design considerations and implementation are provided in this section.

Hardware Design

This section describes the various hardware design parameters involved in the project. It involves conveyor system design, object rejection hardware, object sorting hardware,object detection hardware and conveyor drive hardware.

SJOne Board

SJOne board


The SJOne board is one of the development boards used for the project. Some of the features offered by SJOne board are as follows:


  • Built-in Nordic Wireless (Board-to-Board communication).
  • Software Stack for Mesh Network.
  • 4 Switches and 4 LEDs (both hard-wired).
  • Many GPIOs with two SPI, Multiple UARTs, and I2C availability.
  • Power from USB or External Power.
  • PWM support.


The SJOne board offers whole lot of features out of which wireless communication (Nordic), GPIO, I2C and PWM timers are utilized for the project. The I2C feature was used to communicate with the Raspberry pi, GPIO's used to drive stepper motors, PWM's to drive servo motors, Nordic for wireless communicaton between SJOne boards.

Raspberry Pi

Raspberry Pi is a linux based embedded computer created by the Raspberry Pi Foundation. The Foundation aims at providing cheap and affordable computing solutions.

Raspberry Pi features

Microcontroller: The Raspberry Pi contains a microcontroller manufactured by Broadcom - BCM2836. The processor within this chip is a quad-core Cortex-A7. Operating System: Raspberry pi has many compatible operating systems available. We chose the Raspbian Jessie operating system as it popular. The underlying core for this operating system is a debian linux kernel. Peripherals: The various hardware available on board can be found here.

Installing OpenCV libraries on Raspberry Pi

OpenCV is a collection of libraries that provide various C, C++ and Python functions that can be directly used for Image processing. The advantage is that it provides data structures to hold RGB images and complex image processing algorithms, such as edge detection, is already present and just have to be called within our program. OpenCV installation can be done using various methods:

  1. Cross compiling on a general purpose computer: This is a common method for compiling OpenCV. This method involves compiling OpenCV on a general purpose computer which has a cross compiler installed intended for your target platform, such as the arm-gcc. The advantage of cross compiling is that the compiled libraries can be now used on both the target and host computers. It would now be possible to compile program by using the arm-gcc on your host platform and then copying the generated executable into the target platform and directly run it.
  2. Installing by building a new kernel image for your target platform: Another popular method is to use tools such as Buildroot which compile a kernel image for your target platform. This is useful if you are installing a new OS and want the OpenCV libraries installed along with this new OS. This tool automatically searches and compiles any dependencies that would be needed to install OpenCV, which is the reason this is popular.
  3. Installing directly on the target platform: This is a less preferred method because of the large amount of time it can take and because it needs an . Compiling libraries is a resource intensive process and thus takes a large amount of time (more than 2-3 hours) on a slower processor.

For our project we chose method three for the following reasons:

  1. Raspberry Pi 2 with its quad-core processor is capable of handling these loads.
  2. It was easier to compile on the native system than cross-compiling.
  3. With an internet connectivity adding dependencies was easy using “sudo apt-get install”

After searching the internet for methods to compile the libraries we came across this script to install the libraries and its dependencies.

Raspberry Pi Pin connections

Conveyor Belt

Conveyor belt is a mechanical system with which materials can be transferred from one location to another. Conveyor systems are used in many industrial applications for safely transferring materials where several processing steps happen on the material when it travels on the belt. As the conveyor system is fully automated, it can be ensured to a have a uniform throughput of the deliverable. Industries thus use the conveyors for increasing their production rate. Conveyor systems are very popular in material handling, packaging, automotive, food processing and many other industries.

Basic conveyor structure

Webcam

USB webcam

The USB webcam is an integral part of the Image processing system running as part of the automated assembly line. The camera acts as eyes of the system and is used to capture images, that may be processed later using open CV algorithms. Choice of camera is crucial to the success of the project. A camera with high definition image quality and wide angle view provides better image processing control and lower pixel corruption due to external light related noise. A microsoft HD lifecam 3000 was chosen due to the various high end features it offers and comparatively low cost. Some of the features of this camera are as follows:


  • Widescreen - 16:9 format offers cinematic video recording.
  • TrueColor technology - TrueColor technology automatically delivers bright and colorful video, in virtually all lighting conditions.
  • Universal attachment base - Works on your desktop, laptop, or notebook.
  • Small size and ease of mounting.
  • 2.0 USB connectivity.

Nordic wireless

Nordic communication

SJone board has a nordic wireless chip nRF24L01+ manufactured by the Nordic Semiconductors interfaced to it. Through a Serial peripheral Interface it is possible to operate and configure the Nordic chip. The air data rate, operating modes and RF channel frequency need to be set by configuring the registers in the nRF24L01+.

Air data rate is the modulating signalling rate that is used by nRF24L01+ for transmitting and receiving data. It can be 250 Kbps, 1 Mbps and 2 Mbps. Lower air data rate gives better receiver sensitivity while higher data rate gives a reduced current consumption. The RF channel frequency determines the centre of the channel frequency in which nRF24L01+ operates. Typical operating frequencies are from 2.400GHz to 2.525GHz

For communication to happen,

  • Both nordic transmitter and receiver must be set at the same air data rate. We have set the air data rate at 250 Kbps.
  • Both nordic transmitter and receiver must be operated at the same RF channel frequency.We have set the channel number to 2.499 GHz.

IR Detector

IR Detector
IR Detector Calibration
IR Detector


IR Detector OSEPP IRDET-01 detects obstacles when they are in the calibrated range. The module has 3 pins which are the ground(G), VCC(V) and Signal(S). The signal wire line is digital. It can be either high or low. High if the detected range is further than the calibrated distance, low if the detected range is less than the calibrated distance. The pins are connected to the GND, VCC and GPIO pin of the SJone board, The pin direction is set as input. When no object is detected the pin will read 0 and when object is detected the pin will read 1.

Calibration
CMPE 244 S16 Helios NoObject.png
CMPE 244 S16 Helios object.png

IR detector module device has two potentiometers that can be used to calibrate the device. One for frequency (F) and one for distance (D). Much like an IR Remote, the IR light is oscillated at a defined frequency and detected at that same frequency; this enables the sensor to work in noisy environments. The frequency of the detector module can be adjusted to avoid interference with other detectors or other IR sources.


Turning the frequency potentiometer clockwise increases the frequency and turning it anti clockwise decreases the frequency.The distance at which the sensor is triggered is determined by adjusting the Distance potentiometer. When the sensor is triggered the LED will light up. Turning this pot clockwise increases the triggering distance, turning it counter-clockwise decreases the distance.

Servo Motor

SG90 is a tiny and lightweight servo motor with high output power. Servo can rotate approximately 180 degrees (90 in each direction), and is driven by a PWM wave. It fits in small places and offers good torque. Using servo motors provides precise control of angular movement using relevant duty cycle of a constant frequency PWM wave. Servos consist of an internal motor with a feedback system for position.


SG90 Servo Motor
Servo PWM Waves




The wiring information for SG90 is as follows:

Sl. No Servo Pin Color Purpose
1 Brown Ground
2 Red VCC (max 4.8 Volts)
3 Yellow PWM input to motor


Some of the specifications if SG90 servo are as follows:

  • Weight: 9 g.
  • Dimension: 22.2 x 11.8 x 31 mm approx.
  • Stall torque: 1.8 kgf·cm.
  • Operating speed: 0.1 s/60 degree.
  • Operating voltage: 4.8 V (~5V).


A PWM wave of 50 Hz or 20 ms is needed to drive the servo. The servo works with duty cycle between 5-10%. Provided a wave with duty cycle 5% the motor rotates 90 degree to the left from neutral position (extreme left position), with 10% it rotates 90 degree to the right from neutral position(extreme right position) and with 7.5% duty cycle moves to neutral position.

Stepper Motor

Stepper motor working

A stepper motor is a DC motor which moves in discreet steps. The motor has poles which when energizedFor driving the conveyer system 28BYJ-48 stepper motors were used with ULN2003 darlington array module. The ULN2003 is a high voltage, high current darlington array with 7 NPN darlington pairs for providing high current required by the stepper motor.

28BYJ-48 Stepper motor


Some of the features of the stepper are :

  • Voltage: 5V.
  • Diameter: 28mm.
  • Step angle: 5.625 x 1 / 64.
  • Reduction ratio: 1 / 64
  • 5 Line 4 phase can be driven by ordinary ULN2003 chip.
  • Driven using PWM or GPIO.
  • Simple connections with easy to plug connectors and switch for power.
  • A, B, C, D four-phase LED indicates the status of the stepper motor work.


Hardware Interface

The hardware interface is divided into two parts. Components that were interfaced with SJOne Master board and components that were interfaced with the SJOne Slave board. Both the SJone boards had to be mounted one on each end of the conveyor system. Decsions made by the SJone master board will be implemented by the SJOne slave board.

Building our own conveyor system

Initially it was decided to buy or rent conveyor systems and interface components to it. First option was to buy the conveyor system. On enquiring with many dealers who supplied industrial mini conveyor systems to small scale and large scale industries , it turned out that a simple system was around 1200 US dollars. Second option was to rent the conveyor belt. Renting it for a day would cost around 200 dollars. But the requirement for the project was definitely more than a day. There were development and testing phases of each component that had to be mounted with the conveyor belt and tested frequently. So the options of buying and renting were eliminated. Having an objective of building an economical automated conveyor system, the decision to self build the conveyor system was made by the team.
Nevertheless, it wasn't that easy to build a conveyor system. The conveyor system required the following

  • A rigid frame on both sides to hold the conveyor belt
  • Rotating cylinders that had to be mounted to either ends of the frame.
  • A conveyor belt that could slide on the cylinders bordered by the rigid frame

Interface with SJOne Master Board

The SJOne master board had the following components to it:

  • I2C connection with Raspberry Pi: Raspberry pi passes information about the object detected to the SJOne Master board through I2C. The SJOne board is configured as a slave so that it can receive the information of the color of the object.
  • Nordic transmitter:The information received through I2C is relayed to the slave SJOne board for further sorting of the objects.
  • IR sensor: The IR sensor detects the presence of an object. This is needed because the camera can detect only the valid colors, but there is no way for the camera to know whether there is an object that is of the wrong color or if there are no objects.
  • Servo motor for rejection: The servo motor requires has three pins: VCC, GND and PWM signal. The PWM signal is provided at port 2 pin 1. This servo motor is responsible for rejecting any object that is not blue or green in color. The decision to swing the servo motor arm is based on the information received on the I2C interface and the IR sensor input.

The pin connections for the master SJOne board are as listed in the table below:

Connections SJ One Pin Peripheral Name Peripheral Pin name Description
1 SDA RaspberryPi SDA SDA pin for I2C communication
2 SCL RaspberryPi SCL SCL pin for I2C communication
3 GND RaspberryPi GND Common GND for I2C communication
4 Port 2 Pin 1 Servo Motor Signal PWM for servo motor
5 Port 0 Pin 1 IR sensor Signal Input from IR sensor

Interface with SJOne Slave Board

The SJOne slave board has three interfaces:

  • Nordic wireless receiver:The wireless receiver receives information of the object on the conveyor belt and will take the decision to sort it based on the color of the object
  • IR sensor to detect object: The IR sensor detects the presence of an object. This is needed because the camera can detect only the valid colors, but there is no way for the camera to know whether there is an object that is of the wrong color or if there are no objects.
  • Servo motor for sorting: The servo motor connected to the slave is responsible for sorting the green object, but allow the blue object to pass through. The decision to swing the servo motor arm is based on the information received on the Nordic wireless interface and the IR sensor input.

The pin connections for the slave SJOne board are as listed in the table below:

Connections SJ One Pin Peripheral Name Peripheral Pin name Description
1 Port 2 Pin 1 Servo Motor Signal PWM for servo motor
2 Port 0 Pin 1 IR sensor Signal Input from IR sensor
3 VCC IR sensor and servo motor VCC 3.3 V supply for IR sensor and servo motor
4 GND IR sensor and servo motor GND GNDfor IR sensor and servo motor

Software Design

Assembly line Mechanism

Master SJOne board Flow Sequence
Slave SJOne board Flow Sequence

Object Detection and Analysis with OpenCv and image processing

Image processing is the process of manipulating image data in order to make it suitable for computer vision applications. Computer vision helps to obtain relevant information from images and make decisions based on that information. In our project, we have used OpenCV with C++ to develop our computer vision application. OpenCV is an open source C++ library for image processing and computer vision, originally developed by Intel and now supported by Willow Garage. OpenCV has many inbuilt functions for image processing and computer vision that are mainly aimed at real-time processing.

Digital Image and pixels

A digital image is a collection of pixels. A pixel is the smallest element in a digital image. In an image, each pixel can have a particular value to represent that image. For example, consider a grayscale image where each pixel can be represented by 8-bit value. So, each pixel can have a range of 0 to 255 where 0 is the complete black and 255 is a completely white pixel.

Matrix Representation of GRAYSCALE

A color image consists of 3 planes Red, Green, and Blue. Any color can be formed with the combination these three planes. These planes are called channels of the image. So, an RGB i.e. a color image has 3 channels. Each pixel in RGB has a combination of three values. (255, 0, 0) represent pure red. (0, 255, 0) represent pure green. (255, 0, 255) represents pure violate. Thus, each pixel is represented by 24-bits in an RGB.

RGB Planes
RGB Planes
Color Detection

Object detection and segmentation is the most important task of computer vision. The easiest way to detect and segment the object is through color detection.

The method used for color detection was to first convert the RGB image into HSV and then by thresholding the HSV image, the binary version (Black and White) of the HSV can be obtained. In this binary image, the white space would represent the color which wants to detect. This binary image can then be used to detect the object we are looking using color detection and also to find the shape of the detected object using the available OpenCv library functions.

Need to convert from RGB to HSV

RGB color space describes colors in terms of the amount of red, green, and blue present. HSV color space describes colors in terms of the Hue, Saturation, and Value. The HSV image separates luma (intensity component), from chroma (color component). So that we can now perform histological equalization of the image by adjusting the minimum and maximum values of H, S, and V to such levels that only the pixel with the desired color will be illuminated (white) and the rest of the pixels will be dark. 

void inRange(InputArray src, InputArray lowerb, InputArray upperb, OutputArray dst) library function was used to threshold the HSV image.

Src is the source image. Dst is the thresholded image. The upperb and lowerb boundary parameters are scalars (minimum and maximum values of HSV) which can be determined using trackbars.

HSV and Original Image


Thresholded and Original Image

Trackbars

Trackbar is a very useful tool which allows us to change various parameters while our OpenCv application is running. In our project, we made extensive use of trackbars to determine upper and lower boundary parameters for the particular color which we want to detect. Show the image of trackbars and thresholded image. As shown in the fig. we can change the minimum values of H, S, and V in such a manner that only the desired color will remain as a whitespace and rest of the image should be dark. Once the minimum and maximum values are determined we can lock down these values with inRange() function to get the thresholded image. 

  int createTrackbar(const string& trackbar name, const string& winname, int* value, int count, TrackbarCallback)

is the library function which was used to create the trackbars.

RGB Planes


Trackbarname is the name of the trackbar. Winname is the window name to which trackbar is attached. Integer, pointed by this pointer, holds the value associated with the position of the trackbar. count - The maximum value of the trackbar. The minimum value is always zero.

findContours library function

Using find contours we can retrieve the contours from the binary image that we obtained through thresholding the HSV image. void findContours(InputOutputArray image, OutputArrayOfArrays contours, OutputArray hierarchy, int mode, int method, Point offset=Point()) We can pass appropriate modes to retrieve the different contour information from the image. The contour information is retrieved in the contour is a vector of points. From the contours information, we can find the area of the whitespaces using moments method which is actually the target color and can detect the color.

Algorithm for Color Detection
Step1: Initialize the camera.
Step2: Capture the image frame.
Step3: Convert the image from RGB to HSV.
Step4: Threshold the image for the required color.
Step5: Apply morphological operations to eliminate noise.
Step6: Find contours of the binary image.
Step7: Find the number of whitespace regions using the contour information.
Step8: If no whitespaces found go to step 2, else proceed to step9.
Step9: Check if the area of whitespace is greater than the minimum area. If area is less than the minimum area set, it may be a noise then go to step 2, else proceed to step 10.
Step10: Set object found flag to true. Continue scanning go to step2.

Flowchart for Color Detection

Color Detection Flowchart
Morphological Operations

Filtering is an important part of computer vision. In most of the applications, filtering should be applied to the images before processing on them. In our project, we applied morphological operations to remove noise and to do proper segmentation of the images. The two morphological operations we performed were: Dilation and Erosion.

Dilation: Dilation increases the white pixels along the boundaries of the foreground object. Consider the following example.

Image Dilation

As we can see that dilation increases white pixels along the boundary of the image.

Erosion: Erosion is opposite to dilation. It increases the dark pixels along the boundaries of the foreground object. The example of the erosion is as shown in the figure.

Image Erosion

As we can clearly see the erosion operation increased the black pixels along the boundary of the image. In our project, we applied erosion followed by dilation because the erosion sinks our object, so we dilate it.

Implementation

The implementation of the object color identification was done through image-processing. We used Raspberry pi and OpenCV to perform the image-processing. The image frames were captured from a USB camera connected to the Raspberry pi. Each frame was processed with our OpenCV algorithm to detect the desired object. The information about the detected object was then passed to SJOne board interfaced to the Raspberry pi through I2C. The values for the object present in front of the camera could have three values:

  1. Blue
  2. Green
  3. Invalid

The value invalid is for both when there is no object in front of the camera or an object that does not have the desired color. Once this information is passed to the SJOne Master board the IR sensor connected to this board identifies whether the invalid classification was for an invalid colored object or for an absence of any object. If the decision made computes as an invalid object, the servo motor activates and pushes the object out of the conveyor system else the servo motor remains in its idle position. A similar logic is employed at the slave SjOne board. The information for the color of the object is relayed to this slave board via the Nordic wireless interface. If the object is green, then the sorting servo pushes it away and if it is blue, it lets it pass through.

The flow chart below explains this process graphically.

Helios Implementation Flowchart.png

Testing & Technical Challenges

Testing

Testing for this project was carried out in phases. This was a conscious decision on part of all the team members as previous experiences have taught is that testing everything all at once can lead to problems that could otherwise have been identified earlier. The testing involved the following phases:

Phase 1

Phase once consisted of unit testing each part. The image processing algorithm was first tested on a desktop computer. This was then ported to the Raspberry Pi which resulted in an instant success on the Raspberry Pi. Parallel to this the motor systems, I2C, IR sensor calibration and Nordic wireless messaging were tested and the code was fixed.

Phase 2

Phase 2 testing was to combine the smaller parts of the system. The code for image processing and I2C was merged and tested. At the same time the IR and servo motor was tested so that they worked in conjunction with the messages received over I2C(for the SJOne Master board) and Nordic wireless(for the SJOne slave board). At this point we also ensured the conveyor system was in place and working smoothly.

Final testing

Final testing consisted of combining all the parts and securing them in the housing and testing the rejection system first and then the rejection system was tested. Phased testing enabled us to quickly identify any changes that would be needed when the entire system was in place. This resulted in reducing the time spent on integrating the system.

Technical Challenges

This project was a challenging one due to the varied components involved: from choosing the correct board for image processing to building our own conveyor system. There were a lot of difficult areas in this project. The most significant being the entire housing for all the components and the conveyor system and a lot of the ideas we had at the start had to be changed in order to have the desired result. The most significant challenges are listed below:

Installing OpenCV on ARM 11 (FriendlyARM Tiny6410 SDK)

The Raspberry Pi was not our first choice for a board that can perform image processing, rather it was another board: The FriendlyARM Tiny6410 SDK, it was readily available with us. This board has a Samsung ARM 11 based microcontroller with good memory and decent speed. The board comes with a preinstalled Linux image. The problem was, the documentation for this board is not very comprehensive and the support community is not very active. This meant a lot of trial and error. We tried to install OpenCV but were not successful. The cross compilation kept failing for this board, and eventually we had to give up the idea of using this board due to time constraints in favor of the highly popular Raspberry Pi.

Using a pre-built conveyor system

At the outset, we tried to minimize the amount of custom build required for this project. This meant needed a readymade conveyor system. We found and ordered a conveyor system that could be manually cranked. We thought we could attach motors and drive the conveyor leading to a hassle free conveyor system. The result, however, was the opposite. The conveyor could not be dismantled and so the correct area for the motor drive could not be accessed. We managed to attach a motor at one end, but, by this time the structural integrity of the rubber conveyor belt was compromised and so we could not get the motor to drive the conveyor successfully. Eventually we had to build our own conveyor system and housing for all the components. This was a great build exercise as we decided to choose recyclable material for this and made the assembly using cartons and paper.

Lighting Problem

Lighting was the most critical problem faced during image-processing. In our project, we converted the RGB matrix frames to HSV for the thresholding. The HSV frame extracts the lighting information ‘Luminosity’ and ‘Chroma’ from the RGB planes, so any slight change in the light source angles would alter the thresholding values for HSV frames. Also, we faced a lot of problems due to distortions arising from the light source. We tried to handle the problem by applying morphological operations and implemented the color detection Algorithm. But, still due to slight distortions still present we faced problem in implementing our shape detection Algorithm where even slight distortions would cause problems in detecting the vertices of the detected object.

Conclusion

The learning experience our team wanted to have was, to learn about things we had not done before. This is reflected in the major components in the project, such as the use of image processing. This further led to the realization that SJOne cannot be used for image processing and hence we had to to look for other resources. The use of an embedded linux computer enhanced our understanding of an RTOS. The use of two different Operating Systems: FreeRTOS and Raspbian Jessie helped us in understanding the different use cases for them and how the software complements the hardware. Another takeaway from this project is the build. Building the conveyor system was challenging and needed a lot of original thinking. Cross compiling of OpenCV was another major part of this project and it helped us understand how libraries and compilers work together and help in writing programs easily. This aspect of programming is often neglected in application programming and this project gave us a deep insight into this.In conclusion, it can be said that the project was a success and it has been successful in making us understand Embedded Software better.

Project Video

YouTube

Project Source Code

Sourceforge Source Code Link

References

Acknowledgement

We would like to thank our advisor Preetpal Kang for encouraging us to take on a challenging project.

References Used

Appendix

Retrieved from "http://socialledge.com/sjsu/index.php?title=S16:_Helios&oldid=27102"