Difference between revisions of "S18: Team Nemesis(SJSU Cam)"

From Embedded Systems Learning Academy
Jump to: navigation, search
(Schedule)
(Algorithm:)
 
(235 intermediate revisions by the same user not shown)
Line 1: Line 1:
=== Grading Criteria ===
+
== '''SJSU Cam''' ==
<font color="green">
 
*  How well is Software & Hardware Design described?
 
*  How well can this report be used to reproduce this project?
 
*  Code Quality
 
*  Overall Report Quality:
 
**  Software Block Diagrams
 
**  Hardware Block Diagrams
 
**:  Schematic Quality
 
**  Quality of technical challenges and solutions adopted.
 
</font>
 
  
== Team Nemesis(SJSU Cam) ==
+
[[File:CMPE244_cam_top.png|800px|SJSU_CAM|center]]
  
== Abstract ==
+
== '''Abstract''' ==
This section should be a couple lines to describe what your project does.
+
This project records live stream through camera and displays it on LCD through SJ One microcontroller which is LPC 1758. This camera has longer battery as it runs through portable battery powering microcontroller. As there are many High Quality Cameras available in market, purpose of this camera is found in applications where streaming of medium quality video/photo is required for longer period of time and camera has to survive longer on single battery, which can’t be provided by handy digital camera. Another functionality provided by this project is to enable user to capture images on SD card and view it on LCD providing different resolution and modes as option through interactive UI designed on LCD. Project Hardware enclosed in small box 3D designed for this project enables portability and ease of access to user.
 +
 
 +
== '''Objectives & Introduction''' ==
 +
The objective of this project was to learn how timing and buffer play an important role in capturing the pixels from the camera and displaying them on the LCD screen. The following objectives were set for the project:
 +
 
 +
<br/>1. Write a driver for the SPI camera ov2640, which has a better rate of capturing the image with the SJONE board.
 +
<br/>2. Write the driver for the LCD display (Adafruit 5' TFT)  and integrate it by choosing the right resolution.
 +
<vr/>3. Implement a menu class for different options of the camera on the LCD and program external buttons for navigating across the menu.
 +
<br/>3. Integrate the camera and LCD by displaying the output of the camera on the LCD.
 +
<br/>4. Store the output of the camera on the SD card, if capture mode is selected from LCD menu.
 +
<br/>5. Read the image from the SD card and display on the screen.
  
== Objectives & Introduction ==
 
Show list of your objectives.  This section includes the high level details of your project.  You can write about the various sensors or peripherals you used to get your project completed.
 
  
 
=== Team Members & Responsibilities ===
 
=== Team Members & Responsibilities ===
 
*  Chhavi Mehta
 
*  Chhavi Mehta
**   
+
**  Camera implementation, integration with SD card
*  Deeksha Prakash
+
*  Deeksha Prakash Kankalale
**   
+
**  PCB Design, Camera driver implementation.
*  Parth
+
*  Parth Gujar
**   
+
**  LCD Driver, 3D design.
*  Sagar
+
*  Sagar Kalathia
**
+
**   LCD interfacing, implementing graphics library
*  Saurabh
+
*  Saurabh Badenkal
**   
+
**  LCD Driver, Integration of camera and LCD and data pipelining.
 +
 
 +
== '''Schedule''' ==
  
== Schedule ==
 
Show a simple table or figures that show your scheduled as planned before you started working on the project.  Then in another table column, write down the actual schedule so that readers can see the planned vs. actual goals.  The point of the schedule is for readers to assess how to pace themselves if they are doing a similar project.
 
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 42: Line 40:
 
|-
 
|-
 
! scope="row"| 1
 
! scope="row"| 1
| 4/10
+
| 4/17
 
| 1. Placing order for the components required in the project. <br/> 2. Division of module development.
 
| 1. Placing order for the components required in the project. <br/> 2. Division of module development.
| Completed! Problems Encountered:  
+
| Completed! <br/> Problems Encountered: Finding the right components after going through various datasheets.
 
|-
 
|-
 
! scope="row"| 2
 
! scope="row"| 2
| 4/17
+
| 4/24
| 1. Interfacing camera with SJONE board. <br/> 2. Establishing communication with UART USB. <br/> 3. Writing a python script and use OpenCV to display image.<br/>4. Implement different modes to handle the fames per second and speed of capture.
+
| 1. Making interface with LCD.<br/>2. Drawing simple shapes. <br/>3. Work on the overall design of the menu for LCD.
| CompletedProblems Encountered?
+
| Completed! <br/> Problems Encountered: Setting appropriate initial frequency for the LCD to light up.
 
|-
 
|-
 
! scope="row"| 3
 
! scope="row"| 3
| 4/24
+
| 5/1
| 1. Making interface with LCD.<br/>2. Drawing simple shapes. <br/>3. Image processing parts of the driver.<br/>4. Write mode change driver to walk through the menu.
+
| 1. Interfacing the camera <br/>2. Implementing menu class with sub-menus and items. <br/>3. Designing hardware prototype for 3D printing.
| Completed? Problems Encountered?
+
| Completed! <br/>  Problems Encountered: Not enough RAM for displaying more than 1 picture of size 100x100, Using interrupts for UART receive instead of normal function call.
 
|-
 
|-
 
! scope="row"| 4
 
! scope="row"| 4
| 5/1
+
| 5/8
| 1. Implementing menu which will also have object detection and image processing options. <br/>2. Hardware buttons to make the interface with the user(arrows, select and menu).  
+
| 1. Design PCB and place order. <br/>2. Integrating file explorer of the SD card onto LCD with GUI.<br/>3. Capturing image from the camera, saving it on the SD card.  
| CompletedProblems Encountered?
+
| Completed! <br/> Problems Encountered: Heavy resource utilization while saving and retrieving the file from SD card.
 
|-
 
|-
 
! scope="row"| 5
 
! scope="row"| 5
| 5/8
+
| 5/15
| 1. Design PCB and place order. <br/>2. Implement the storage on the SD card.<br/>3. Putting everything together and making the working camera which capture image or video, and saving it on the SD card.  
+
| 1. Displaying the image on the LCD. <br/>2. Integrating the modules together with the 3D case having the LCD on one side and SJONE and camera on the other with buttons.
| CompletedProblems Encountered?
+
| Completed! <br/> Problems Encountered: Resource constraints only allow a resolution of 320p. Extremely low frame rate due to SD read speeds. 
 
|-
 
|-
 
! scope="row"| 6
 
! scope="row"| 6
| 5/15
+
| 5/22
| 1. Integrating the modules together.<br/>2. The whole case has LCD & driver and other side has SJONE and camera, perpendicular to this is the button with its PCB board. <br/>3. Final testing and debugging of the system as a whole.
+
| 1. Final testing and debugging of the system as a whole.
| CompletedProblems Encountered?
+
| Completed! <br/> Problems Encountered: PCB arrived late.
 
|}
 
|}
  
== Parts List & Cost ==
+
== '''Parts List & Cost''' ==
Give a simple list of the cost of your project broken down by components. Do not write long stories here.
+
 
 +
{| class="wikitable"
 +
|-
 +
! scope="col"| Components
 +
! scope="col"| Model Number
 +
! scope="col"| Cost
 +
|-
 +
! scope="row"| SJONE board
 +
| LPC 1758
 +
|      80$
 +
|-
 +
! scope="row"| LCD
 +
| 5 inch TFT LCD screen
 +
|      32$
 +
|-
 +
! scope="row"| LCD driver
 +
| RA8875
 +
|      10$
 +
|-
 +
! scope="row"| Camera
 +
| OV 2640
 +
|      20$
 +
|-
 +
! scope="row"| PCB
 +
|    -
 +
| 50$
 +
|}
  
 
== Design & Implementation ==
 
== Design & Implementation ==
The design section can go over your hardware and software design.  Organize this section using sub-sections that go over your design and implementation.
+
This design section outlines the hardware and software design of the project.   
  
 
=== Hardware Design ===
 
=== Hardware Design ===
Discuss your hardware design hereShow detailed schematics, and the interface here.
+
 
 +
The main hardware components used in the project are discussed in detail below.
 +
 
 +
''LCD Display:''
 +
This is a 5.0" TFT screen with 800x480 pixels. These screens are commonly seen in consumer electronics, such as miniature TV's, GPS's, handheld games car displays, etc. A 40-pin connector has 8 red, 8 green, and 8 blue parallel pins, for 24-bit color capability. The display is supposed to be constantly refreshed, at 60Hz, with a pixel clock. The backlight requires a constant-current mode boost converter that can go as high as 24V. Hence along with the display we use an RA8875 driver board.
 +
 
 +
[[File:CMPE244_nemesis_l.jpg|thumb|400px|center|Figure 1: 5 inch LCD screen]]
 +
 
 +
 
 +
''LCD Driver :''
 +
The RA8875 is a powerful TFT driver chip, it has 768KB of RAM, so it can buffer the display (and depending on the screen size also have double overlaying).The RA8875 can also handle standard 4-wire over the SPI interface.
 +
 
 +
[[File:CMPE244_nemesis_ld.jpg|thumb|400px|Figure 2: RA8875 Driver|center]]
 +
 
 +
 
 +
 
 +
 
 +
''Camera:''
 +
The OV2640 Camera Module is a small image sensor, works at low operating voltages. It provides all functions of a single chip of VGA camera and image processor. Through Serial Camera bus control, the sensor can output the whole frame, sampling at various resolution 8 bits of dataThe product VGA image can reach up to a maximum of 30 frames per second. We can control the image quality, data format, and transmission mode. All the process of image processing functions can through the SCCB programming interface, including gamma curve, white balance, saturation, and chrome.
 +
 
 +
[[File:CMPE244_nemesis_c.jpg|thumb|300px|Figure 3: OV2640 camera|center]]
 +
 
 +
 
 +
[[File:CMPE244_nemesis_Cam_Module.jpg|thumb|400px|center|Figure 4: Block diagram of the camera module]]
 +
 
 +
OV2640 support 4 output format: YcbCr422,YCbCr420, RGB565, Bayer raw RGB and GRB, YUV422 JPEG. How to choose the right output format for camera phone design or other applications? The data format at LCD driver are always RGB. For example, RGB444, RGB565, RGB555, RGB888 etc. The data format and memory interface are always JPEG. The JPEG data is compressed from YCbCr data. So Both RGB and YCbCr data are needed inside the back-end chip. The “X” block is different for different back-end chips.
 +
 
 +
The general diagram of back-end chip is as below:
 +
 
 +
[[File:CMPE244_nemesis_backend-cam.png|thumb|400px|center|Figure 5: Back-end chip ]]
 +
 
 +
 
 +
'''System block diagram :''' The integral part of the entire system is the SJOne board which is the main controller for the LCD as well as the camera. Both the LCD and camera are interfaced via an SPI bus with the SJOne board. Apart from the power supply and ground pins, the LCD has one pins, namely, reset which is connected to the general purpose Input-Output pin of the controller. This pin is used while initializing the display.
 +
 
 +
 
 +
[[File:CMPE244_nemesis_block.png|thumb|800px|center|System Block diagram]]
  
 
=== Hardware Interface ===
 
=== Hardware Interface ===
In this section, you can describe how your hardware communicates, such as which BUSes used.  You can discuss your driver implementation here, such that the '''Software Design''' section is isolated to talk about high level workings rather than inner working of your project.
 
  
=== Software Design ===
+
The PCB design was done on Dip Trace, which is a CAD software for creating schematic diagrams and printed circuit boards. The SPI interface is done to both the camera and the driver board of the LCD display.
Show your software design. For example, if you are designing an MP3 Player, show the tasks that you are using, and what they are doing at a high level. Do not show the details of the codeFor example, do not show exact code, but you may show psuedocode and fragments of codeKeep in mind that you are showing DESIGN of your software, not the inner workings of it.
+
 
 +
The PCB was designed in such a way that it can be powered by two sources, USB from PC or from PowerBank, by flipping a switch. Using this feature SJONE can be directly connected to PC via the PCB without any direct connection from PC to SJONE, for easier deployment of code once the whole project was packaged inside the camera box.
 +
 
 +
 
 +
[[File:CMPE244_nemesis_Schemetic.jpeg|thumb|1000px|Schematic|center]]
 +
 
 +
 
 +
 
 +
  [[File:CMPE244_nemesis_pcb.jpeg|thumb|1000px|PCB Design|left]]
 +
 
 +
  [[File:Cmpe244_S18_NEMESIS_PCB_3D.jpeg|thumb|500px|3D view of PCB Design|right]]
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
 
 +
=== Software Design and Flowchart ===
 +
 
 +
The general flow of our project can be explained by the flowchart shown below:
 +
 
 +
 
 +
[[File:CMPE244_Nemesis_Flowchart.jpg|thumb|600px|center|Program Flow]]
  
 
=== Implementation ===
 
=== Implementation ===
This section includes implementation, but again, not the details, just the high level. For example, you can list the steps it takes to communicate over a sensor, or the steps needed to write a page of memory onto SPI Flash. You can include sub-sections for each of your component implementation.
+
The camera is interfaced to the board where Image Sensor Configuration is done through I2C Protocol and Sending pixel data through SPI communication protocol. The raw pixels are read from the camera to the SJONE board. There are five buttons on the camera. One of them used to capture an image and store it on the SD card. The LCD screen is interfaced through SPI communication protocol. The raw pixels received from the camera are displayed on to the LCD screen.
 +
 
 +
===  Algorithm: ===
 +
 
 +
The first step for making the LCD and the camera work is the initialization of these components. The algorithm for it is as described:
 +
 
 +
'''1. Initialization of the LCD: '''
 +
* According to the desired resolution(as the LCD driver supports two resolutions 800x480 & 480x272), the resolution is first set and the basic touch.non-touch config is made. In our case the LCD is non-touch.
 +
* After setting the chip-select and reset pins as outputs, the SPI is initialized. SSP1 was configured to be used for LCD interfacing.
 +
* Register at address 0x00 is read for getting the version of the LCD over SPI.
 +
* PLL is initialized and the HSYNC, VSYNC and Pixel clock([https://www.nxp.com/wcm_documents/techzones/microcontrollers-techzone/Presentations/graphics.lcd.technologies.pdf terminologies described here]) parameters are set according to the resolution chosen. (as per the [https://cdn-shop.adafruit.com/datasheets/RA8875_DS_V19_Eng.pdf RA8875 ADafruit Datasheet])
 +
* Active window X and window Y are set by writing to the appropriate registers.
 +
* Clear the entire window.
 +
* Set SPI clock frequency.{There's a [https://github.com/saurabhrb/CMPE_244_Project_CAM/blob/master/FINAL%20DEMOed%20Code/LCD.cpp#L87 trick] in this as well, the adafruit RA8875 first responds to 2MHz SPI only, and after initializing the PLL and other timing related configs, it listens to up to 30MHz SPI}
 +
 
 +
'''2. Initialization of the Camera: '''
 +
* I2C is initialized for Image Sensor Configuration, this configuration was retrieved from the [Software notes of OV2640 Camera https://github.com/saurabhrb/CMPE_244_Project_CAM/blob/master/OV2640_fifo/OV2640%20Camera%20Module%20Software%20Application%20Notes.pdf].
 +
* SPI is initialized with the desired frequency(only up to 8MHz) for receiving the pixel data from the buffer.
 +
* According to the resolution, appropriate I2C sensor registers are set later.
 +
 
  
== Testing & Technical Challenges ==
 
Describe the challenges of your project.  What advise would you give yourself or someone else if your project can be started from scratch again?
 
Make a smooth transition to testing section and described what it took to test your project.
 
  
Include sub-sections that list out a problem and solution, such as:
+
In Terms of Pseudo Code & Algorithm, the following is the methodology we followed which consists of API's and its functions. The design is mainly of two parts:
  
=== <Bug/issue name> ===
 
Discuss the issue and resolution.
 
  
== Conclusion ==
+
'''1. Design for LCD''' :
Conclude your project here. You can recap your testing and problems. You should address the "so what" part here to indicate what you ultimately learnt from this project. How has this project increased your knowledge?
+
LCD would continuously display a menu on its right with different options based on the currently selected menu. Menu driver is programmed dynamically using pointers & function Pointer to create sub-menus and sub-items inside sub-menus. A main menu object pointer is created in the main function which is propagated along the menu-tree to choose which should be the menu which needs to be displayed on the LCD currently.
 +
 
 +
Various API & functions as follows:
 +
 
 +
* ''Add_Menu'' API, whenever gets called would add one more node to the linked list.
 +
 
 +
* ''Add_Submenu'' API would create Linked List inside that Node.
 +
 
 +
* "Navigate" API enables a user to navigate through Menus. This API navigates pointer to different nodes and calling respective functions such as capture image or going inside sub_menu.
 +
 
 +
[[File:Cmpe244_S18_NEMESIS_LCD_MENU_DRIVER.jpg|thumb|600px|LCD Menu working|center]]
 +
 
 +
With this dynamic functionality, we can create menus as required inside which we can call the respective function to be performed according to the submenu.
 +
 
 +
 
 +
 
 +
''Psuedo Code for LCD_Menu - ''
 +
 
 +
A menu was designed for the LCD screen using an LCD_MENU class. This class included functions like adding sub-menu, displaying them or loading previous ones. It helped in dynamically adding options for the innumerous camera settings like changing image resolution and format.The private members of this class are as shown:
 +
 
 +
    private:
 +
    type myType;                        //to know if current object is itemType or menuType
 +
    str myName;                          //name of this object
 +
    LCD_MENU *ParentMenu;                //parent menu of current menu object
 +
    LCD_MENU *CurrentMenu;              //reference to global CurrentMenuG
 +
    VECTOR<Element> all_elements;        //to store all elements by Element .name .id and .type
 +
    VECTOR<str> all_elements_name;      //to store all elements by name to use by Find function
 +
    int highlighted;                    //to track currently highlighted sub-element
 +
 
 +
Similar to this class, a file_explorer class was implemented for navigating through the SD card files.
 +
 
 +
 
 +
 
 +
'''2. Design for SJOne & CAM''' :
 +
The controller has to Configure camera, receive data from the camera and send this to LCD as soon as it gets the data.
 +
 
 +
Function includes :
 +
 
 +
* Configuration of Camera Module by configuring different registers specified in datasheet & setting image format.
 +
 
 +
* To receive Data from Camera at 8 MHz through SPI. After receiving One Frame, it is sent to LCD via SPI. In both the case, Controller is Master and another device is a slave.
 +
 
 +
* To Store Image in SD card, the whole frame is sent to SD card through SPI, and a new text file with a unique name is created and pixel values in that text file would be ASCII characters coming from 16-bit image buffer.
 +
 
 +
== '''Testing & Technical Challenges''' ==
 +
 
 +
 
 +
''' Problem 1: ''' Making any camera work on SJOne can be little tricky, due to its less RAM to store one full B/W or RGB frame. Also interfacing the regular CMOS camera modules with Pixel Clks requires perfect timing to get a proper or any image at all.
 +
 
 +
''' Solution: ''' The solution was to use the already buffered and stored camera frame and then transfer it one by one based on SJONE's reading speed(via SPI/I2C) to display it on the LCD pixel by pixel. After trying a number of cameras like OV7670 without fifo and VC0706 serial camera(which had unsupported output format), we were able to solve the problem by using OV2640 SPI camera with fifo buffer which made capturing and retrieving the image easier because the buffer saved the whole frame, and cleared only after SJONE has finished reading the frame.
 +
 
 +
''' Problem 2: ''' The camera displayed two frames on the LCD screen for resolutions greater than 352x288. This was because the buffer size of the camera could only hold upto 16bits pixels of max 352x288 resolution, as the buffer size is [http://www.arducam.com/arducam-mini-released/ ~384KBytes].
 +
 
 +
''' Solution: ''' This could be avoided by upgrading the buffer size on the camera module or by using an external buffer to store the whole image before previewing on LCD.
 +
 
 +
''' Problem 2: ''' The camera was displaying image in form of random pixel values.
 +
 
 +
''' Solution: ''' After taking a closer look at datasheet we realized that we were configuring the CMOS registers for wrong/unsupported format (by LCD) and not RGB565 format. After configuring the registers for RGB565 with different resolution types, the LCD displayed proper images.
 +
 
 +
== '''Conclusion''' ==
 +
 
 +
We have successfully designed and implemented a digital camera with a number of features. This project has helped us gain insight knowledge of the implementation of FreeRTOS concepts like prioritizing tasks and using semaphore and mutex for SD card reading and writing. We learned to implement communication protocol drivers such as SPI. Working with pixel data helped us go into deeper details of the SPI protocol as each byte had to be handled accurately. We have tried the PCB design on both Eagle and Dip Trace, we were able to complete the PCB design starting from schematic diagram to 3D imaging of the PCB. We were also able to learn the CAD design required for 3D printing.
 +
 
 +
Apart from the technical aspects of the project, a sense of cordial teamwork was also developed amongst the team members which led to the successful accomplishment of the project's objectives.
  
 
=== Project Video ===
 
=== Project Video ===
Upload a video of your project and post the link here.
+
 
 +
*  [https://youtu.be/GtHGJm6FQZo SJSU_CAM Video Link]
  
 
=== Project Source Code ===
 
=== Project Source Code ===
*  [https://sourceforge.net/projects/sjsu/files/CmpE_S2016/ Sourceforge Source Code Link]
+
*  [https://github.com/saurabhrb/CMPE_244_Project_CAM SJSU_CAM Github Link]
  
 
== References ==
 
== References ==
 
=== Acknowledgement ===
 
=== Acknowledgement ===
Any acknowledgement that you may wish to provide can be included here.
+
We would like to thank our professor, Preetpal Kang for designing such a course where learning was his guidance and consistent feedback. We would also like to thank the ISA team for their valuable inputs as per their own experience of the course when taken and their support and motivation at every phase of this project. There were some unsung, selfless classmates who helped us whenever necessary, whenever any immediate hardware was required or any other guidance needed in spite of being from the rival teams. We would like to thank each one of them who forgot their differences or considered the course and project as a competition to selflessly help us when in need.
  
 
=== References Used ===
 
=== References Used ===
List any references used in project.
+
[1] Preetpal Kang's lecture notes of CMPE 244, Computer Engineering, San Jose State University, Jan-May 2018.
 +
 
 +
[2] [http://www.arducam.com/camera-modules/2mp-ov2640/ OV2640 Technical Description]
 +
 
 +
[3] [https://www.adafruit.com/product/1680 5.0" LCD & Datasheet]
  
=== Appendix ===
+
[4] [https://cdn-shop.adafruit.com/datasheets/KD50G21-40NT-A1.pdf/ RA8875 LCD Driver Description]
You can list the references you used.
 

Latest revision as of 18:20, 27 May 2018

SJSU Cam

SJSU_CAM

Abstract

This project records live stream through camera and displays it on LCD through SJ One microcontroller which is LPC 1758. This camera has longer battery as it runs through portable battery powering microcontroller. As there are many High Quality Cameras available in market, purpose of this camera is found in applications where streaming of medium quality video/photo is required for longer period of time and camera has to survive longer on single battery, which can’t be provided by handy digital camera. Another functionality provided by this project is to enable user to capture images on SD card and view it on LCD providing different resolution and modes as option through interactive UI designed on LCD. Project Hardware enclosed in small box 3D designed for this project enables portability and ease of access to user.

Objectives & Introduction

The objective of this project was to learn how timing and buffer play an important role in capturing the pixels from the camera and displaying them on the LCD screen. The following objectives were set for the project:


1. Write a driver for the SPI camera ov2640, which has a better rate of capturing the image with the SJONE board.
2. Write the driver for the LCD display (Adafruit 5' TFT) and integrate it by choosing the right resolution. <vr/>3. Implement a menu class for different options of the camera on the LCD and program external buttons for navigating across the menu.
3. Integrate the camera and LCD by displaying the output of the camera on the LCD.
4. Store the output of the camera on the SD card, if capture mode is selected from LCD menu.
5. Read the image from the SD card and display on the screen.


Team Members & Responsibilities

  • Chhavi Mehta
    • Camera implementation, integration with SD card
  • Deeksha Prakash Kankalale
    • PCB Design, Camera driver implementation.
  • Parth Gujar
    • LCD Driver, 3D design.
  • Sagar Kalathia
    • LCD interfacing, implementing graphics library
  • Saurabh Badenkal
    • LCD Driver, Integration of camera and LCD and data pipelining.

Schedule

Week# Date Task Actual
1 4/17 1. Placing order for the components required in the project.
2. Division of module development.
Completed!
Problems Encountered: Finding the right components after going through various datasheets.
2 4/24 1. Making interface with LCD.
2. Drawing simple shapes.
3. Work on the overall design of the menu for LCD.
Completed!
Problems Encountered: Setting appropriate initial frequency for the LCD to light up.
3 5/1 1. Interfacing the camera
2. Implementing menu class with sub-menus and items.
3. Designing hardware prototype for 3D printing.
Completed!
Problems Encountered: Not enough RAM for displaying more than 1 picture of size 100x100, Using interrupts for UART receive instead of normal function call.
4 5/8 1. Design PCB and place order.
2. Integrating file explorer of the SD card onto LCD with GUI.
3. Capturing image from the camera, saving it on the SD card.
Completed!
Problems Encountered: Heavy resource utilization while saving and retrieving the file from SD card.
5 5/15 1. Displaying the image on the LCD.
2. Integrating the modules together with the 3D case having the LCD on one side and SJONE and camera on the other with buttons.
Completed!
Problems Encountered: Resource constraints only allow a resolution of 320p. Extremely low frame rate due to SD read speeds.
6 5/22 1. Final testing and debugging of the system as a whole. Completed!
Problems Encountered: PCB arrived late.

Parts List & Cost

Components Model Number Cost
SJONE board LPC 1758 80$
LCD 5 inch TFT LCD screen 32$
LCD driver RA8875 10$
Camera OV 2640 20$
PCB - 50$

Design & Implementation

This design section outlines the hardware and software design of the project.

Hardware Design

The main hardware components used in the project are discussed in detail below.

LCD Display: This is a 5.0" TFT screen with 800x480 pixels. These screens are commonly seen in consumer electronics, such as miniature TV's, GPS's, handheld games car displays, etc. A 40-pin connector has 8 red, 8 green, and 8 blue parallel pins, for 24-bit color capability. The display is supposed to be constantly refreshed, at 60Hz, with a pixel clock. The backlight requires a constant-current mode boost converter that can go as high as 24V. Hence along with the display we use an RA8875 driver board.

Figure 1: 5 inch LCD screen


LCD Driver : The RA8875 is a powerful TFT driver chip, it has 768KB of RAM, so it can buffer the display (and depending on the screen size also have double overlaying).The RA8875 can also handle standard 4-wire over the SPI interface.

Figure 2: RA8875 Driver



Camera: The OV2640 Camera Module is a small image sensor, works at low operating voltages. It provides all functions of a single chip of VGA camera and image processor. Through Serial Camera bus control, the sensor can output the whole frame, sampling at various resolution 8 bits of data. The product VGA image can reach up to a maximum of 30 frames per second. We can control the image quality, data format, and transmission mode. All the process of image processing functions can through the SCCB programming interface, including gamma curve, white balance, saturation, and chrome.

Figure 3: OV2640 camera


Figure 4: Block diagram of the camera module

OV2640 support 4 output format: YcbCr422,YCbCr420, RGB565, Bayer raw RGB and GRB, YUV422 JPEG. How to choose the right output format for camera phone design or other applications? The data format at LCD driver are always RGB. For example, RGB444, RGB565, RGB555, RGB888 etc. The data format and memory interface are always JPEG. The JPEG data is compressed from YCbCr data. So Both RGB and YCbCr data are needed inside the back-end chip. The “X” block is different for different back-end chips.

The general diagram of back-end chip is as below:

Figure 5: Back-end chip


System block diagram : The integral part of the entire system is the SJOne board which is the main controller for the LCD as well as the camera. Both the LCD and camera are interfaced via an SPI bus with the SJOne board. Apart from the power supply and ground pins, the LCD has one pins, namely, reset which is connected to the general purpose Input-Output pin of the controller. This pin is used while initializing the display.


System Block diagram

Hardware Interface

The PCB design was done on Dip Trace, which is a CAD software for creating schematic diagrams and printed circuit boards. The SPI interface is done to both the camera and the driver board of the LCD display.

The PCB was designed in such a way that it can be powered by two sources, USB from PC or from PowerBank, by flipping a switch. Using this feature SJONE can be directly connected to PC via the PCB without any direct connection from PC to SJONE, for easier deployment of code once the whole project was packaged inside the camera box.


Schematic


PCB Design
3D view of PCB Design






 






 






 





 

Software Design and Flowchart

The general flow of our project can be explained by the flowchart shown below:


Program Flow

Implementation

The camera is interfaced to the board where Image Sensor Configuration is done through I2C Protocol and Sending pixel data through SPI communication protocol. The raw pixels are read from the camera to the SJONE board. There are five buttons on the camera. One of them used to capture an image and store it on the SD card. The LCD screen is interfaced through SPI communication protocol. The raw pixels received from the camera are displayed on to the LCD screen.

Algorithm:

The first step for making the LCD and the camera work is the initialization of these components. The algorithm for it is as described:

1. Initialization of the LCD:

  • According to the desired resolution(as the LCD driver supports two resolutions 800x480 & 480x272), the resolution is first set and the basic touch.non-touch config is made. In our case the LCD is non-touch.
  • After setting the chip-select and reset pins as outputs, the SPI is initialized. SSP1 was configured to be used for LCD interfacing.
  • Register at address 0x00 is read for getting the version of the LCD over SPI.
  • PLL is initialized and the HSYNC, VSYNC and Pixel clock(terminologies described here) parameters are set according to the resolution chosen. (as per the RA8875 ADafruit Datasheet)
  • Active window X and window Y are set by writing to the appropriate registers.
  • Clear the entire window.
  • Set SPI clock frequency.{There's a trick in this as well, the adafruit RA8875 first responds to 2MHz SPI only, and after initializing the PLL and other timing related configs, it listens to up to 30MHz SPI}

2. Initialization of the Camera:


In Terms of Pseudo Code & Algorithm, the following is the methodology we followed which consists of API's and its functions. The design is mainly of two parts:


1. Design for LCD : LCD would continuously display a menu on its right with different options based on the currently selected menu. Menu driver is programmed dynamically using pointers & function Pointer to create sub-menus and sub-items inside sub-menus. A main menu object pointer is created in the main function which is propagated along the menu-tree to choose which should be the menu which needs to be displayed on the LCD currently.

Various API & functions as follows:

  • Add_Menu API, whenever gets called would add one more node to the linked list.
  • Add_Submenu API would create Linked List inside that Node.
  • "Navigate" API enables a user to navigate through Menus. This API navigates pointer to different nodes and calling respective functions such as capture image or going inside sub_menu.
LCD Menu working

With this dynamic functionality, we can create menus as required inside which we can call the respective function to be performed according to the submenu.


Psuedo Code for LCD_Menu -

A menu was designed for the LCD screen using an LCD_MENU class. This class included functions like adding sub-menu, displaying them or loading previous ones. It helped in dynamically adding options for the innumerous camera settings like changing image resolution and format.The private members of this class are as shown:

   private:
   type myType;                         //to know if current object is itemType or menuType
   str myName;                          //name of this object
   LCD_MENU *ParentMenu;                //parent menu of current menu object
   LCD_MENU *CurrentMenu;               //reference to global CurrentMenuG
   VECTOR<Element> all_elements;        //to store all elements by Element .name .id and .type
   VECTOR<str> all_elements_name;       //to store all elements by name to use by Find function
   int highlighted;                     //to track currently highlighted sub-element

Similar to this class, a file_explorer class was implemented for navigating through the SD card files.


2. Design for SJOne & CAM : The controller has to Configure camera, receive data from the camera and send this to LCD as soon as it gets the data.

Function includes :

  • Configuration of Camera Module by configuring different registers specified in datasheet & setting image format.
  • To receive Data from Camera at 8 MHz through SPI. After receiving One Frame, it is sent to LCD via SPI. In both the case, Controller is Master and another device is a slave.
  • To Store Image in SD card, the whole frame is sent to SD card through SPI, and a new text file with a unique name is created and pixel values in that text file would be ASCII characters coming from 16-bit image buffer.

Testing & Technical Challenges

Problem 1: Making any camera work on SJOne can be little tricky, due to its less RAM to store one full B/W or RGB frame. Also interfacing the regular CMOS camera modules with Pixel Clks requires perfect timing to get a proper or any image at all.

Solution: The solution was to use the already buffered and stored camera frame and then transfer it one by one based on SJONE's reading speed(via SPI/I2C) to display it on the LCD pixel by pixel. After trying a number of cameras like OV7670 without fifo and VC0706 serial camera(which had unsupported output format), we were able to solve the problem by using OV2640 SPI camera with fifo buffer which made capturing and retrieving the image easier because the buffer saved the whole frame, and cleared only after SJONE has finished reading the frame.

Problem 2: The camera displayed two frames on the LCD screen for resolutions greater than 352x288. This was because the buffer size of the camera could only hold upto 16bits pixels of max 352x288 resolution, as the buffer size is ~384KBytes.

Solution: This could be avoided by upgrading the buffer size on the camera module or by using an external buffer to store the whole image before previewing on LCD.

Problem 2: The camera was displaying image in form of random pixel values.

Solution: After taking a closer look at datasheet we realized that we were configuring the CMOS registers for wrong/unsupported format (by LCD) and not RGB565 format. After configuring the registers for RGB565 with different resolution types, the LCD displayed proper images.

Conclusion

We have successfully designed and implemented a digital camera with a number of features. This project has helped us gain insight knowledge of the implementation of FreeRTOS concepts like prioritizing tasks and using semaphore and mutex for SD card reading and writing. We learned to implement communication protocol drivers such as SPI. Working with pixel data helped us go into deeper details of the SPI protocol as each byte had to be handled accurately. We have tried the PCB design on both Eagle and Dip Trace, we were able to complete the PCB design starting from schematic diagram to 3D imaging of the PCB. We were also able to learn the CAD design required for 3D printing.

Apart from the technical aspects of the project, a sense of cordial teamwork was also developed amongst the team members which led to the successful accomplishment of the project's objectives.

Project Video

Project Source Code

References

Acknowledgement

We would like to thank our professor, Preetpal Kang for designing such a course where learning was his guidance and consistent feedback. We would also like to thank the ISA team for their valuable inputs as per their own experience of the course when taken and their support and motivation at every phase of this project. There were some unsung, selfless classmates who helped us whenever necessary, whenever any immediate hardware was required or any other guidance needed in spite of being from the rival teams. We would like to thank each one of them who forgot their differences or considered the course and project as a competition to selflessly help us when in need.

References Used

[1] Preetpal Kang's lecture notes of CMPE 244, Computer Engineering, San Jose State University, Jan-May 2018.

[2] OV2640 Technical Description

[3] 5.0" LCD & Datasheet

[4] RA8875 LCD Driver Description