Object Detection System with ESP32-CAM and Tensorflow

Published on


In the this tutorials of the ESP32-CAM series, we saw that using the original code, we will be able to process face image from face recognition to face separation, but in cases where we need to recognize different objects, different models must be introduced to our code. To be able to identify the objects we want with self-learning. But no matter how high the processing power of ESP chips, we can not leave all this complex processing to this small chip, so we will use Tensorflow.JS to combine it with the video sent from ESP32-CAM. Note that in this tutorial, Tensorflow.JS runs in the computer browser and therefore the machine learning model runs inside your browser. In this tutorial, we will use the COCO-SSD model to identify objects in a video stream from ESP32-CAM. Visit CiferTech for more tutorials, and be sure to follow my Instagram page to support me. ^-^

Object Detection System with ESP32-CAM and Tensorflow




TensorFlow allows developers to create data flow diagrams, structures that describe how data moves through diagrams or a series of processing nodes. Each node in the diagram represents a mathematical operation, and each connection or edge between nodes is a multidimensional or tensile data array. TensorFlow provides all of this process through the Python language. It is easy to learn and work with Python and provides convenient ways to express high-level abstractions together. But real math is not done in Python. The libraries available for TensorFlow are written in high-performance C binary. Python only directs traffic between components and provides high-level programming abstracts to connect them.


pre-trained models

In machine learning, there is a topic called model, which is actually the objects and things that we want to recognize, for example, recognizing the human face, so for this we have to teach our machine that the appearance of a human, female or male, Small or large, old or young. For this we have to introduce different images to our car to increase the ability to detect our car. This result of this process is called modeling, which is the hallmark of our machine, and then all of these models must be labeled so that the result is visually understandable to the user.




ESP32 is the advanced generation of ESP8266. One of the differences is its built-in Bluetooth. It also has a 2.4 GHz WiFi core and built-in Bluetooth with 40-nanometer technology from TSMC. This module has the best performance in energy consumption, that is, it brings the best results for us with the least energy consumption. If we want to take a closer look at this board, we must say that this is a chip in which the NodeMCU platform is implemented, which is also called System on a chip microcontrollers.



How to detect objects with the camera

In this project, by streaming images using the ESP32-CAM board and receiving and displaying them in the browser, we will also use Tensorflow.JS to process images using the default models applied. As soon as the image is received by the web server running in the browser, it will anticipate and analyze the items in the image. In this project, only the default objects are recognizable. As you may know, Tensorflow has several pre-trained models that we can use to easily start learning the machine. COCO-SSD is an ML model used to localize and identify objects in an image. The same model is used in this tutorial.

Object Detection System with ESP32-CAM and Tensorflow


Items needed

ESP32-CAM module

FT232 converter module for program

Object Detection System with ESP32-CAM and Tensorflow


Install ESP32 plugin on Arduino

To get started, the library for this module must be installed in the Arduino IDE software. First, the following link must be copied and pasted to File> Preferences.

https://dl.espressif.com/dl/package_esp32_index.json, http://arduino.esp8266.com/stable/package_esp8266com_index.json


Then you will be able to download the relevant library by going to the Boards section in Tools> Board> Boards Manager and ESP32 Search.



Programming ESP32-cam

To program the ESP-CAM board, we need Arduino-IDE software, and of course, download the relevant board in the software environment, as well as install the required libraries. For information on these cases, you can refer to this tutorial. Make the connections according to the schematic below according to the USB TO TTL used. Note that when programming code, ie after compiling the code, the two pins GPIO 0 and GND are connected to each other, and after successfully compiling the code, you must disconnect this connection to run the project.



Object Detection System with ESP32-CAM and Tensorflow


  • The connections are as follows.




Launching and implementing the project

The first step in using ESP32-CAM in conjunction with Tensorflow.js is to identify the objects that make up the web page where the conclusion occurs. To use the Tensorflow Javascript library, we need to follow these steps: first import the Tensorflow JavaScript libraries, then load the model, in this project the COCO-SSD trained ML model will be used. And create labels for processed objects, which are displayed using the COCO-SSD model on the input video of the identified items by drawing rectangles around the objects.


  • In this part of the action code, we will introduce Tensorflow.js to analyze the received images in the browser. In the next line, we will also import COCO-SSD models along with Tensorflow.js.


  • In the next section, we will load the introduced models so that they can be recognized as a result of processing the received image for our machine.


  • After our machine detects an object in the image, it should now be visible to the user in the form that squares around the detected image are usually used in image processing, the following code is for this.


  • Complete Object Detection system design project code with ESP32-CAM



Final result

After uploading the desired code, I reset the board in the serial ip of the web server will be displayed for us, such as, which we will search for in the browser to enter the web server page, we have to wait for a while until Load the related models, then click on the StartDetect icon to display the images and recognize the defined items. Also, if there is a problem with the image processing or other parts, click on the Restart option.


In the web server, it is possible to count it by specifying the desired item.


By selecting the desired object, the number of detected items will be displayed in front of it.


In this section you adjust the output video settings. For example, it is possible to change the resolution or mirror the image in this section.


In this section, the contrast, quality and brightness of the received image can be adjusted.


If the object is detected, its name will be displayed in this section, and also the percentage of probability that our machine analysis is correct, as well as the coordinates and number of detected objects can be displayed in this section.


Finally, we have the received image, which will inform the user if he detects an object with a square and also enters its name.

Object Detection System with ESP32-CAM and Tensorflow




We described how to implement object detection in ESP32-CAM using Tensorflow.js. This tutorial introduces how to stream video from ESP32-CAM and use the Javascript Tensorflow library to identify and classify objects. Do not forget that the inference process runs inside the browser and is not in ESP32-CAM. And with Tensorflow models already trained as COCO-SSD, we are able to identify objects.



6 thoughts on “Object Detection System with ESP32-CAM and Tensorflow

  1. G.Gheorghe

    The tutorial is successful. I tried it and it works.
    Good luck !

  2. Wonderful post! We are linking to this great post on our
    site. Keep up the great writing.

  3. Lucio

    I will right away take hold of your rss as I can’t to find your email subscription link or
    e-newsletter service. Do you’ve any? Kindly allow
    me recognise so that I could subscribe. Thanks.

  4. ทางเข้า sbo

    Helpful information. Fortunate me I found your web site
    unintentionally, and I am shocked why this coincidence
    did not happened earlier! I bookmarked it.

  5. I really like your blog.. very nice colors & theme.
    Did you create this website yourself or did you hire someone
    to do it for you? Plz respond as I’m looking to construct my own blog
    and would like to find out where u got this from. kudos

Leave a Reply

Your email address will not be published. Required fields are marked *