esp-idf

esp-idf, linux, open-source

While exploring my local network, I stumbled upon several devices named espressif, and after some searching online I found the ESP32 was the likely culprit. I’ve always been interested in hardware and have experience working with electrical circuits from 120-480 volts, but I have never identified a resistor or even considered purchasing a breadboard.

So, I bought a board on Amazon because it was fast shipping and I had a long weekend coming up. I won’t share an Amazon link here for several reasons, but the exact model that I purchased was the ESP32-DevKitC V4 which comes with the ESP32-WROOM chip soldered on.

When the board first arrived, I tried out the Arduino IDE. It was fine for learning purposes, but it felt like the IDE was doing a lot for me without having the chance to really understand everything. Personally I prefer CLion, and I wanted to get back to the IDE that was familiar to me. More importantly, I wanted to work on the ESP32 without tethering myself to any IDE. I wanted the project to be completely stand-alone, and to document and install any dependencies manually. Using Sketches in the Arduino IDE wasn’t going to get me there.

Enter the Espressif IoT Development Framework, or the ESP-IDF for short. This looks slightly intimidating at first, but if you’re at all familiar with C programming you’ll get along fine. Before we dive into the ESP-IDF, lets take a step back and gain our bearings.

Ecosystems

Typically the ESP32 examples recommended for beginners will be using the arduino-esp32 APIs - Espressif provides one such example called WiFiClient. These examples are also made available in the Arduino IDE’s menus where you can create a new project (AKA a new sketch) based on this WiFiClient example. This example uses higher level Arduino APIs compared to what you’ll find in the ESP-IDF examples.

Why would Espressif, creators of the ESP32 and not the Arduino, want to support Arduino APIs? Simply because Arduinos are a popular point of entry for learning programming on microcontrollers, and supporting these APIs lowers the barrier of entry for ESP32 devices.

The ESP-IDF build system supports using those same Arduino APIs via an IDF component configuration marking arduino-esp32 as a dependency. See the Espressif GitHub example here, paired with the IDF component documentation from espressif.

ESP-IDF

In the sections below, we will use the simpler arduino-esp32 APIs and examples to get familiar with deploying to the ESP32 using the ESP-IDF. This example isn’t going to do very much interesting other than validate our build system. If we are successful, it will simply write some output to the serial monitor. No lights, buttons, sensors, wifi, etc.

In a future post, I’ll look at drawing to an LCD display using the I2C communication protocol with the lower-level ESP APIs instead, and drop the dependency for arduino-esp32 entirely.

Installing

First we need to install the ESP-IDF following GitHub instructions on ubuntu 24.04

WARNING: If you are using arduino-esp32 APIs mentioned above, pay attention to arduino-esp32 releases for the latest supported ESP-IDF version. The latest ESP-IDF version is likely not currently supported by arduino-esp32. At the time of this writing, the latest arduino-esp32 release v3.1.1 is based on ESP-IDF 5.3.2, so below we checkout the v5.3.2 branch of github.com/espressif/esp-idf.git.

Install the prerequisites required for the ESP-IDF

1
sudo apt-get install -y git wget flex bison gperf python3 python3-pip python3-venv cmake ninja-build ccache libffi-dev libssl-dev dfu-util libusb-1.0-0

Clone the ESP-IDF using the v5.3.2 branch and install. There will be a lot of output produced from these commands that is not shown below.

1
2
3
4
git clone -b v5.3.2 git@github.com:espressif/esp-idf.git
cd esp-idf
./install.sh
. ./export.sh

Switching Versions

To switch versions after previously installing the ESP-IDF

1
2
3
4
5
cd /path/to/esp-idf
git checkout v5.3.2
git submodule update --init --recursive
./install.sh
. ./export.sh

If everything worked correctly, you should end up with this final output in your terminal - 

1
2
3
4
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:

  idf.py build

Project Example

First set up your environment to use ESP-IDF tools and to have the correct exports required for building with CMake. Unfortunately the way this is done is by sourcing a script in the ESP-IDF repository, after you’ve installed the ESP-IDf.

1
source /path/to/esp-idf/export.sh

You should now have the idf.py tool available in your shell.

1
2
3
4
5
6
idf.py

Usage: idf.py [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

  ESP-IDF CLI build management tool. For commands that are not known to idf.py an attempt to execute
  it as a build system target will be made. Selected target: None

To make initializing my environment for using the ESP-IDF simpler, I threw an alias in my .bash_aliases file.

1
echo "alias init-idf='source $HOME/Code/Clones/esp-idf/export.sh'" >> ~/.bash_aliases

So next time when I want to work on ESP things I can just run init-idf in a terminal to initialize my environment. From here we can use CMake to create an ESP-IDF project, and even flash to our device or open a serial monitor.

CMake

Setting up the project in CMake is fairly simple. For more information about build system see ESP-IDF - Build System

1
2
3
mkdir esp-example
cd esp-example
vim CMakeLists.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
cmake_minimum_required(VERSION 3.26)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)

project(
    #[[NAME]]   esp-idf-arduino
    VERSION     0.1
    DESCRIPTION "Example ESP-IDF cmake project"
    LANGUAGES   CXX
)

# For writing pure cmake components, see the documentation
# https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/build-system.html#writing-pure-cmake-components
idf_build_set_property(COMPILE_OPTIONS "-Wno-error" APPEND)

And then create our main/ directory where we’ll store our source code and write a quick CMakeLists.txt

1
2
mkdir main
vim main/CMakeLists.txt
1
2
3
4
idf_component_register(
    SRCS "main.cpp"
    INCLUDE_DIRS "."
)

The project structure now looks like the following file tree. We’ll configure and build with cmake later, once we set up our dependencies in the ESP-IDF.

1
2
3
4
esp-example/
├── CMakeLists.txt
└── main
    └── CMakeLists.txt

Source Code

Now we write the source code for our application. I hope you’re prepared, because it’s a lot.

1
2
cd esp-example
vim main/main.cpp
1
2
3
4
5
6
7
8
9
10
#include "Arduino.h"

void setup() {
  Serial.begin(115200);
}

void loop() {
  Serial.println("Hello world!");
  delay(1000);
}

This code simply attaches to the serial monitor at 115200 baud and prints Hello world! every second. That’s it, the project is completed - we have our cmake lists set up and our source code in place. Next we will configure the project with cmake using the idf.py tool.

Dependencies

Now that we have the tools to configure and build ESP-IDF projects with cmake, we can look at creating our first project. The example code that I’m going to build here is available at git.shaunreed.com/shaunrd0/klips. The commands below will not clone this repository, but instead set up the project as I did when creating it.

First set the build target to the ESP32 device. This tells the ESP-IDF which microcontroller we are working with. 

1
idf.py set-target esp32

Notice that the idf.py command created the sdkconfig file in our current directory, and produced a build/ directory -

1
2
3
4
5
6
7
esp-example/
├── build/
├── CMakeLists.txt
├── main
│   ├── CMakeLists.txt
│   └── main.cpp
└── sdkconfig

Next we need to add a dependency for the arduino-esp32 APIs. These simpler APIs will be a nice way to test our build system and get familiar with the tools.

IMPORTANT: We set the 3.1.1 version because the release is compatible with ESP-IDF v5.3.2 we installed in the first step.

1
2
3
4
5
6
7
# https://github.com/espressif/arduino-esp32/releases/tag/3.1.1
idf.py add-dependency "espressif/arduino-esp32^3.1.1"

Executing action: add-dependency
NOTICE: Created "/home/shaun/Code/esp-example/main/idf_component.yml"
NOTICE: Successfully added dependency "espressif/arduino-esp32": "^3.1.1" to component "main"
NOTICE: If you want to make additional changes to the manifest file at path /home/shaun/Code/esp-example/main/idf_component.yml manually, please refer to the documentation: https://docs.espressif.com/projects/idf-component-manager/en/latest/reference/manifest_file.html

After the idf.py commands above, we have a new idf_component.yml file -

1
2
3
4
5
6
7
8
esp-example/
├── build/
├── CMakeLists.txt
├── main
│   ├── CMakeLists.txt
│   ├── idf_component.yml
│   └── main.cpp
└── sdkconfig

The contents should look like this, note the arduino-esp32 dependency is listed on the last line

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
## IDF Component Manager Manifest File
dependencies:
  ## Required IDF version
  idf:
    version: '>=4.1.0'
  # # Put list of dependencies here
  # # For components maintained by Espressif:
  # component: "~1.0.0"
  # # For 3rd party components:
  # username/component: ">=1.0.0,<2.0.0"
  # username2/component2:
  #   version: "~1.0.0"
  #   # For transient dependencies `public` flag can be set.
  #   # `public` flag doesn't have an effect dependencies of the `main` component.
  #   # All dependencies of `main` are public by default.
  #   public: true
  espressif/arduino-esp32: ^3.1.1

Now we’re ready to configure the project and prepare to flash to our device. These commands will automatically pull in the arduino-esp32 dependency into our project directory so it may take some time to complete.

1
cmake -B build

If you’re lucky after some output from cmake you’ll get the following error

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
-- Project sdkconfig file /home/shaun/Code/test-esp/sdkconfig
-- Compiler supported targets: xtensa-esp-elf
-- App "esp-idf-arduino" version: 0.1
-- Adding linker script /home/shaun/Code/test-esp/build/esp-idf/esp_system/ld/memory.ld
-- Adding linker script /home/shaun/Code/test-esp/build/esp-idf/esp_system/ld/sections.ld.in
-- Adding linker script /home/shaun/Code/Clones/esp-idf/components/esp_rom/esp32/ld/esp32.rom.ld
-- Adding linker script /home/shaun/Code/Clones/esp-idf/components/esp_rom/esp32/ld/esp32.rom.api.ld
-- Adding linker script /home/shaun/Code/Clones/esp-idf/components/esp_rom/esp32/ld/esp32.rom.libgcc.ld
-- Adding linker script /home/shaun/Code/Clones/esp-idf/components/esp_rom/esp32/ld/esp32.rom.newlib-data.ld
-- Adding linker script /home/shaun/Code/Clones/esp-idf/components/esp_rom/esp32/ld/esp32.rom.syscalls.ld
-- Adding linker script /home/shaun/Code/Clones/esp-idf/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld
-- Adding linker script /home/shaun/Code/Clones/esp-idf/components/soc/esp32/ld/esp32.peripherals.ld
-- DEBUG: Use esp-modbus component folder: /home/shaun/Code/test-esp/managed_components/espressif__esp-modbus.
-- git rev-parse returned 'fatal: not a git repository (or any of the parent directories): .git'
ESP Insights Project commit: HEAD-HASH-NOTFOUND
-- git rev-parse returned 'fatal: not a git repository (or any of the parent directories): .git'
ESP RainMaker Project commit: HEAD-HASH-NOTFOUND

CMake Error at managed_components/espressif__arduino-esp32/CMakeLists.txt:371 (message):
  esp32-arduino requires CONFIG_FREERTOS_HZ=1000 (currently 100)

Run the following sed command to fix it. This just modifies a value that we could have probably tracked down in the idf.py menuconfig interface, but that doesn’t sound like a good time.

1
sed -i -e 's/CONFIG_FREERTOS_HZ=100/CONFIG_FREERTOS_HZ=1000/' sdkconfig

Now rerun the cmake command to configure the project and it should complete normally. 

1
cmake -B build

Finally, we can build the project

1
cmake --build build

We’ll hit this error, which was confusing at first for me because I wasn’t familiar with the differences between app_main() and loop() / setup()

1
2
3
4
5
/home/shaun/.espressif/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/freertos/libfreertos.a(app_startup.c.obj):(.literal.main_task+0x24): undefined reference to `app_main'
/home/shaun/.espressif/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/freertos/libfreertos.a(app_startup.c.obj): in function `main_task':
/home/shaun/Code/Clones/esp-idf/components/freertos/app_startup.c:199:(.text.main_task+0x99): undefined reference to `app_main'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

The build failed because we are using the ESP-IDF which expects the app_main() convention. We can use the Arudino style by setting some configurations in the ESP-IDF.

We need to automatically start Arduino under the ESP-IDF so that we can make use of the Arduino loop() and setup() functions in our example. You can also use the ESP-IDF app_main() function if preferred, see the examples below for differences between the two.

ESP-IDF example using app_main()
ESP-IDF example using Arduino loop() and setup()

You can alternatively do this in the TUI tool idf.py menuconfig, but here is a bash command to append the same configuration to the sdkconfig in your project directory that was generated by the ESP-IDF. 

1
echo "CONFIG_AUTOSTART_ARDUINO=y" >> sdkconfig

We can once again build the project, only this this time it won’t fail to build.

1
cmake --build build

If everything completed normally, the build output will end with 

1
2
3
4
5
6
7
8
Project build complete. To flash, run:
 idf.py flash
or
 idf.py -p PORT flash
or
 python -m esptool --chip esp32 -b 460800 --before default_reset --after hard_reset write_flash --flash_mode dio --flash_size 2MB --flash_freq 40m 0x1000 build/bootloader/bootloader.bin 0x8000 build/partition_table/partition-table.bin 0x10000 build/main.bin
or from the "/home/shaun/Code/klips/esp/cpp/04_esp-idf-arduino/build" directory
 python -m esptool --chip esp32 -b 460800 --before default_reset --after hard_reset write_flash "@flash_args"

In the next and final section, we’ll look at flashing to the ESP32 and validating the program is behaving correctly using the serial monitor.

Flashing

Find the device for the ESP32. Mine is /dev/ttyUSB0 in the output below.

1
2
3
4
5
6
7
8
9
10
$ ls /dev/tty*
/dev/tty    /dev/tty16  /dev/tty24  /dev/tty32  /dev/tty40  /dev/tty49  /dev/tty57  /dev/tty8       /dev/ttyS14  /dev/ttyS22  /dev/ttyS30
/dev/tty0   /dev/tty17  /dev/tty25  /dev/tty33  /dev/tty41  /dev/tty5   /dev/tty58  /dev/tty9       /dev/ttyS15  /dev/ttyS23  /dev/ttyS31
/dev/tty1   /dev/tty18  /dev/tty26  /dev/tty34  /dev/tty42  /dev/tty50  /dev/tty59  /dev/ttyprintk  /dev/ttyS16  /dev/ttyS24  /dev/ttyS4
/dev/tty10  /dev/tty19  /dev/tty27  /dev/tty35  /dev/tty43  /dev/tty51  /dev/tty6   /dev/ttyS0      /dev/ttyS17  /dev/ttyS25  /dev/ttyS5
/dev/tty11  /dev/tty2   /dev/tty28  /dev/tty36  /dev/tty44  /dev/tty52  /dev/tty60  /dev/ttyS1      /dev/ttyS18  /dev/ttyS26  /dev/ttyS6
/dev/tty12  /dev/tty20  /dev/tty29  /dev/tty37  /dev/tty45  /dev/tty53  /dev/tty61  /dev/ttyS10     /dev/ttyS19  /dev/ttyS27  /dev/ttyS7
/dev/tty13  /dev/tty21  /dev/tty3   /dev/tty38  /dev/tty46  /dev/tty54  /dev/tty62  /dev/ttyS11     /dev/ttyS2   /dev/ttyS28  /dev/ttyS8
/dev/tty14  /dev/tty22  /dev/tty30  /dev/tty39  /dev/tty47  /dev/tty55  /dev/tty63  /dev/ttyS12     /dev/ttyS20  /dev/ttyS29  /dev/ttyS9
/dev/tty15  /dev/tty23  /dev/tty31  /dev/tty4   /dev/tty48  /dev/tty56  /dev/tty7   /dev/ttyS13     /dev/ttyS21  /dev/ttyS3   /dev/ttyUSB0

To flash run the following commands.

1
2
3
4
5
6
# Manually selecting port with above output
# Baud rate matters: https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/establish-serial-connection.html#connect-esp32-to-pc
idf.py -p /dev/ttyUSB0 -b 115200 flash

# Let esp-idf automatically detect port and baud rate
idf.py flash

To open serial monitor via the commandline -

1
idf.py monitor -b 115200

Or in CLion the Serial Monitor extension works well.

If everything is working correctly, you should see Hello world! printed to the serial monitor repeatedly.

See the full source code for this project on gitea