Breaking Down The Build Pipeline

It looks scary, yes. But fret not, for we are now about to decipher how our build pipeline works!

If you remember from our last post, we got Zig running on the ESP32-P4 with zero C source files in our main folder. But how exactly did we achieve this black magic? The secret lies in a beautiful, albeit slightly chaotic, dance between CMake, Python, and Zig.

Let’s tear down the build pipeline and see what makes it tick!

The Orchestrator: main/CMakeLists.txt

ESP-IDF expects things to be built its way—which usually means a lot of C and C++. To get it to accept our Zig code seamlessly, we have to trick it a little bit. That’s where our main/CMakeLists.txt comes in.

Instead of passing it C source files, we register an empty component and hook into the build process:

# 1. Register the component (No C files here!)
idf_component_register(SRCS "" INCLUDE_DIRS ".")

if(NOT IDF_BUILD_PREPARING)
  set(ZIG_LIB "${CMAKE_CURRENT_SOURCE_DIR}/../zig-out/lib/libzig_app.a")
  set(PATHS_SCRIPT "${CMAKE_CURRENT_SOURCE_DIR}/../generate_paths.py")

  # 2. Chain the Python script and Zig build together!
  add_custom_command(
        OUTPUT "${ZIG_LIB}"
        COMMAND ${PYTHON} "${PATHS_SCRIPT}"
        COMMAND zig build
        WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/.."
        VERBATIM
    )

  add_custom_target(zig_app_target DEPENDS "${ZIG_LIB}")

  target_link_libraries(${COMPONENT_LIB} INTERFACE "${ZIG_LIB}")
  add_dependencies(${COMPONENT_LIB} zig_app_target)

  # Force the linker to look for app_main in the Zig static lib
  target_link_options(${COMPONENT_LIB} INTERFACE "-u" "app_main")
endif()

This file is doing the heavy lifting of coordinating the build. It says: “Hey ESP-IDF, before you finish linking, run this Python script, then run zig build, and finally, take this static library (libzig_app.a) and link it. Oh, and make sure you grab app_main from it!”

The Pathfinder: generate_paths.py

You might be wondering, “Why the Python script?”

Well, ESP-IDF has a massive amount of include paths that change depending on where you installed it (e.g., ~/.espressif, /opt/esp-idf, or some other custom directory). Zig’s C-interop is amazing, but it needs to know where those headers are!

Instead of hardcoding paths and crying when it breaks on another machine, generate_paths.py acts as our scout. It waits for CMake to generate a compile_commands.json file (which contains all the correct, localized include paths), parses it, and hunts down the exact newlib platform headers.

Once it finds everything, it writes it all out into a clean main/paths.zig file:

# A snippet of the magic...
        with open('main/paths.zig', 'w') as f:
            f.write('// Auto-generated by generate_paths.py — do not edit, do not commit\n')
            f.write(f'pub const newlib_include = "{clean(newlib_include)}";\n')
            f.write(f'pub const newlib_platform = "{clean(newlib_platform)}";\n')
            f.write('\n')
            f.write('pub const include_paths = &[_][]const u8{\n')
            for path in sorted(include_paths):
                f.write(f'    "{clean(path)}",\n')
            f.write('};\n')

This script is an absolute lifesaver. It guarantees that our build is environment-agnostic!

The Builder: build.zig

Finally, we arrive at Zig itself. With paths.zig generated, our build.zig script knows exactly what to do.

It configures the target for the riscv32 architecture, specifically tailored for the esp32p4 (complete with single-precision hard float support!). Then, it dynamically pulls in all those include paths we discovered:

const std = @import("std");
const idf_data = @import("main/paths.zig");

pub fn build(b: *std.Build) void {
    const target = b.resolveTargetQuery(.{
        .cpu_arch = .riscv32,
        .os_tag = .linux,
        .abi = .musl,
        .cpu_model = .{ .explicit = &std.Target.riscv.cpu.esp32p4 },
        // ESP32-P4 has an FPU — tell Zig to use single-precision hard float
        .cpu_features_add = std.Target.riscv.featureSet(&.{
            .f, // single-precision float extension (F)
        }),
    });

    const lib = b.addLibrary(.{
        .linkage = .static,
        .name = "zig_app",
        .root_module = b.createModule(.{
            .root_source_file = b.path("main/app.zig"),
            .target = target,
            .optimize = .ReleaseSmall,
            .link_libc = true,
        }),
    });

    // Defining macros that ESP-IDF headers expect
    lib.root_module.addCMacro("ESP_PLATFORM", "1");
    lib.root_module.addCMacro("__IEEE_LITTLE_ENDIAN", "1");
    // ...

    // Feeding the dynamically discovered paths directly to Zig!
    lib.root_module.addIncludePath(.{ .cwd_relative = idf_data.newlib_include });
    lib.root_module.addIncludePath(.{ .cwd_relative = idf_data.newlib_platform });

    for (idf_data.include_paths) |path| {
        lib.root_module.addIncludePath(.{ .cwd_relative = path });
    }

    b.installArtifact(lib);
}

By feeding idf_data.include_paths into addIncludePath, we seamlessly bridge the gap between ESP-IDF’s complex C environment and Zig’s build system. The output is our neatly packaged libzig_app.a, ready for CMake to link.

Putting It All Together

And there you have it!

1. CMake kicks off the build and generates compile_commands.json.
2. CMake runs generate_paths.py.
3. Python reads the commands, finds the includes, and generates paths.zig.
4. CMake runs zig build.
5. Zig uses paths.zig to find the C headers, compiles our code, and creates a static library.
6. CMake links that static library into the final ESP-IDF binary.

It’s a beautiful symphony of tools working together to give us a flawless developer experience. Who knew mixing three different build philosophies could be this satisfying?

Next time, we might just dive into how we’re handling the actual application code!