feat: allow providing source assets from container images by utkuozdemir · Pull Request #686 · siderolabs/kres

utkuozdemir · 2026-07-29T14:56:37Z

Projects can now declare source assets: files pulled from pinned container images into the source tree of the build, without being committed to the repository, e.g. binaries referenced by go:embed directives.

Example:

kind: common.SourceAssets
spec:
  images:
    - ref: ghcr.io/siderolabs/ipxe:v1.11.0
      copies:
        - platform: linux/amd64
          source: /usr/libexec/
          destination: internal/ipxe/data/amd64

The declaration generates everything derived from it: the image stages and a collector stage in the Dockerfile, the copy into the source tree of the base stage so lint, tests, and builds all see the files, the git and docker ignore rules for the destinations, a fetch-source-assets make target which materializes the same collector stage locally for native builds outside docker, and a renovate manager so the image references receive update PRs.

Invalid configurations fail generation: an empty image reference, a relative source path, a destination escaping the source tree, and overlapping destinations.

A set of improvements that remove the provider's dependency on its container image and fix UEFI HTTP boot. Together they also make the emulated development flow work natively on macOS, where QEMU uses the vmnet network. The iPXE boot binaries are now embedded into the provider binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The compressor, its libraries, and the iPXE binaries are all dropped from the image. The HTTP boot file names handed out by the DHCP proxy now also resolve. Previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The qemu-up tool gains an option to have the provisioner's own DHCP server hand out the PXE boot server and boot file directly, instead of relying on the ProxyDHCP responses. Some UEFI PXE stacks, such as the firmware QEMU uses on macOS, do not accept ProxyDHCP offers, so without this the emulated machines never boot. The boot file is selected for the host architecture and served from the provider's TFTP server on the gateway address. The build tooling is regenerated with the kres feature that injects artifacts from docker stages into the build source tree. Part of siderolabs/kres#686. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

The iPXE boot binaries are now embedded into the booter binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP, so booter no longer needs a writable filesystem. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The HTTP boot file names handed out by the DHCP proxy now also resolve, previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The container image now contains nothing beyond the booter binary itself, and running booter as a plain binary outside the container is fully supported. The iPXE binaries that had been tracked in the repository since the initial commit are removed and ignored. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Dependency versions flagged by the vulnerability audit are bumped as well. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

A set of improvements that remove the provider's dependency on its container image and fix UEFI HTTP boot. Together they also make the emulated development flow work natively on macOS, where QEMU uses the vmnet network. The iPXE boot binaries are now embedded into the provider binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The compressor, its libraries, and the iPXE binaries are all dropped from the image. The HTTP boot file names handed out by the DHCP proxy now also resolve. Previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The qemu-up tool gains an option to have the provisioner's own DHCP server hand out the PXE boot server and boot file directly, instead of relying on the ProxyDHCP responses. Some UEFI PXE stacks, such as the firmware QEMU uses on macOS, do not accept ProxyDHCP offers, so without this the emulated machines never boot. The boot file is selected for the host architecture and served from the provider's TFTP server on the gateway address. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

The iPXE boot binaries are now embedded into the booter binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP, so booter no longer needs a writable filesystem. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The HTTP boot file names handed out by the DHCP proxy now also resolve, previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The container image now contains nothing beyond the booter binary itself, and running booter as a plain binary outside the container is fully supported. The iPXE binaries that had been tracked in the repository since the initial commit are removed and ignored. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Dependency versions flagged by the vulnerability audit are bumped as well. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

A set of improvements that remove the provider's dependency on its container image and fix UEFI HTTP boot. Together they also make the emulated development flow work natively on macOS, where QEMU uses the vmnet network. The iPXE boot binaries are now embedded into the provider binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The compressor, its libraries, and the iPXE binaries are all dropped from the image. The HTTP boot file names handed out by the DHCP proxy now also resolve. Previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The qemu-up tool gains an option to have the provisioner's own DHCP server hand out the PXE boot server and boot file directly, instead of relying on the ProxyDHCP responses. Some UEFI PXE stacks, such as the firmware QEMU uses on macOS, do not accept ProxyDHCP offers, so without this the emulated machines never boot. The boot file is selected for the host architecture and served from the provider's TFTP server on the gateway address. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

Projects can now declare source assets: files pulled from pinned container images into the source tree of the build, without being committed to the repository, e.g. binaries referenced by go:embed directives. Example: ```yaml kind: common.SourceAssets spec: images: - ref: ghcr.io/siderolabs/ipxe:v1.11.0 copies: - platform: linux/amd64 source: /usr/libexec/ destination: internal/ipxe/data/amd64 ``` The declaration generates everything derived from it: the image stages and a collector stage in the Dockerfile, the copy into the source tree of the base stage so lint, tests, and builds all see the files, the git and docker ignore rules for the destinations, a fetch-source-assets make target which materializes the same collector stage locally for native builds outside docker, and a renovate manager so the image references receive update PRs. Invalid configurations fail generation: an empty image reference, a relative source path, a destination escaping the source tree, and overlapping destinations. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

A set of improvements that remove the provider's dependency on its container image and fix UEFI HTTP boot. Together they also make the emulated development flow work natively on macOS, where QEMU uses the vmnet network. The iPXE boot binaries are now embedded into the provider binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The compressor, its libraries, and the iPXE binaries are all dropped from the image. The HTTP boot file names handed out by the DHCP proxy now also resolve. Previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The qemu-up tool gains an option to have the provisioner's own DHCP server hand out the PXE boot server and boot file directly, instead of relying on the ProxyDHCP responses. Some UEFI PXE stacks, such as the firmware QEMU uses on macOS, do not accept ProxyDHCP offers, so without this the emulated machines never boot. The boot file is selected for the host architecture and served from the provider's TFTP server on the gateway address. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

utkuozdemir · 2026-07-30T09:38:35Z

/m

The iPXE boot binaries are now embedded into the booter binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP, so booter no longer needs a writable filesystem. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The HTTP boot file names handed out by the DHCP proxy now also resolve, previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The container image now contains nothing beyond the booter binary itself, and running booter as a plain binary outside the container is fully supported. The iPXE binaries that had been tracked in the repository since the initial commit are removed and ignored. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Dependency versions flagged by the vulnerability audit are bumped as well. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

A set of improvements that remove the provider's dependency on its container image and fix UEFI HTTP boot. Together they also make the emulated development flow work natively on macOS, where QEMU uses the vmnet network. The iPXE boot binaries are now embedded into the provider binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The compressor, its libraries, and the iPXE binaries are all dropped from the image. The HTTP boot file names handed out by the DHCP proxy now also resolve. Previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The qemu-up tool gains an option to have the provisioner's own DHCP server hand out the PXE boot server and boot file directly, instead of relying on the ProxyDHCP responses. Some UEFI PXE stacks, such as the firmware QEMU uses on macOS, do not accept ProxyDHCP offers, so without this the emulated machines never boot. The boot file is selected for the host architecture and served from the provider's TFTP server on the gateway address. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

A set of improvements that remove the provider's dependency on its container image and fix UEFI HTTP boot. Together they also make the emulated development flow work natively on macOS, where QEMU uses the vmnet network. The iPXE boot binaries are now embedded into the provider binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The compressor, its libraries, and the iPXE binaries are all dropped from the image. The HTTP boot file names handed out by the DHCP proxy now also resolve. Previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The directory holding the local boot assets is now configurable via a flag instead of being a fixed path that only exists inside the container image. When local boot assets are enabled, the provider verifies at startup that the directory contains every file it can serve and fails listing the missing ones, instead of failing only when a machine attempts to boot. The qemu-up tool gains an option to have the provisioner's own DHCP server hand out the PXE boot server and boot file directly, instead of relying on the ProxyDHCP responses. Some UEFI PXE stacks, such as the firmware QEMU uses on macOS, do not accept ProxyDHCP offers, so without this the emulated machines never boot. The boot file is selected for the host architecture and served from the provider's TFTP server on the gateway address. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

The iPXE boot binaries are now embedded into the booter binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP, so booter no longer needs a writable filesystem. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The HTTP boot file names handed out by the DHCP proxy now also resolve, previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The container image now contains nothing beyond the booter binary itself, and running booter as a plain binary outside the container is fully supported. The iPXE binaries that had been tracked in the repository since the initial commit are removed and ignored. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Dependency versions flagged by the vulnerability audit are bumped as well. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

A set of improvements that remove the provider's dependency on its container image and fix UEFI HTTP boot. Together they also make the emulated development flow work natively on macOS, where QEMU uses the vmnet network. The iPXE boot binaries are now embedded into the provider binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The compressor, its libraries, and the iPXE binaries are all dropped from the image. The HTTP boot file names handed out by the DHCP proxy now also resolve. Previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The directory holding the local boot assets is now configurable via a flag instead of being a fixed path that only exists inside the container image. When local boot assets are enabled, the provider verifies the directory at startup, requiring a complete and usable set of assets for at least one architecture, instead of failing only when a machine attempts to boot. The qemu-up tool gains an option to have the provisioner's own DHCP server hand out the PXE boot server and boot file directly, instead of relying on the ProxyDHCP responses. Some UEFI PXE stacks, such as the firmware QEMU uses on macOS, do not accept ProxyDHCP offers, so without this the emulated machines never boot. The boot file is selected by machine firmware and architecture and served from the provider's TFTP server on the gateway address. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

The iPXE boot binaries are now embedded into the booter binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP, so booter no longer needs a writable filesystem. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The HTTP boot file names handed out by the DHCP proxy now also resolve, previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The container image now contains nothing beyond the booter binary itself, and running booter as a plain binary outside the container is fully supported. The iPXE binaries that had been tracked in the repository since the initial commit are removed and ignored. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Dependency versions flagged by the vulnerability audit are bumped as well. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

A set of improvements that remove the provider's dependency on its container image and fix UEFI HTTP boot. Together they also make the emulated development flow work natively on macOS, where QEMU uses the vmnet network. The iPXE boot binaries are now embedded into the provider binary at build time instead of being read from a directory. The container build injects them from the pinned iPXE image, and a native build populates them with a make target first. The embedded files are listed explicitly, so a binary can never be built without them. Patching now happens entirely in memory and the patched binaries are served from memory over both TFTP and HTTP. The BIOS binaries are compressed in-process using the go-zbin library instead of executing the compressor tool bundled in the container image. The compressor, its libraries, and the iPXE binaries are all dropped from the image. The HTTP boot file names handed out by the DHCP proxy now also resolve. Previously they pointed at paths nothing created, so UEFI HTTP boot only worked with manual filesystem setup. The directory holding the local boot assets is now configurable via a flag instead of being a fixed path that only exists inside the container image. When local boot assets are enabled, the provider verifies the directory at startup, requiring a complete and usable set of assets for at least one architecture, instead of failing only when a machine attempts to boot. The qemu-up tool gains an option to have the provisioner's own DHCP server hand out the PXE boot server and boot file directly, instead of relying on the ProxyDHCP responses. Some UEFI PXE stacks, such as the firmware QEMU uses on macOS, do not accept ProxyDHCP offers, so without this the emulated machines never boot. The boot file is selected by machine firmware and architecture and served from the provider's TFTP server on the gateway address. The build tooling is regenerated with the kres feature that provides source assets from container images, so the whole asset pipeline is derived from a single declaration. Part of siderolabs/kres#686. Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>

talos-bot added the status/ok-to-test label Jul 29, 2026

talos-bot added this to Planning Jul 29, 2026

github-project-automation Bot moved this to To Do in Planning Jul 29, 2026

talos-bot moved this from To Do to In Review in Planning Jul 29, 2026

utkuozdemir mentioned this pull request Jul 29, 2026

feat: support running the provider and emulated machines natively siderolabs/omni-infra-provider-bare-metal#124

Merged

utkuozdemir force-pushed the custom-step-build-artifacts branch from b481f61 to 2c8e0c4 Compare July 29, 2026 15:44

utkuozdemir changed the title ~~feat: support injecting docker stage artifacts into the source tree~~ feat: allow embedding docker stage artifacts into Go binaries Jul 29, 2026

utkuozdemir force-pushed the custom-step-build-artifacts branch from 2c8e0c4 to a3b95b2 Compare July 29, 2026 16:23

utkuozdemir changed the title ~~feat: allow embedding docker stage artifacts into Go binaries~~ feat: allow providing source assets from container images Jul 29, 2026

utkuozdemir marked this pull request as draft July 29, 2026 16:23

utkuozdemir mentioned this pull request Jul 29, 2026

feat: use embedded iPXE binaries and serve them from memory siderolabs/booter#16

Merged

utkuozdemir force-pushed the custom-step-build-artifacts branch from a3b95b2 to de56c80 Compare July 29, 2026 16:28

utkuozdemir marked this pull request as ready for review July 29, 2026 16:31

utkuozdemir self-assigned this Jul 29, 2026

utkuozdemir force-pushed the custom-step-build-artifacts branch from de56c80 to 6a3c8a4 Compare July 29, 2026 16:44

utkuozdemir force-pushed the custom-step-build-artifacts branch from 6a3c8a4 to 34f68df Compare July 29, 2026 19:01

oguzkilcan approved these changes Jul 30, 2026

View reviewed changes

github-project-automation Bot moved this from In Review to Approved in Planning Jul 30, 2026

talos-bot added the status/ok-to-merge label Jul 30, 2026

talos-bot merged commit 34f68df into siderolabs:main Jul 30, 2026
15 checks passed

github-project-automation Bot moved this from Approved to Done in Planning Jul 30, 2026

talos-bot removed the status/ok-to-test label Jul 30, 2026

talos-bot removed the status/ok-to-merge label Jul 30, 2026

utkuozdemir deleted the custom-step-build-artifacts branch July 30, 2026 09:38

Navigation Menu

Uh oh!

feat: allow providing source assets from container images - #686

feat: allow providing source assets from container images#686
talos-bot merged 1 commit into
siderolabs:mainfrom
utkuozdemir:custom-step-build-artifacts

utkuozdemir commented Jul 29, 2026 •

edited

Loading

Uh oh!

utkuozdemir commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

utkuozdemir commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

utkuozdemir commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

utkuozdemir commented Jul 29, 2026 •

edited

Loading