jam 2.17.1 breaks buildpack building

[sgaist]

Expected Behavior

Successful buildpack build

Current Behavior

All builds are failing with errors such as:

failed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_dotnet-execute/port-chooser/exec.d/0-port-chooser': fork/exec /layers/paketo-buildpacks_dotnet-execute/port-chooser/exec.d/0-port-chooser: permission denied

Possible Solution

Sorry, currently I don't know what has broken the behaviour

Steps to Reproduce

1. Clone any of the existing buildpacks
2. Switch to the github-config update branch
3. Launch integration tests

Motivations

It breaks the github-config updates which make things complicated to propagate the go-diskfs exclusion to allow for the dependencies update to happen.

[dmikusa]

In nginx, I'm seeing:

            [builder]   Executing build process
            [builder]     Installing Nginx Server 1.29.8
            [builder] failed to fetch dependency: failed to open file: open /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/linux/amd64/dependencies/0a716708a57462b89110f49097b4b5d3f457fc8e2704c1a97846eaf8ff7d2a62: no such file or directory

Oh, I see what's going on. This is the contents of the offline buildpack. Note how the path to dependencies is /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies, but when run it's looking for them under /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/linux/amd64/dependencies. There's an extra linux/amd64 in there.

  ┌─ Layer [1/1]
  │  Digest    : sha256:21c64749a12be52d6f2a8f0951f6da88366cd6f018af3282a20aef1a42097ed1
  │  Media type: application/vnd.docker.image.rootfs.diff.tar.gzip
  │  Size      : 47174992 bytes (44.98 MB)
  └─ Contents:
     drwxr-xr-x  0 0      0           0 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx
     drwxr-xr-x  0 0      0           0 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3
     drwxr-xr-x  0 0      0           0 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/bin
     lrwxr-xr-x  0 0      0           0 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/bin/build -> run
     -rwxr-xr-x  0 0      0     3342498 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/bin/configure
     lrwxr-xr-x  0 0      0           0 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/bin/detect -> run
     -rwxr-xr-x  0 0      0    61735074 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/bin/run
     -rw-r--r--  0 0      0       15010 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/buildpack.toml
     drwxr-xr-x  0 0      0           0 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies
     -rw-r--r--  0 0      0     3384769 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies/14f908022fba964bc1809a6697de5bb8ef0e755d7f1ab3c23e917fe3f9ee6009
     -rw-r--r--  0 0      0     3209195 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies/2af64c15c383c98720546ab470d9109f9f81f631d3c5fb88e0c5a2363aca83e8
     -rw-r--r--  0 0      0     3323331 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies/33e75734171c6d0f0a85ff2288e97f60041a7608b605d286bac0b257e23564c9
     -rw-r--r--  0 0      0     3208234 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies/ab639801e971cfd4f1c62184c7ff98e5b81f42756c49ca797dfa4005753cf422
     -rw-r--r--  0 0      0     3327506 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies/ad046b7efe9031c6ef384f90af494f319f4bdafc1c007b68698fcd0580b8a0b8
     -rw-r--r--  0 0      0     3384549 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies/b1c5edc019821d6b5b603b83862f4eb05dab89969bf96974033eaba5bcf81f8f
     -rw-r--r--  0 0      0     3262420 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies/c57e0b46f4cc452b889d289da90de44ebbfabf7978a5a3aac1c7618f5e904100
     -rw-r--r--  0 0      0     3264560 Dec 31  1979 /cnb/buildpacks/paketo-buildpacks_nginx/1.2.3/dependencies/d20d89d81e29cd16ee6575c28c39220137b5e8b007a6492e8b92a2b7464728c9

the buildpack.toml in the offline buildpack has the wrong path:

  [[metadata.dependencies]]
    arch = "amd64"
    checksum = "sha256:c49aed784ba1876f70602e4df87ecdc982e823ee63a931fe2ecbdc79727cfde5"
    cpe = "cpe:2.3:a:nginx:nginx:1.29.7:*:*:*:*:*:*:*"
    deprecation_date = "2026-04-30T00:00:00Z"
    id = "nginx"
    licenses = ["BSD-2-Clause", "BSD-2-Clause-NetBSD"]
    name = "Nginx Server"
    os = "linux"
    purl = "pkg:generic/nginx@1.29.7?checksum=673f8fb8c0961c44fbd9410d6161831453609b44063d3f2948253fc2b5692139&download_url=https://nginx.org/download/nginx-1.29.7.tar.gz"
    source = "https://nginx.org/download/nginx-1.29.7.tar.gz"
    source-checksum = "sha256:673f8fb8c0961c44fbd9410d6161831453609b44063d3f2948253fc2b5692139"
    stacks = ["io.buildpacks.stacks.jammy"]
    uri = "file:///linux/amd64/dependencies/c49aed784ba1876f70602e4df87ecdc982e823ee63a931fe2ecbdc79727cfde5"
    version = "1.29.7"

That's where the extra linux/amd64 is being introduced.

[sgaist]

It seems that the issue started with 2.17.0, 2.16.0 is working fine.

[sgaist]

@dmikusa The patch looks good for the offline related error but there's also the permission denied error which seems to be unrelated to that.

As an example: https://github.com/paketo-buildpacks/pip/actions/runs/24283740406/job/70909597665?pr=964

[dmikusa]

Oh, interesting. I'm not sure why it would be changing the permissions.

Here's what I see in the buildpack with an older jam:

> ls -lth cnb/buildpacks/paketo-buildpacks_cpython/1.18.21/bin/
total 125968
lrwxr-xr-x  1 dmikusa  staff     3B Dec 31  1979 build@ -> run
lrwxr-xr-x  1 dmikusa  staff     3B Dec 31  1979 detect@ -> run
-rwxr-xr-x  1 dmikusa  staff   1.8M Dec 31  1979 env*
-rwxr-xr-x  1 dmikusa  staff    60M Dec 31  1979 run*

which is owner executable, but not group executable.

In the latest jam...

cnb/buildpacks/paketo-buildpacks_cpython/1.18.21/bin:
total 370464
-rwxr-xr-x  1 dmikusa  staff    60M Dec 31  1979 build*
-rwxr-xr-x  1 dmikusa  staff    60M Dec 31  1979 detect*
-rw-r--r--  1 dmikusa  staff   1.8M Dec 31  1979 env
-rw-r--r--  1 dmikusa  staff    60M Dec 31  1979 run

so it definitely lost that executable bit on env and run.

[pacostas]

After landing this PR, the issue on pip should be resolved #548

[pacostas]

#548

[till]

For me builder building has been broken now — Version 2.17.3 is still broken:

Error: could not find any valid tag for registry.example.org/repo/runimage
Usage:
  jam update-builder [flags]

Flags:
      --builder-file string    path to the builder.toml file (required)
  -h, --help                   help for update-builder
      --lifecycle-uri string   URI for lifecycle image (optional: default=index.docker.io/buildpacksio/lifecycle) (default "index.docker.io/buildpacksio/lifecycle")

It cannot pull images like this:

[run]
  [[run.images]]
    image = "registry.example.org/repo/runimage:noble-full"

I think the culprit is:

jam/internal/image.go

Lines 142 to 154 in 08e47f6

	for _, tag := range tags {
	version, err := semver.StrictNewVersion(tag)
	if err != nil {
	continue
	}
	if version.Prerelease() != "" {
	continue
	}
	versions = append(versions, version)
	}
	if len(versions) == 0 {
	return Image{}, fmt.Errorf("could not find any valid tag for %s", repo.Name())

Currently testing if this works — the "legacy" format:

[stack]
  build-image = "docker.io/paketobuildpacks/build-jammy-full:0.1.147"
  id = "io.buildpacks.stacks.jammy"
  run-image = "registry.example.org/repo/runimage:jammy-full"

[till]

@pacostas @dmikusa — I don't know if what I found is "new" or has been there a bit. Should I move it to another ticket? I just started using noble a few days ago.

[pacostas]

Im looking at it

[pacostas]

@till The changes are not related to the recent pr with packaging. Although, can you please confirm the following ones?

1. do you have a version of jam that this works?
2. Is this a valid repo? registry.example.org/repo/runimage:noble-full
3. Why do you specify both stacks and images on your builder.toml
4. Why do you use update-builder if you want to build the builder and not what we use on the builder workflows? https://github.com/paketo-buildpacks/builder-jammy-full/blob/126792fab0cd6f6e6998b634f5ef617bfd0c9298/scripts/publish.sh#L96
5. Do you have a minimal steps for reproducing it? e.g. the builder.toml and the jam command?

[till]

I phrased it wrong, it's not about building, it's dependency updates.

It started day before yesterday, but it's probably not related to this but another track. I'll make a new issue.