Closes #386: Add ephemeral port support to generated apps by nrnvgh · Pull Request #399 · nautobot/cookiecutter-nautobot-app

nrnvgh · 2026-06-17T23:33:32Z

Closes #386

What's Changed

Added ephemeral port support to apps built using this cookiecutter repo. The basic logic is a replication of ephemeral support in the core as done in nautobot/nautobot#6896, nautobot/nautobot#7717, and possibly others I failed to find.

Summary

Add an opt-in ephemeral_ports Invoke setting for generated app development stacks, matching Nautobot core's default-compose guard behavior.
Add ephemeral Docker Compose override files for generated apps, including an explicit Mattermost override for ChatOps custom compose usage.
Document enablement via invoke.yml and environment variables, ignore .service_ports.json, and update sample Invoke configs for custom compose lists.

Validation

Rebaked all four generated app variants with cookiecutter --overwrite-if-exists before running the matrix below.

Static Ports Disabled

All four generated apps started successfully with ephemeral_ports: false, reported fixed host ports in invoke ps, wrote matching .service_ports.json, and stopped cleanly with invoke stop.

Generated app	`docs`	`nautobot`
Base	`8001 -> 8080`	`8080 -> 8080`
Commercial	`8001 -> 8080`	`8080 -> 8080`
ChatOps	`8001 -> 8080`	`8080 -> 8080`
SSoT	`8001 -> 8080`	`8080 -> 8080`

Raw output

final-static-full-stack.txt

Ephemeral Ports Enabled

Each generated app was validated twice: once with ephemeral_ports: true in invoke.yml, and once with the corresponding INVOKE_*_EPHEMERAL_PORTS=1 environment variable. In all cases, invoke ps showed dynamic host ports for docs and nautobot, .service_ports.json matched those mappings, and cleanup used invoke stop.

Generated app	Enablement	`docs`	`nautobot`
Base	`invoke.yml`	`50321 -> 8001`	`50335 -> 8080`
Base	env var	`50442 -> 8001`	`50477 -> 8080`
Commercial	`invoke.yml`	`62307 -> 8001`	`62335 -> 8080`
Commercial	env var	`62451 -> 8001`	`62469 -> 8080`
ChatOps	`invoke.yml`	`64499 -> 8001`	`64533 -> 8080`
ChatOps	env var	`64631 -> 8001`	`64651 -> 8080`
SSoT	`invoke.yml`	`62224 -> 8001`	`62273 -> 8080`
SSoT	env var	`62354 -> 8001`	`62377 -> 8080`

Full outputs

testing-app
commercial-app
chatops-testing-app
ssot-testing-app

ChatOps Mattermost Custom Compose

Validated ChatOps Mattermost with an explicit custom compose_files list containing both ephemeral override files:

docker-compose.ephemeral-ports.yml
mattermost/docker-compose.ephemeral-ports.yml
invoke ps showed dynamic host ports for all published services, .service_ports.json included all three mappings, and cleanup used invoke stop.

Service	Mapping
`docs`	`63871 -> 8001`
`mattermost`	`63872 -> 8065`
`nautobot`	`63902 -> 8080`

Full output

chatops-mattermost

Other tests

All stacks were tested with both with and without ephemeral ports to ensure that a stack brought up with invoke debug would be properly torn down if ^C was entered in the terminal. All passed this test with the exception of the chatops cookie, however it failed for the same reason as running it under debug on develop:

nautobot-1  | Traceback (most recent call last):
nautobot-1  |   File "/usr/local/lib/python3.12/site-packages/nautobot_chatops/__init__.py", line 153, in ready
nautobot-1  |     super().ready()
nautobot-1  |   File "/usr/local/lib/python3.12/site-packages/nautobot_chatops/urls.py", line 8, in <module>
nautobot-1  |     from nautobot_chatops.integrations.grafana.urls import urlpatterns as grafana_urlpatterns
nautobot-1  |   File "/usr/local/lib/python3.12/site-packages/nautobot_chatops/integrations/grafana/grafana.py", line 387, in <module>
nautobot-1  |     handler = GrafanaHandler(PLUGIN_SETTINGS)
nautobot-1  |   File "/usr/local/lib/python3.12/site-packages/nautobot_chatops/integrations/grafana/grafana.py", line 69, in __init__
nautobot-1  |     self.config = _get_settings_from_chatops(config)
nautobot-1  |   File "/usr/local/lib/python3.12/site-packages/nautobot_chatops/integrations/grafana/grafana.py", line 47, in _get_settings_from_chatops
nautobot-1  |     return GrafanaConfigSettings(
nautobot-1  | pydantic_core._pydantic_core.ValidationError: 1 validation error for GrafanaConfigSettings
nautobot-1  | default_timespan
nautobot-1  |   Input should be a valid timedelta, input is too short [type=time_delta_parsing, input_value='', input_type=str]

To Do

Update the documentation.
Add changelog.
Verify your changes by testing them in the dev-example repository before merging.

gsnider2195 · 2026-06-27T01:10:18Z

+    """Write service port mappings for tools that cannot query Docker directly."""
+    service_ports = {}
+
+    for _ in range(4):


What's the purpose of trying multiple times? In case the services haven't come up yet?

yeah; it's functionally identical to the corresponding function in in the nautobot core. I've updated the docstring to also match the one used by the core, which is more explicit.

gsnider2195

I just have the one question but otherwise this looks good! Thanks

joewesch · 2026-06-29T18:20:15Z

+            except (json.decoder.JSONDecodeError, AttributeError, IndexError, KeyError):
+                continue
+
+        if set(["nautobot", "worker"]).issubset(service_ports.keys()):


Is this the correct list of expected services to be exposed by default? I added these changes to one of my dev environments and I saw this output:

Running docker compose command "ps --format json" Running docker compose command "ps --format json" Running docker compose command "ps --format json" Running docker compose command "ps --format json"

@nrnvgh this still needs changed. We don't expose the worker port by default, so this causes this loop to run all 4 times regardless. The docs service is enabled by default, but regardless I think we shouldn't expect any service except nautobot to be up to end this loop.

Dag nabbit. Good catch; fixed.

Before

% time invoke start Starting all services in detached mode... Running docker compose command "up --detach" Network testing-app_default Creating [...] Container testing-app-beat-1 Started invoke start 0.68s user 0.34s system 2% cpu 37.277 total

After

Starting all services in detached mode... Running docker compose command "up --detach" Network testing-app_default Creating Container testing-app-worker-1 Started [...] invoke start 0.37s user 0.17s system 3% cpu 16.770 total

joewesch · 2026-06-29T18:31:29Z

+    service_ports = {}
+
+    for _ in range(4):
+        result = docker_compose(context, "ps --format json", hide=True)


Based on my previous comment, it looks like we need to actually make hide=True work as it did not hide the print statements.

Easy enough. Would you rather I rolled that into this PR or submitted as its own?

It looks like just that one print statement, so I'd say roll it into this one.

Added. Command output:

Before

% pwd /<redacted>/cookiecutter-app-test-multioutput-bug/nautobot-app-testing-app % time invoke start -s db Starting db in detached mode... Running docker compose command "up --detach" Network testing-app_default Creating Network testing-app_default Created Container testing-app-db-1 Creating Container testing-app-db-1 Created Container testing-app-db-1 Starting Container testing-app-db-1 Started Running docker compose command "ps --format json" Running docker compose command "ps --format json" Running docker compose command "ps --format json" Running docker compose command "ps --format json" invoke start -s db 0.56s user 0.24s system 3% cpu 21.093 total

After

% pwd /<redacted>/cookiecutter-app-test-multioutput-bugfixed/nautobot-app-testing-app % time invoke start -s db Starting db in detached mode... Running docker compose command "up --detach" Network testing-app_default Creating Network testing-app_default Created Container testing-app-db-1 Creating Container testing-app-db-1 Created Container testing-app-db-1 Starting Container testing-app-db-1 Started invoke start -s db 0.57s user 0.25s system 3% cpu 21.129 total

joewesch · 2026-06-30T15:31:26Z

  nautobot_ver: "{{ min_nautobot_version }}"
  local: false
  python_ver: "3.12"
+  # ephemeral_ports: false


I assume this option will not work if the "default" example is to have a list of compose files that is different than the ORIGINAL_COMPOSE_FILES (same as mysql) so we should either drop it as a suggestion or add a comment here that explains when it will work.

Done. I also tweaked the dev_environment.md file to clarify the behavior there.

joewesch · 2026-06-30T15:51:47Z

FYI, the bake-template failures are a known issue with djlint changing the error code when it can't find any files to lint. We will have a fix for that shortly and we can merge it into your PR afterwards.

nrnvgh · 2026-06-30T20:18:40Z

I ran into another issue. By default, the cookiecutter repo brings up the docs container listening on port 8001 (ext) -> 8080 (int): ref: docker-compose.dev.yml (develop). This differs from how the nautobot core works; it maps 8001 -> 8001. I failed to catch it on that in my testing, but the result is that the docs container isn't reachable from the assigned ephemeral port.

The quick fix for this patch is to update things so that the port override will be applied to 8080 rather than 8001, thusly:

--- a/nautobot-app/{{ cookiecutter.project_slug }}/development/docker-compose.ephemeral-ports.yml       
+++ b/nautobot-app/{{ cookiecutter.project_slug }}/development/docker-compose.ephemeral-ports.yml       
@@ -6,7 +6,7 @@ services:
       - "8080"
   docs:
     ports: !override
-      - "8001"
+      - "8080"

That said, I think a better solution would be to update things so that both core and the cookiecutter code use the same ports. I can certainly think of reasons why that might not be acceptable, or wouldn't be acceptable in develop, tho. It also feels a little out of scope for this patch. I see three options:

Apply the minimal patch above and call it done
Apply the minimal patch above to complete the ephemeral ports work (modulo fixing any other findings) and submit a second PR to change the docs container to use internal port 8001
Don't apply the patch above, just change this repo to use internal port 8001

Thoughts?

Drop the misleading `ephemeral_ports` suggestion and add a comment to the invoke configuration examples that use a customized compose_files list.

nrnvgh · 2026-07-01T00:30:56Z

+  docs:
+    ports: !override
+      - "8001"


This is incorrect. cookiecutter-based apps used 8080 as the internal port, so the for ephemeral ports, 8001 should be changed to 8080...if we want to keep the docs container listening on 8080. Alternatively, we can change it to 8001, in line with the nautobot core, and leave this file as-is.

cmsirbu

@nrnvgh thanks for all the PRs and fixes! Our usual process is to first test any changes via PR to https://github.com/nautobot/nautobot-app-dev-example and once it is working and merged there, backport it into this repo as template changes.

As this PR is starting to have quite a few moving parts and testing surface, I will ask that we test it out in dev-example before merging, so that we discover any issues before these changes are propagated to 30+ repos via our drift management tooling.

On the topic of the docs container internal port, way too many years have passed and I don't think it makes much of a difference what it is since we override it anyway. If it makes it simpler, we should align it to 8001 (same as configured in mkdocs.yml too) and make sure it works as expected in dev-example.

joewesch · 2026-07-01T13:07:17Z

I agree that we should align to 0.0.0.0:8001 inside the container.

The docs container served mkdocs on 8080 and published it as 8001:8080, so the ephemeral-ports override (which publishes container port 8001) left docs unreachable on its assigned host port. Serve and publish on 8001 instead, matching mkdocs.yml's dev_addr and Nautobot core. Also replace the duplicated chatops ephemeral-ports override with a symlink to the base template, consistent with every other chatops compose file, so the two copies cannot drift apart.

Un-wrap the compose_command_tokens.append() call, which fits within the 120 column limit, so `invoke ruff --fix` has nothing to collapse in a freshly baked cookie.

Drop the leading slash from the .service_ports.json gitignore entry so the file is ignored at any depth, matching core. Raise the port discovery retry sleep from 5 to 15 seconds, also matching core.

nrnvgh · 2026-08-02T06:08:33Z

@nrnvgh thanks for all the PRs and fixes! Our usual process is to first test any changes via PR to https://github.com/nautobot/nautobot-app-dev-example and once it is working and merged there, backport it into this repo as template changes.

As this PR is starting to have quite a few moving parts and testing surface, I will ask that we test it out in dev-example before merging, so that we discover any issues before these changes are propagated to 30+ repos via our drift management tooling.

I updated this repo submitted nautobot/nautobot-app-dev-example#180 with the same changes (modulo some unrelated differences between this repo and that one). In the future, what would my workflow for changes to the cookiecutter repo be? Or, put another way, is that workflow documented somewhere I can see it? For example, if I submit a PR to dev-example, once it's merged do you have processes which will handle the backport to this repo or should I submit my own PR here as well?

nrnvgh added 2 commits June 17, 2026 18:15

Closes nautobot#386: Add ephemeral port support to generated apps

c8696ea

removed section from docs added in previous commit

4cb4310

gsnider2195 reviewed Jun 27, 2026

View reviewed changes

Comment thread nautobot-app/{{ cookiecutter.project_slug }}/docs/dev/dev_environment.md

gsnider2195 reviewed Jun 27, 2026

View reviewed changes

Restore mistakenly removed invoke configuration docs

3bec447

gsnider2195 approved these changes Jun 27, 2026

View reviewed changes

nrnvgh and others added 2 commits June 26, 2026 21:20

Align service port dump docstring with Nautobot core

1b7e09c

Merge branch 'develop' into u/nrnvgh-386-ephemeral-ports

47f235c

joewesch reviewed Jun 29, 2026

View reviewed changes

nrnvgh added 2 commits June 29, 2026 22:17

Don't emit 'Running docker compose command' when hide=True

e019d14

bugfix: reduced time it takes 'invoke start' to return

4728368

joewesch reviewed Jun 30, 2026

View reviewed changes

nrnvgh added 2 commits June 30, 2026 20:23

Clarify ephemeral ports usage in invoke config examples

a01813b

Drop the misleading `ephemeral_ports` suggestion and add a comment to the invoke configuration examples that use a customized compose_files list.

Clarify .service_ports.json behavior in dev environment docs

b00f792

nrnvgh commented Jul 1, 2026

View reviewed changes

cmsirbu requested changes Jul 1, 2026

View reviewed changes

nrnvgh and others added 4 commits August 1, 2026 23:26

Merge branch 'develop' into u/nrnvgh-386-ephemeral-ports

8cc463d

Fix isort violation in generated tasks.py

1b0e7c4

Un-wrap the compose_command_tokens.append() call, which fits within the 120 column limit, so `invoke ruff --fix` has nothing to collapse in a freshly baked cookie.

Align service port handling with Nautobot core

7a944f0

Drop the leading slash from the .service_ports.json gitignore entry so the file is ignored at any depth, matching core. Raise the port discovery retry sleep from 5 to 15 seconds, also matching core.

Uh oh!

Conversation

nrnvgh commented Jun 17, 2026

Closes #386

What's Changed

Summary

Validation

Static Ports Disabled

Raw output

Ephemeral Ports Enabled

Full outputs

ChatOps Mattermost Custom Compose

Full output

Other tests

To Do

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gsnider2195 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nrnvgh Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Before

After

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Before

After

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joewesch commented Jun 30, 2026

Uh oh!

nrnvgh commented Jun 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmsirbu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joewesch commented Jul 1, 2026

Uh oh!

nrnvgh commented Aug 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nrnvgh Jun 30, 2026 •

edited

Loading

cmsirbu left a comment •

edited

Loading