Closes #386: Add ephemeral port support to generated apps

changes/386.added

@@ -0,0 +1 @@
+Added opt-in ephemeral Docker host ports for generated app development environments.

nautobot-app-chatops/{{ cookiecutter.project_slug }}/development/docker-compose.ephemeral-ports.yml

@@ -0,0 +1,19 @@
+# Set host ports to ephemeral values to avoid conflicts with other local services.
+---
+services:
+  nautobot:
+    ports: !override
+      - "8080"
+  docs:
+    ports: !override
+      - "8001"
+# To expose postgres (5432), mysql (3306) on db service or redis (6379) to the host using
+# ephemeral ports, uncomment the following. Ensure to match the 2 indented spaces which
+# have the service nested under services.
+# db:
+#   ports: !override
+#     - "5432"
+#     - "3306"
+# redis:
+#   ports: !override
+#     - "6379"

nautobot-app-chatops/{{ cookiecutter.project_slug }}/development/mattermost/docker-compose.ephemeral-ports.yml

@@ -0,0 +1,6 @@
+# Set Mattermost host ports to ephemeral values to avoid conflicts with other local services.
+---
+services:
+  mattermost:
+    ports: !override
+      - "8065"

nautobot-app-chatops/{{ cookiecutter.project_slug }}/invoke.mattermost.yml

@@ -10,4 +10,9 @@
     - "docker-compose.redis.yml"
     - "docker-compose.postgres.yml"
     - "docker-compose.dev.yml"
+    # Uncomment the ephemeral-ports overrides below to use dynamic host ports with this
+    # custom compose file list. The `ephemeral_ports` invoke option only auto-applies when
+    # compose_files is left at its default, so it has no effect here and is intentionally omitted.
+    # - "docker-compose.ephemeral-ports.yml"
     - "mattermost/docker-compose.yml"
+    # - "mattermost/docker-compose.ephemeral-ports.yml"

nautobot-app-commercial/{{ cookiecutter.project_slug }}/development/docker-compose.ephemeral-ports.yml

@@ -0,0 +1 @@
+../../../nautobot-app/{{ cookiecutter.project_slug }}/development/docker-compose.ephemeral-ports.yml

nautobot-app-ssot/{{ cookiecutter.project_slug }}/development/docker-compose.ephemeral-ports.yml

@@ -0,0 +1 @@
+../../../nautobot-app/{{ cookiecutter.project_slug }}/development/docker-compose.ephemeral-ports.yml

nautobot-app/{{ cookiecutter.project_slug }}/.gitignore

@@ -308,6 +308,7 @@ invoke.yml
 public
 /compose.yaml
 /dump.sql
+/.service_ports.json
 /{{ cookiecutter.app_name }}/static/{{ cookiecutter.app_name }}/docs
 
 # Claude Code

nautobot-app/{{ cookiecutter.project_slug }}/development/docker-compose.ephemeral-ports.yml

@@ -0,0 +1,19 @@
+# Set host ports to ephemeral values to avoid conflicts with other local services.
+---
+services:
+  nautobot:
+    ports: !override
+      - "8080"
+  docs:
+    ports: !override
+      - "8001"
+# To expose postgres (5432), mysql (3306) on db service or redis (6379) to the host using
+# ephemeral ports, uncomment the following. Ensure to match the 2 indented spaces which
+# have the service nested under services.
+# db:
+#   ports: !override
+#     - "5432"
+#     - "3306"
+# redis:
+#   ports: !override
+#     - "6379"

nautobot-app/{{ cookiecutter.project_slug }}/docs/dev/dev_environment.md

@@ -19,6 +19,7 @@ The [Invoke](http://www.pyinvoke.org/) library is used to provide some helper co
 - `local`: a boolean flag indicating if invoke tasks should be run on the host or inside the docker containers (default: False, commands will be run in docker containers)
 - `compose_dir`: the full path to a directory containing the project compose files
 - `compose_files`: a list of compose files applied in order (see [Multiple Compose files](https://docs.docker.com/compose/extends/#multiple-compose-files) for more information)
+- `ephemeral_ports`: Setting this value to `true` and not using any custom compose files will make all Nautobot containers with published ports expose themselves with dynamic ports. This is useful when running multiple Nautobot versions at the same time on the same machine so you won't experience system port conflicts. If setting `compose_files`, this will have no effect so please ensure to manually add the applicable `docker-compose.ephemeral-ports.yml` file or files to your list.
 
 Using **Invoke** these configuration options can be overridden using [several methods](https://docs.pyinvoke.org/en/stable/concepts/configuration.html). Perhaps the simplest is setting an environment variable `INVOKE_{{ cookiecutter.app_name.upper() }}_VARIABLE_NAME` where `VARIABLE_NAME` is the variable you are trying to override. The only exception is `compose_files`, because it is a list it must be overridden in a YAML file. There is an example `invoke.yml` (`invoke.example.yml`) in this directory which can be used as a starting point.
 
@@ -45,6 +46,8 @@ invoke start
 
 The Nautobot server can now be accessed at [http://localhost:8080](http://localhost:8080) and the live documentation at [http://localhost:8001](http://localhost:8001).
 
+Every `invoke start` and `invoke debug` writes the published host port mappings to `.service_ports.json`, listing only the services that publish a port to the host. When ephemeral ports are enabled, Docker assigns dynamic host ports and this file captures the resulting values; with fixed ports it captures the static values instead. You can also inspect them with `invoke ps` or `docker compose port`, for example `docker compose port nautobot 8080`. To enable ephemeral ports with an environment variable, set `INVOKE_{{ cookiecutter.app_name.upper() }}_EPHEMERAL_PORTS=1`; to disable them, unset the environment variable, set it to an empty value, or set it to `0`.
+
 To either stop or destroy the development environment use the following options.
 
 - **invoke stop** - Stop the containers, but keep all underlying systems intact

nautobot-app/{{ cookiecutter.project_slug }}/invoke.example.yml

@@ -3,6 +3,7 @@
   nautobot_ver: "{{ min_nautobot_version }}"
   python_ver: "3.12"
   # local: false
+  # ephemeral_ports: false
   # compose_dir: "/full/path/to/{{ cookiecutter.project_slug }}/development"
 
 # The following is an example of using MySQL as the database backend

nautobot-app/{{ cookiecutter.project_slug }}/invoke.mysql.yml

@@ -10,3 +10,7 @@
     - "docker-compose.redis.yml"
     - "docker-compose.mysql.yml"
     - "docker-compose.dev.yml"
+    # Uncomment the ephemeral-ports override below to use dynamic host ports with this
+    # custom compose file list. The `ephemeral_ports` invoke option only auto-applies when
+    # compose_files is left at its default, so it has no effect here and is intentionally omitted.
+    # - "docker-compose.ephemeral-ports.yml"

nautobot-app/{{ cookiecutter.project_slug }}/tasks.py

@@ -12,6 +12,8 @@
 limitations under the License.
 """
 
+import concurrent.futures
+import json
 import os
 import re
 import shutil
@@ -24,6 +26,14 @@
 from invoke.tasks import task as invoke_task
 
 
+ORIGINAL_COMPOSE_FILES = [
+    "docker-compose.base.yml",
+    "docker-compose.redis.yml",
+    "docker-compose.postgres.yml",
+    "docker-compose.dev.yml",
+]
+
+
 def is_truthy(arg):
     """Convert "truthy" strings into Booleans.
 
@@ -56,13 +66,9 @@ def is_truthy(arg):
             "project_name": "{{ cookiecutter.app_slug }}",
             "python_ver": "3.12",
             "local": False,
+            "ephemeral_ports": False,
             "compose_dir": os.path.join(os.path.dirname(__file__), "development"),
-            "compose_files": [
-                "docker-compose.base.yml",
-                "docker-compose.redis.yml",
-                "docker-compose.postgres.yml",
-                "docker-compose.dev.yml",
-            ],
+            "compose_files": ORIGINAL_COMPOSE_FILES.copy(),
             "compose_http_timeout": "86400",
         }
     }
@@ -138,19 +144,70 @@ def docker_compose(context, command, **kwargs):
         compose_file_path = os.path.join(context.{{ cookiecutter.app_name }}.compose_dir, compose_file)
         compose_command_tokens.append(f' -f "{compose_file_path}"')
 
+    if (
+        context.{{ cookiecutter.app_name }}.ephemeral_ports
+        and context.{{ cookiecutter.app_name }}.compose_files == ORIGINAL_COMPOSE_FILES
+    ):
+        compose_file_path = os.path.join(
+            context.{{ cookiecutter.app_name }}.compose_dir, "docker-compose.ephemeral-ports.yml"
+        )
+        compose_command_tokens.append(
+            f' -f "{compose_file_path}"'
+        )
+
     compose_command_tokens.append(command)
 
     # If `service` was passed as a kwarg, add it to the end.
     service = kwargs.pop("service", None)
     if service is not None:
         compose_command_tokens.append(service)
 
-    print(f'Running docker compose command "{command}"')
+    if "hide" not in kwargs:
+        print(f'Running docker compose command "{command}"')
     compose_command = " ".join(compose_command_tokens)
 
     return context.run(compose_command, env=build_env, **kwargs)
 
 
+@task
+def dump_service_ports_to_disk(context):
+    """Useful for downstream utilities without direct docker access to determine ports.
+
+    This function will sometimes be called asynchronously while containers are still
+    firing up, hence the `attempt` loop.
+    """
+    service_ports = {}
+
+    for _ in range(4):
+        result = docker_compose(context, "ps --format json", hide=True)
+
+        for line in result.stdout.splitlines():
+            try:
+                service_def = json.loads(line)
+                service_name = re.search(
+                    r"com\.docker\.compose\.service=(?P<service>\w+)", service_def["Labels"]
+                ).group("service")
+
+                ports_found = {}
+                for port in service_def["Publishers"]:
+                    if port.get("PublishedPort", 0):
+                        ports_found[port["TargetPort"]] = port["PublishedPort"]
+
+                if ports_found:
+                    service_ports[service_name] = ports_found
+            except (json.decoder.JSONDecodeError, AttributeError, IndexError, KeyError):
+                continue
+
+        # Confirm nautobot has started
+        if set(["nautobot"]).issubset(service_ports.keys()):
+            break
+
+        sleep(5)
+
+    with open(".service_ports.json", "w", encoding="utf-8") as file:
+        json.dump(service_ports, file, indent=4)
+
+
 def run_command(context, command, service="nautobot", **kwargs):
     """Wrapper to run a command locally or inside the nautobot container."""
     if is_truthy(context.{{ cookiecutter.app_name }}.local):
@@ -298,7 +355,9 @@ def debug(context, service=None):
     """Start specified or all services and its dependencies in debug mode."""
     service = " ".join(service) if service else ""
     print(f"Starting {service or 'all services'} in debug mode...")
-    docker_compose(context, "up", service=service)
+    with concurrent.futures.ThreadPoolExecutor() as executor:
+        executor.submit(dump_service_ports_to_disk, context)
+        docker_compose(context, "up", service=service)
 
 
 @task(
@@ -312,6 +371,7 @@ def start(context, service=None):
     service = " ".join(service) if service else ""
     print(f"Starting {service or 'all services'} in detached mode...")
     docker_compose(context, "up --detach", service=service)
+    dump_service_ports_to_disk(context)
 
 
 @task(