Skip to content

Commit 04ae1c0

Browse files
druvusclaude
andcommitted
fix(deployment): harden offline bundle import for move-to-another-computer
Audit of the Deployment tab's move-to-another-computer workflow surfaced gaps on the import side that only bite at first launch on the field machine. Harden import_bundle (warn-only where import-first-point-later is valid; hard-fail only on a genuinely broken bundle): - Empty/placeholder Kraken2 DB: flag kraken_db_unset + warn instead of silently leaving the ${KRAKEN_DB} placeholder (matched as a substring since ConfigLoader normalises it to <cwd>/${KRAKEN_DB}). - Missing main.nf in a bundled pipeline: hard-fail the import (was a warning). - Empty Nextflow plugins dir: warn (would otherwise fall back to the online registry and break offline). - Re-verify checksums AFTER the copy to home (excluding the templated/mutated genome_metadata.json and config.yaml) to catch a truncated copy. - Record a Nextflow version floor (26.04.0) in the manifest and warn on import when the field machine is older. - Path-aware genome_metadata templating on export: report any path outside the data home instead of silently leaking it. UI: the import callback surfaces kraken_db_unset / plugins_empty as a prominent "Action required" row so the operator does not read "Offline mode activated" as "ready to run". Docs: add a step-by-step "Move Nanometa Live to another computer" section to OPERATOR_GUIDE.md and document nxf_conda_cachedir / nxf_plugins_dir / the import-sets-offline_mode behaviour in configuration.md. Tests: new test_bundle_roundtrip.py (real export -> import into a fresh home, asserting config rebasing / offline_mode / restored data / re-templated metadata) plus per-finding classes in test_bundle_manager.py. Full suite 2618 passed; a live same-machine round-trip verified end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 1100e0d commit 04ae1c0

6 files changed

Lines changed: 629 additions & 12 deletions

File tree

docs/OPERATOR_GUIDE.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,74 @@ The most operationally relevant tabs:
124124
watchlists or load a custom one.
125125
- **Preparation** -- before running validation on a new organism, or when
126126
setting up an offline-deployment bundle.
127+
- **Deployment** -- to package this installation for another computer, or to
128+
install a package received from another computer (see the next section).
129+
130+
---
131+
132+
## Move Nanometa Live to another computer (offline deployment)
133+
134+
Use this to run on an air-gapped or field computer that has no internet
135+
access. You prepare a bundle on a build machine that does have internet,
136+
transfer it, and import it on the field machine.
137+
138+
### What transfers
139+
140+
The bundle (a single `.tar.gz`) contains the reference genomes, BLAST
141+
databases, taxid mappings, taxonomy cache, watchlists, the pipeline source,
142+
and the Nextflow plugins -- optionally also pre-warmed conda environments or
143+
container images. The **Kraken2 database is not included** because of its size;
144+
copy it separately (for example on the same drive).
145+
146+
### 1. On the build machine -- export
147+
148+
1. Open the **Deployment** tab and use **Export Bundle**.
149+
2. Choose the container engine:
150+
- **conda** -- smallest bundle, but the field machine must have the **same
151+
operating system and CPU architecture** as the build machine (for example
152+
both Linux x86_64, or both macOS arm64). Conda environments are not
153+
portable across platforms.
154+
- **docker** or **singularity/apptainer** -- portable across operating
155+
systems; the bundle carries the container images.
156+
3. Optionally tick **pre-warm conda envs** (conda only). This bakes the
157+
per-process environments so the field machine does not build them on first
158+
run; it adds roughly 30 minutes and several GB, and keeps the same-OS/arch
159+
restriction above.
160+
4. The bundle is written to the chosen folder.
161+
162+
### 2. Transfer
163+
164+
Copy the bundle `.tar.gz` and the Kraken2 database directory to the field
165+
machine (USB drive, etc.). The field machine needs roughly three times the
166+
bundle size in free space for extraction.
167+
168+
### 3. On the field machine -- import
169+
170+
1. Open the **Deployment** tab and use **Import Bundle**.
171+
2. Provide the path to the bundle **and** the path to the Kraken2 database on
172+
this machine.
173+
3. If the bundle was pre-warmed, run `source activate_offline_envs.sh` from the
174+
install directory in the shell that launches Nanometa Live.
175+
176+
Messages you may see on import, and what to do:
177+
178+
- *"Kraken2 database path was not provided"* -- you imported without giving the
179+
database path. Set `kraken_db` on the Watchlist & Preparation tab (or in
180+
`config.yaml`) before starting analysis.
181+
- *Platform mismatch* -- the bundle was built on a different OS/architecture.
182+
With pre-warmed conda envs the import is refused; rebuild the bundle on a
183+
matching machine, or use docker/singularity mode.
184+
- *"missing main.nf"* -- the bundled pipeline is incomplete or was truncated in
185+
transfer; re-export and re-transfer.
186+
- *"empty plugins directory"* -- re-export from a machine with the Nextflow
187+
plugins cached, or the offline run will fail when Nextflow probes the online
188+
plugin registry.
189+
190+
### 4. After import
191+
192+
Offline mode is enabled automatically. Open the **Watchlist & Preparation**
193+
tab, run the **Readiness** checklist until everything is green, then click
194+
**Start Analysis**.
127195

128196
---
129197

docs/configuration.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ nanopore_output_directory: "/path/to/input"
3131
| `processing_mode` | string | "batch" | `batch` or `realtime` |
3232
| `sample_handling` | string | "by_barcode" | `by_barcode`, `single_sample`, or `per_file` |
3333
| `sample_name` | string | "sample" | Name when using single_sample mode |
34-
| `offline_mode` | bool | false | Skip network calls and use cached data only |
34+
| `offline_mode` | bool | false | Skip network calls and use cached data only. Set automatically to `true` by a Deployment-tab bundle import; if the Kraken2 database path was not supplied at import, set `kraken_db` before starting analysis. |
3535

3636
### Kraken2 Classification
3737

@@ -48,6 +48,8 @@ nanopore_output_directory: "/path/to/input"
4848
|-----------|------|---------|-------------|
4949
| `pipeline_profile` | string | "conda" | `conda` is the canonical and supported profile for nanometanf. `docker` and `singularity` exist but are not used by Nanometa Live. |
5050
| `pipeline_source` | string | "remote:master" | Pipeline location (see below) |
51+
| `nxf_conda_cachedir` | path | null | Field-machine path to a bundle's pre-warmed Nextflow conda cache. Set automatically by bundle import; exported as `NXF_CONDA_CACHEDIR`. |
52+
| `nxf_plugins_dir` | path | null | Field-machine path to a bundle's Nextflow plugin cache. Set automatically by bundle import; suppresses the online plugin-registry probe in offline mode. |
5153
| `pipeline_cores` | int | 1 | CPU cores for pipeline |
5254
| `kraken_cores` | int | 1 | CPU cores for Kraken2 classification |
5355
| `validation_cores` | int | 1 | CPU cores for validation tasks |

nanometa_live/app/tabs/preparation_tab.py

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -763,6 +763,30 @@ def import_bundle(n_clicks, bundle_path, kraken_db_path):
763763
html.I(className="bi bi-check-circle me-2"),
764764
html.Strong("Bundle imported. Offline mode activated."),
765765
]
766+
# Setup that is not yet complete (action required) -- surface
767+
# these prominently so the operator does not read "activated"
768+
# as "ready to run".
769+
action_needed = []
770+
if result.get("kraken_db_unset"):
771+
action_needed.append(
772+
"Set the Kraken2 database path before starting analysis "
773+
"(it is transferred separately from the bundle)."
774+
)
775+
if result.get("plugins_empty"):
776+
action_needed.append(
777+
"Bundled Nextflow plugins are missing; re-export from a "
778+
"machine with the plugins cached, or the offline run will "
779+
"fail when Nextflow probes the online plugin registry."
780+
)
781+
if action_needed:
782+
children.append(
783+
dbc.Alert(
784+
[html.Strong("Action required: ")]
785+
+ [html.Div(a, className="small") for a in action_needed],
786+
color="warning",
787+
className="mt-2 mb-2",
788+
)
789+
)
766790
if result["warnings"]:
767791
children.append(html.Br())
768792
children.append(html.Strong("Warnings: "))

0 commit comments

Comments
 (0)