Skip to content

As a node operator, I want harvest to fail when file_ref paths are not converted to web URLs by replacePrefix #329

Description

@jordanpadams

Summary

As a node operator, I want harvest to fail loudly when a file_ref path is not converted to a web URL by replacePrefix, so that local filesystem paths are never silently stored in the registry as product references.

Motivation

When fileInfo/fileRef is configured with a replacePrefix but a product's file path does not match any configured prefix, harvest currently stores a raw local file path (e.g., /path/to/archive/data.xml or a file:// URI) as the file_ref value in the registry. File references in the registry must be publicly accessible HTTP/HTTPS URLs — local paths are meaningless to end users and downstream services that resolve those references.

This silent failure can go undetected, leading to registry data that cannot be dereferenced. Harvest should detect this condition and fail for affected products so operators are alerted immediately.

Acceptance Criteria

  • After all fileRef rules are evaluated, if the resulting file_ref value does not start with http:// or https://, harvest must log a clear error message identifying the affected product (LID/VID) and file path, and the product must not be loaded into the registry.
  • The error message must include: the file path that failed conversion, and the LID/VID of the affected product.
  • When the -f / --force flag is provided, harvest logs a warning instead of an error and continues loading the product (preserving existing behavior for operators who explicitly opt in).
  • Unit tests cover:
    • (a) File path matching a replacePrefix is correctly converted to an HTTP URL → product loads successfully.
    • (b) File path not matching any replacePrefix → validation error, product is not loaded (no --force).
    • (c) File path not matching any replacePrefix with --force → warning logged, product is loaded.

Additional Context

Relevant code:

  • Replacement logic: FileMetadataExtractor.getFileRef() in registry-common — if no rule prefix matches, the raw file URI path is returned unchanged.
  • file_ref is set for both Label_File_Info/file_ref and Data_File_Info/file_ref.
  • The -f/--force flag is already defined in HarvestCli and propagated to RegistryManager — it should also suppress this new validation.

Example config (expected usage):

<fileRef replacePrefix="/path/to/archive" with="https://url/to/archive/" />

With this config, any product whose label path does not begin with /path/to/archive would fail validation unless --force is supplied.

Metadata

Metadata

Assignees

Fields

No fields configured for Feature.

Projects

Status
ToDo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions