Add support for updating specified modules via go get and record as patch to go.mod, go.sum

[e4t]

To address issues with vendored modules - in particular to address CVEs quickly, obs-service-go_modules has been extended to 'patch' go.mod and go.sum.
The changes are described in a YAML file. The service will apply these changes to go.mod and go.sum before the the vendor files are downloaded. A patch will be created that can be applied during the %prep stage, also the changes file is updated with any changes not yet recorded.
This avoids having to touch the source tarball itself and allows to keep it pristine. At the same time, the updates to the YAML file can be done by an automated tool evaluating the results from govulncheck. Multiple updates to the same module should be harmless as long as the update to the later version comes later in the list.
Without additional entries to the _service file, obs-service-go_modules will behave as before.

These changes will complement the ideas discussed on the Go module dependency handling nicely.

[jfkw]

Thanks @e4t for the comprehensive PR and its useful functionality. Apologies for the delay in review, I'm currently working on priority tasks with the go1.x toolchain and it will be another week or two before I can review, test and align with some related functionality in #57 and #59.

This PR's functionality will be reviewed as a potential complement to #57 and #59. We also need to consider the VEX Vulnerability Exchange Format and upcoming its support in govulncheck. Currently purl-spec upstream is discussing whether to evolve the GOPATH-era golang: purl tag or create a modern go: successor tag which is modules-aware. govulncheck upstream is waiting on that decision.

I would prefer to avoid taking on an external dependency on YAML, so we would express config using _service parameters, a file format supported in the standard library, or as a last option a bespoke format e.g. key: value. I do like YAML the format for the essential scalars, lists, and dicts, it would be great here if that subset were in the standard library. Managing C extension different versions across distros is proving unwieldy. On that topic, I will cherry-pick your commit handling the libarchive API change ASAP, thanks for thinking of that run-time check.

Thanks again.

[jfkw]

Related use case in #61.

[e4t]

@jfkw - The YAML file allows for an ordered list where each list element contains two entries: uri and comment. This was chosen to support automation:

1. The fact that the list is ordered is important: it can be amended with new updates, potentially overriding older ones (without having to remove the previous ones).
2. An (optional) description can be added which will will be used to generate a changelog entry. This will relieve a tool from having to deal with potentially multiple changes files.
   The service infrastructure in osc only knows service names and parameters. It does seem to be possible to specify multiple parameter entries with the same name which will be added to the command line in the same order, but it is unclear whether this is a guaranteed. Also, a there is no concept of multiple elements per named parameter.
   Therefore, a

go_module:
   module_updates:
   - uri: golang.org/x/net@v0.23.0
     comment: "Update golang.org/x/net to v0.23 to fix CVE-2023-45288\n
(bnc#1236528)."
   - uri: github.com/containers/image/v5@v5.30.2
     comment: "Fixes an interoperability issue while listing tags from JFrog Artifactory."

would look somehwat like:

<services>
  <service name="go_modules" mode="disabled">
    <param name="updateuri">github.com/containers/image/v5@v5.30.2</param>
    <param name="updatecomment">Update golang.org/x/net to v0.23 to fix CVE-2023-45288\n
    (bnc#1236528).</param>
    < param name="updateuri">github.com/containers/image/v5@v5.30.2</param>
    <param name="updatecomment">Fixes an interoperability issue while listing tags from JFrog Artifactory.</param>
    <param name="moddiff">go-module.patch</param>
  </service>
[...]
</services>

which would then be converted to:

/usr/lib/obs/service/go_modules [...] --updateuri github.com/containers/image/v5@v5.30.2 \
   --updatecomment "Update golang.org/x/net to v0.23 to fix CVE-2023-45288\n(bnc#1236528)." \
   --updateuri github.com/containers/image/v5@v5.30.2 \
   --updatecomment "Fixes an interoperability issue while listing tags from JFrog Artifactory."

(AFAICT, the spaces on the entry text will be handled correctly as the entire text will be passed as a single entry of a list.)
Until here, the updateuri and updatecomment elements are associated. This association is lost when go_modules has parsed the command line: each command line argument will be stored in its own list. This would require to have as many comments as update URIs (or none at all).
I'm afraid the simplicity of the _service file model will limit what we are able to do. Using YAML will get us around it.
Using some bespoke key: value format in the xml text element would be ugly. If we are able to assume that the full URI will never contain spaces, we may get by with:

 < param name="update">github.com/containers/image/v5@v5.30.2 Fixes an interoperability issue while listing tags from JFrog Artifactory.</param>

and just treat text after the URI as a comment.

This PR's functionality will be reviewed as a potential complement to #57 and #59.

Both documents talk about running go_modules during the 'online' phase - are you referring to running this service 'server side'? Could you elaborate how this would would work, considering that vendoring will require access to the upstream repos while obs workers do not have access to the internet - at least as long as we do not maintain our own mirrors of these repositories.

We also need to consider the VEX Vulnerability Exchange Format and upcoming its support in govulncheck. Currently purl-spec upstream is discussing whether to evolve the GOPATH-era golang: purl tag or create a modern go: successor tag which is modules-aware. govulncheck upstream is waiting on that decision.

The version of govulncheck I've tested seems to use neither format. For our purposes the use of a full PURL would be redundant in the YAML of _service file.
I would like to suggest to not make the integration of govulncheck into obs-service-go_modules mandatory: for both of my use cases it does not work with the package sources (as these require configuration outside of the Go eco system). It does, however, work with the binaries. These may not be available, though, when the source service is run.

[mcepl]

Would you consider my comments?

[mcepl]

Why is that line here when it is the default value? Is there some hidden purpose for it?

[e4t]

It's an example of / template for the service file. It's not uncommon to use the default value in examples, is it? There's no hidden agenda, no.

[mcepl]

So, why not just omit it there, and omit this paragraph here?

[e4t]

This is to avoid confusion. I can move the comment up - in front of the example - but I would like to leave this with all options in place to serve as a template.

[jfkw]

@e4t thanks for configuring the PR branch to be open for collaborator push. I added one small fix for a merge conflict. There will be more pre-merge fixup commits in the coming days as new commits land. If there are any suggested changes to how the PR works, they'll be discussed here before pushing.

[e4t]

@jfkw - Thank you! Please let me know if you want me to try and convert the use of an external YAML file to xml - using the existing infrastructure of the _service file.
I may not be able to do it this week though as I'm at the SUSE Labs conference.

[jfkw]

@e4t. No changes needed at the moment, enjoy the conference.

[mcepl]

OK, I feel persuaded.

[mcepl]

The need for this has been replaced by (merged) #57, wasn’t it?