Add Tool Assessment Predicate Type - #496
Conversation
Signed-off-by: JosephKyser <josephkyserjr@gmail.com>
74f411c to
89065dc
Compare
|
Bumping request for review |
adityasaky
left a comment
adityasaky
left a comment
There was a problem hiding this comment.
This is pretty cool! Thanks for proposing this predicate type. :)
|
|
||
| ## Purpose | ||
|
|
||
| **The tool assessment attestation references the results and metadata associated with tools used to assess software before or after its creation.** It's primary purpose is to provide an immutable attestation of tool assessment of a software so that it can be bundled with its provenance. This enables to mapping of build provenances to assessments on a per build basis and can reflect entire DevSecOps pipeline processes. |
There was a problem hiding this comment.
nit: I would make the specific provenance bit here part of the use cases below. Ideally, the purpose section is more general in tone and content.
There was a problem hiding this comment.
+1
I would also like to see a bit more specificity for what "bundled"/"mapped" to provenance means. in-toto for now mostly assumes that related attestations will be shipped together in jsonl format, and we have predicates that allow for explicit references of one attestation from another (e.g., evidence field in SCAI). I bring this up to make sure we're being clear about how the tool assessment predicate would be used with respect to other predicate types.
|
|
||
| There are many existing predicates that describe the use of specific tool types. While these predicates are well defined, they are narrowly scoped. There should exist a predicate that is general enough to effectively attest the use of any tool that can be used if a tool type does not have a predicate type yet. The tool assessment attestation type aims to solve this. | ||
|
|
||
| - The [cyclonedx](cyclonedx.md) and [spdx](spdx.md) predicate types describe SBOM standards. |
There was a problem hiding this comment.
I wonder if this is not too far from witness's commandrun attestor. @jkjell and others may have some thoughts there. If there's overlap, maybe we can consolidate the two.
There was a problem hiding this comment.
I wonder if this is not too far from witness's commandrun attestor. @jkjell and others may have some thoughts there. If there's overlap, maybe we can consolidate the two.
I've actually looked a bit at witness and I really like the concept of command run. One of my concerns is how do I map results to an ingestible format in the attestation? The answer may be a 1-2 punch where witness attests "this is what was ran with these raw results" and tool assessment could transform the raw results into an ingestible format, but then you lose the chain of trust that witness aims to provide by allowing that second step to interpret findings.
There was a problem hiding this comment.
We could also have it so that a tool like witness produces an assessment predicate about the tool that was run and captured in the command run predicate; assuming witness can sufficiently identify the information that goes here.
|
|
||
| ### Control Gates | ||
|
|
||
| Control gates are an essential and increasingly prevalent requirement in many |
There was a problem hiding this comment.
Could you add an example here? Not necessarily a full attestation but a scenario where a tool is used, its invocation is attested, and what the assessors may be looking at in the attestation.
|
|
||
| ### Policy as Code Enabling via Attachment to Build Artifacts | ||
|
|
||
| Tool assessment attestations bundled with container build provenance can enable policy-as-code enforcement of containers or software on IT systems. |
There was a problem hiding this comment.
I'm not entirely sure I follow this brief example.
|
|
||
| **config.exclusions, optional** string list | ||
|
|
||
| >> List of deviations from the profile, such as rule IDs, file names, ignores, etc. |
There was a problem hiding this comment.
Can these be resource descriptors? If we're using files to populate exclusions, it'd be good to hash them for transparency, for eg.
There was a problem hiding this comment.
That was my initial thought however I found that it does not cover the case where exclusions originate from the command line as options. I did include config.files so that any exclusions that exists in files (e.g. .semgrepignore) could be included as a resource descriptor.
There was a problem hiding this comment.
As config.files can contain more than exclusions, I worry about that field becoming confusing. Maybe this should not be a resource descriptor but should be a more explicit type stating the exclusion and the source of the exclusion (e.g., config value, CLI value, etc).
There was a problem hiding this comment.
Looking at it, I agree that the field is too permissive. I'm not sure how to strike a balance between being broad enough to cover all use-cases while being specific enough to provide auditability and transparency while also not being a PITA to write for the user.
Maybe something along the line of discrepancies or alterations.
As for the schema format? Maybe you're right regarding an object that explicitly states the type. I'm just trying to avoid making more sub-schemas as necessary but it might be necessary in this case.
There was a problem hiding this comment.
As I noted above, I think a simple sub-schema is totally acceptable for fields if they can offer clarity on the semantics of the fields. Other general predicates like SLSA Provenance and SCAI have sub-schemas in one or more fields, so there is certainly precedent.
| } | ||
|
|
||
| // Profile Object Type | ||
| Profile: { |
There was a problem hiding this comment.
More generally, can this be a resource descriptor?
There was a problem hiding this comment.
More generally, can this be a resource descriptor?
I think it absolutely could for many cases. Databases and file based configurations come to mind for this. The problem is when an internal configuration changes how a tool behaves.
Take semgrep as an example. It provides language specific registries you can use by adding them in the tool options. I am unsure how a case like this would be described by a resource descriptor other than maybe a URI and heavy use of the annotations field.
There was a problem hiding this comment.
Curious what @marcelamelara thinks about the last bit. How would language specific registries and the like live in the current schema in your view? Are they still annotations?
Thinking "aloud". There's a recurring question about how to make our generic building blocks (SCAI, resource descriptors) more specific. I could see "known" annotations. For example, the annotations field could have known types for semgrep-specific properties. Same with SCAI properties. But I'm not sure where they'd live. Maybe the solution is to state here that the properties go there and require the tools producing the attestation to enumerate the annotations and define their semantics. 🤔
There was a problem hiding this comment.
Another way to look at it perhaps is that profile is a specific, opinionated resource descriptor for this use case, sort of answering my question about ways of making a generic building block more specific.
There was a problem hiding this comment.
I don't think it's unreasonable to define a sub-schema for predicates, though in this case, it's not clear to me how a Profile solves the issue raised in the semgrep example.
Structure-wise, the difference between Profile and ResourceDescriptor seems relatively small. profile replaces the name field, and there are added version and last_updated fields. The version could be included in the URI or name (as is done in SLSA Provenance, for example), and the Attestation Framework spec allows for extension fields in any object, so, version and last_updated could just become part of an "opinionated" ResourceDescriptor specific to this predicate, without needing to name the structure differently.
What properties of the configuration are we actually trying to tease out and capture explicitly with Profiles besides information already captured in ResourceDescriptors?
FWIW, we have received several requests to add a timestamp field to the ResourceDescriptor, so I'm counting this as another data point in favor!
|
|
||
| >> List of deviations from the profile, such as rule IDs, file names, ignores, etc. | ||
|
|
||
| **config.files, optional** ResourceDescriptor list |
There was a problem hiding this comment.
As my other comments may suggest, I think I'm a bit confused by profiles vs files. Do we have files that won't be profiles or exclusions? :)
There was a problem hiding this comment.
Sure! My thought process was that the predicate had to be able to included many different configurations such as databases, config files, internal options, entire rulesets, etc. My challenge was how can I create a model that could be any of those configurations, and profiles are what I came up with. AV runs with this database? That's a profile. Fortify uses these internal ruleset to apply SAST findings? That's a profile.
Files come into play where we want to deviate from the profile. This usually comes in the form of exclusions/ignores. Some tools ingest a specific file to apply these exclusions, such as a .semgrepignores. The ultimate goal is for the predicate to have enough information to replicate (or validate) the findings where findings = tool([profiles-exclusions])
|
|
||
| **output, required** ResourceDescriptor list | ||
|
|
||
| > Artifacts associated with the result of the execution of the tool. |
There was a problem hiding this comment.
This raises an interesting question. What do we foresee as the subject of this attestation? Is it the artifact under test, i.e., that the tool operated on? It would be good to add this to the model section.
There was a problem hiding this comment.
Could the output point to another attestation? For example, the test results or vuln scan predicates could be good counterparts to this one in that model.
There was a problem hiding this comment.
I work in the realm of containers. For my use-case, I envisioned all of the subjects pointing to the image digest of the containers, but I think the artifact under test would be just as valid.
There was a problem hiding this comment.
Could the output point to another attestation? For example, the test results or vuln scan predicates could be good counterparts to this one in that model.
I suppose it could, however I think those attestations could just stand on their own merits. In a perfect world where all tool types have a corresponding predicate type, tool attestations wouldn't be needed.
There was a problem hiding this comment.
however I think those attestations could just stand on their own merits
I agree. I think this one can make those other ones better. A generic test predicate answers the question "did tests pass" while this one answers questions about the test framework itself.
There was a problem hiding this comment.
Just based on the name output my expectation would be for this field to either contain something like a log of the tool's execution. To avoid any confusion, I would add a clarification at the top of the Field section clarifying that the subject field for this predicate contains the assessed artifacts (e.g., container images).
ea99683 to
79fb7d3
Compare
Kusari Analysis Results:
No pinned version dependency changes, code issues or exposed secrets detected! Note View full detailed analysis result for more information on the output and the checks that were run.
Found this helpful? Give it a 👍 or 👎 reaction! |
79fb7d3 to
e4a3c1b
Compare
|
Kusari PR Analysis rerun based on - e4a3c1b performed at: 2026-02-11T17:22:00Z - link to updated analysis |
|
@kusari-inspector re-run |
|
🔄 Run triggered at 17:36:46 UTC. Starting fresh analysis... |
|
Kusari PR Analysis rerun based on - e4a3c1b performed at: 2026-02-11T17:37:09Z - link to updated analysis |
adityasaky
left a comment
adityasaky
left a comment
There was a problem hiding this comment.
I propose we add this to the agenda for the next in-toto community meeting. WDYT @marcelamelara and @SantiagoTorres?
|
|
||
| There are many existing predicates that describe the use of specific tool types. While these predicates are well defined, they are narrowly scoped. There should exist a predicate that is general enough to effectively attest the use of any tool that can be used if a tool type does not have a predicate type yet. The tool assessment attestation type aims to solve this. | ||
|
|
||
| - The [cyclonedx](cyclonedx.md) and [spdx](spdx.md) predicate types describe SBOM standards. |
There was a problem hiding this comment.
We could also have it so that a tool like witness produces an assessment predicate about the tool that was run and captured in the command run predicate; assuming witness can sufficiently identify the information that goes here.
| } | ||
|
|
||
| // Profile Object Type | ||
| Profile: { |
There was a problem hiding this comment.
Curious what @marcelamelara thinks about the last bit. How would language specific registries and the like live in the current schema in your view? Are they still annotations?
Thinking "aloud". There's a recurring question about how to make our generic building blocks (SCAI, resource descriptors) more specific. I could see "known" annotations. For example, the annotations field could have known types for semgrep-specific properties. Same with SCAI properties. But I'm not sure where they'd live. Maybe the solution is to state here that the properties go there and require the tools producing the attestation to enumerate the annotations and define their semantics. 🤔
| } | ||
|
|
||
| // Profile Object Type | ||
| Profile: { |
There was a problem hiding this comment.
Another way to look at it perhaps is that profile is a specific, opinionated resource descriptor for this use case, sort of answering my question about ways of making a generic building block more specific.
|
|
||
| **config.exclusions, optional** string list | ||
|
|
||
| >> List of deviations from the profile, such as rule IDs, file names, ignores, etc. |
There was a problem hiding this comment.
As config.files can contain more than exclusions, I worry about that field becoming confusing. Maybe this should not be a resource descriptor but should be a more explicit type stating the exclusion and the source of the exclusion (e.g., config value, CLI value, etc).
|
|
||
| **output, required** ResourceDescriptor list | ||
|
|
||
| > Artifacts associated with the result of the execution of the tool. |
There was a problem hiding this comment.
however I think those attestations could just stand on their own merits
I agree. I think this one can make those other ones better. A generic test predicate answers the question "did tests pass" while this one answers questions about the test framework itself.
|
|
||
| **tool.type, required** string | ||
|
|
||
| >> Description of the type of tool (SAST, DAST, SECRETS, etc). |
There was a problem hiding this comment.
I'm guessing we don't expect this to be exhaustive, is that fair to say?
There was a problem hiding this comment.
Do we expect these types to be up to the producer and consumer? If so, I would include this detail in this description.
| "full_command": "semgrep scan --config p/python --config rules/python --exclude='bar.py'" | ||
| }, | ||
| "result": "PASS", | ||
| "output": ["<ResourceDescriptor(semgrep_output.txt)>"] |
There was a problem hiding this comment.
To my point above, this, if not the subject, could point to the attestation that records the tool run's output!
| ```jsonc | ||
| { | ||
| "_type": "https://in-toto.io/Statement/v1", | ||
| "subject": [{ | ||
| "name": "foo", | ||
| "digest": { "sha256": "78ec328..." } | ||
| }], | ||
| "predicateType": "https://in-toto.io/attestation/tool-assessment/v0.1", | ||
| "predicate": { | ||
| "tool": { | ||
| "name": "Trufflehog", | ||
| "type": "Secrets Scanning", | ||
| "uri": "pkg:github/trufflesecurity/trufflehog@466da4b", | ||
| "version": "3.90.8" | ||
| }, | ||
| "config": { | ||
| "profiles": [ | ||
| { | ||
| "profile": "Custom", | ||
| "uri": "https://example.com/trufflehog_config.yml", | ||
| "last_updated": "2025-06-04T19:25:00Z" | ||
| }], | ||
| "exclusions": ["excluded_files.txt"], | ||
| "files": [ | ||
| "<ResourceDescriptor(trufflehog_config.yml)>", | ||
| "<ResourceDescriptor(excluded_files.txt)>" | ||
| ], | ||
| "full_command": "trufflehog --config=trugglehog_config.yml --no-update git file://. --exclude-paths='excluded_files.txt' --json > th.json" | ||
| }, | ||
| "result": "PASS", | ||
| "output": ["<ResourceDescriptor(th.json)>"] |
There was a problem hiding this comment.
Would an example policy operating over this attestation check the right version, configs, etc. are used?
There was a problem hiding this comment.
I'd say it's up to how controlling a policy writer wants to be.
An assessor could be as permissive as .tool.type == "Secrets Scaning" and .result == "PASS"
or as stringent as .tool.uri is github/trufflesecurity/trufflehog and .config.profile.uri == "tough_config.yml".
As long as the predicate has enough of the relevant properties.
marcelamelara
left a comment
marcelamelara
left a comment
There was a problem hiding this comment.
@JosephKyser Thanks so much for submitting this new predicate proposal and for your patience in the long delay in getting this reviewed! I overall see the utility of this predicate, but think a few things can be fine-tuned before we merge this spec.
|
|
||
| ## Purpose | ||
|
|
||
| **The tool assessment attestation references the results and metadata associated with tools used to assess software before or after its creation.** It's primary purpose is to provide an immutable attestation of tool assessment of a software so that it can be bundled with its provenance. This enables to mapping of build provenances to assessments on a per build basis and can reflect entire DevSecOps pipeline processes. |
There was a problem hiding this comment.
+1
I would also like to see a bit more specificity for what "bundled"/"mapped" to provenance means. in-toto for now mostly assumes that related attestations will be shipped together in jsonl format, and we have predicates that allow for explicit references of one attestation from another (e.g., evidence field in SCAI). I bring this up to make sure we're being clear about how the tool assessment predicate would be used with respect to other predicate types.
|
|
||
| **The tool assessment attestation references the results and metadata associated with tools used to assess software before or after its creation.** It's primary purpose is to provide an immutable attestation of tool assessment of a software so that it can be bundled with its provenance. This enables to mapping of build provenances to assessments on a per build basis and can reflect entire DevSecOps pipeline processes. | ||
|
|
||
| However, this attestation can be used to describe the assessment of any target using any tool for any purpose and is not restricted to pipelines. |
There was a problem hiding this comment.
I don't think this clarification is needed if the build provenance description is moved to the Use Cases section.
|
|
||
| Prior existing predicates still have their own important use-cases. This predicate type does not aim to replace them but to provide a specification flexible enough to use for any type of tool. | ||
|
|
||
| ## Use Cases |
There was a problem hiding this comment.
I wonder if SAST scanning could be another interesting use case of this predicate.
| - The [cyclonedx](cyclonedx.md) and [spdx](spdx.md) predicate types describe SBOM standards. | ||
| - The [test result](test-result.md) predicate type describes test running tests in the software supply chain. | ||
| - The [vulnerabilities](vulns_02.md) predicate type describes the results of a vulnerability scan. This predicate closely resembles the type of information desired to be captured by the tool assessment attestation but is too narrowly scoped to producers of vulnerability information. | ||
| - The [SCAI](scai.md) predicate type captures functional attribute and integrity information about software and its supply chain. It is the closest predicate for this use-case but fails to cleanly map a result to its tooling while providing appropriate metadata on the tooling or the policy requiring its execution. The tool assessment attestation would serve well as an attribute predicate in the SCAI framework. |
There was a problem hiding this comment.
This makes sense to me, especially based on what @adityasaky also mentioned below re: tool assessment providing the details that even predicates like test results do not.
|
|
||
| ### Policy as Code Enabling via Attachment to Build Artifacts | ||
|
|
||
| Tool assessment attestations bundled with container build provenance can enable policy-as-code enforcement of containers or software on IT systems. |
There was a problem hiding this comment.
I'm not entirely sure I follow this brief example.
|
|
||
| **tool.uri, required** string (ResourceURI) | ||
|
|
||
| >> URI indicating the identity of the source of the tool. |
There was a problem hiding this comment.
Is there an assumption about the tool's immutability? If so, I think a URI may not be a sufficient "identifier" for a tool.
|
|
||
| **config.exclusions, optional** string list | ||
|
|
||
| >> List of deviations from the profile, such as rule IDs, file names, ignores, etc. |
There was a problem hiding this comment.
As I noted above, I think a simple sub-schema is totally acceptable for fields if they can offer clarity on the semantics of the fields. Other general predicates like SLSA Provenance and SCAI have sub-schemas in one or more fields, so there is certainly precedent.
|
|
||
| **result, required** string | ||
|
|
||
| > Result of the tool execution. Usually `PASS` or `FAIL`. |
There was a problem hiding this comment.
Can we really assume that all tool assessments will have a single result value that represents the assessment? What is this field capturing that output wouldn't?
|
|
||
| **output, required** ResourceDescriptor list | ||
|
|
||
| > Artifacts associated with the result of the execution of the tool. |
There was a problem hiding this comment.
Just based on the name output my expectation would be for this field to either contain something like a log of the tool's execution. To avoid any confusion, I would add a clarification at the top of the Field section clarifying that the subject field for this predicate contains the assessed artifacts (e.g., container images).
|
|
||
| >> Timestamp of the last update of the profile | ||
|
|
||
| **Profile.annoations, optional** object |
There was a problem hiding this comment.
small typo
| **Profile.annoations, optional** object | |
| **Profile.annotations, optional** object |
3 participants
This PR ads the predicate type
tool assessment.The tool assessment predicate aims to provide a general template to attest to the results of tools ran on software while also providing configuration data in order to replicate those results, leading to open and auditable security artifacts.
Currently the attestation bundle alludes to packaging different points in the software supply chain to be grouped together, however only has predicate types for sbom, vulnerability scans, test results, etc. Other type of software supply chain artifacts like SAST, DAST, Secret Scanning, STIG, etc. are missing. Instead of exhaustively creating a predicate type for every use case, tool assessment allows the ability to cover every type of tool.
If nothing else, it allows for these non-covered tool types to be represented until more narrowly defined predicates scoped for each respective tool type is defined.