Description:
Originally I thought there was an issue on our tooling side that was create very large numbers of GraphQL requests and we were (and still are) planning a control for that (see #921).
However in discussions internally we think this is actually a bug in the implementation on the GhostWriter side around the ordering and severity_id fields in conjunction with some form of change monitoring functionality that makes sure it doesn't have two finding items that are sharing the same ordering position in a report.
We are a little short on time but we wanted to try and confirm this so we made requests to our AI overlords and it appears to agree and has come up with something interesting.
The below comes from an LLM that was given the codebase, a request to identify any situations that would be a bug and a description of the symptoms:
The most likely cause is in the collaborative editing save path rather than the direct “attach finding” action. The collaborative editor server saves documents in javascript/src/collab_server/index.ts, inside onStoreDocument(). After each save, it writes save-status metadata back into the same Yjs document:
tx.doc.get("serverInfo", Y.Map).set("saveError", false);
That is risky because the server is mutating the collaborative document immediately after storing it. If Hocuspocus treats that mutation as another document update, it can schedule another store cycle. In practice, a single user edit could repeatedly trigger onStoreDocument(), causing repeated GraphQL mutations even when the user has made no further changes.
For report findings, each collab save runs the SET_REPORT_FINDING_LINK mutation in javascript/src/collab_server/handlers/report_finding_link.ts. That mutation performs both:
update_reportedFinding_by_pk(...)
setTags(...)
So one save cycle is already more than one operation. If the Yjs status update causes a save loop, the system can generate a large number of Hasura mutations for a single edited finding.
There is also an amplifier specific to report findings. The collab save mutation always includes fields such as severityId in the _set payload, even when the value has not changed. The Hasura metadata for reporting_reportfindinglink has an event trigger watching updates to severity_id and position, which calls the Django GraphqlReportFindingChangeEvent handler. That handler recalculates finding positions. It usually avoids further position updates if nothing changed, but the webhook invocation and associated database reads still happen. This means repeated unchanged saves can still produce extra Hasura event traffic.
Are you intending to fix this bug?
Kind of... I am about to submit a PR, but it came from AI and has not been tested (owing to time constraints). Please therefore treat with appropriate level of skepticism.
Component(s) Affected:
- GraphQL / Hasura interface
- API
- PostgreSQL
- Database Backups get very large
Steps to Reproduce:
Pretty sure this is triggered by bulk uploading of findings in a single request to the GraphQL endpoint. Not 100% sure but as it is a non-standard interaction I would prefer not to be that specific.
Expected Behavior:
Just one transaction set throughout the stack.
Actual Behavior:
In some cases, this resulted in literally millions of transsactions within a 24 hour period.
Environment Information:
Ghostwriter: v6.3.6 (we were waiting to test the breaking changes for integration with our tooling)
Potential Solution (optional):
See upcoming PR
Description:
Originally I thought there was an issue on our tooling side that was create very large numbers of GraphQL requests and we were (and still are) planning a control for that (see #921).
However in discussions internally we think this is actually a bug in the implementation on the GhostWriter side around the ordering and severity_id fields in conjunction with some form of change monitoring functionality that makes sure it doesn't have two finding items that are sharing the same ordering position in a report.
We are a little short on time but we wanted to try and confirm this so we made requests to our AI overlords and it appears to agree and has come up with something interesting.
The below comes from an LLM that was given the codebase, a request to identify any situations that would be a bug and a description of the symptoms:
The most likely cause is in the collaborative editing save path rather than the direct “attach finding” action. The collaborative editor server saves documents in
javascript/src/collab_server/index.ts, insideonStoreDocument(). After each save, it writes save-status metadata back into the same Yjs document:That is risky because the server is mutating the collaborative document immediately after storing it. If Hocuspocus treats that mutation as another document update, it can schedule another store cycle. In practice, a single user edit could repeatedly trigger
onStoreDocument(), causing repeated GraphQL mutations even when the user has made no further changes.For report findings, each collab save runs the
SET_REPORT_FINDING_LINKmutation injavascript/src/collab_server/handlers/report_finding_link.ts. That mutation performs both:So one save cycle is already more than one operation. If the Yjs status update causes a save loop, the system can generate a large number of Hasura mutations for a single edited finding.
There is also an amplifier specific to report findings. The collab save mutation always includes fields such as
severityIdin the_setpayload, even when the value has not changed. The Hasura metadata forreporting_reportfindinglinkhas an event trigger watching updates toseverity_idand position, which calls the DjangoGraphqlReportFindingChangeEventhandler. That handler recalculates finding positions. It usually avoids further position updates if nothing changed, but the webhook invocation and associated database reads still happen. This means repeated unchanged saves can still produce extra Hasura event traffic.Are you intending to fix this bug?
Kind of... I am about to submit a PR, but it came from AI and has not been tested (owing to time constraints). Please therefore treat with appropriate level of skepticism.
Component(s) Affected:
Steps to Reproduce:
Pretty sure this is triggered by bulk uploading of findings in a single request to the GraphQL endpoint. Not 100% sure but as it is a non-standard interaction I would prefer not to be that specific.
Expected Behavior:
Just one transaction set throughout the stack.
Actual Behavior:
In some cases, this resulted in literally millions of transsactions within a 24 hour period.
Environment Information:
Ghostwriter: v6.3.6 (we were waiting to test the breaking changes for integration with our tooling)
Potential Solution (optional):
See upcoming PR