[Enhancement] Record IVM source version/timestamp ranges in MV task run EXTRA_MESSAGE by Youngwb · Pull Request #74605 · StarRocks/starrocks

Youngwb · 2026-06-10T06:58:20Z

Why I'm doing:

An incremental (IVM) materialized view refresh consumes a specific snapshot range from each base table, but that range is currently only visible in FE logs. Exposing it on the task run makes every refresh auditable via information_schema.task_runs.EXTRA_MESSAGE ("which slice of the source did this refresh consume?"), and it is the data source for the IMV_SOURCE_* columns of the planned MV refresh-jobs system table.

What I'm doing:

Record two new JSON sub-fields in the MV task run extra message for every IVM refresh, right after MVIVMRefreshProcessor stages the per-table TVR deltas:

imvSourceVersionRange: {"catalog.db.tbl": {"start": "100", "end": "128"}} — the TVR version (snapshot id) range consumed from each base table. TvrVersion MIN/MAX sentinels are rendered as "MIN"/"MAX" (readable, instead of leaking Long.MIN_VALUE/Long.MAX_VALUE literals).
imvSourceTimestampRange: same shape, values are the commit times of those versions in epoch millis. Resolution is delegated to Table.getVersionCommitTimeMillis(version) — a polymorphic seam whose base implementation returns empty and which each table format overrides (Iceberg resolves the snapshot's commit time); endpoints that cannot be resolved (sentinel versions, expired/unknown snapshots, formats with no version commit time) are omitted, so a table degrades to {} instead of failing the refresh. This keeps MVIVMRefreshProcessor format-agnostic, so other IVM-capable table formats only need to override that one method.

Recording goes through the existing best-effort updateTaskRunStatus() helper (null-guarded, exception-swallowing), so it can never fail a refresh. Both setters cap map entries with MvUtils.shrinkToSize, mirroring pinnedSnapshotIdMap. PCT runs leave both fields as {}.

Example after an incremental refresh:

SELECT get_json_string(EXTRA_MESSAGE, '$.imvSourceVersionRange')
FROM information_schema.task_runs ORDER BY FINISH_TIME DESC LIMIT 1;
-- {"iceberg_cat.mv_db.t1":{"start":"4912319873255147742","end":"6987413294356100802"}}

Tests:

FE UT: MVTaskRunExtraMessageTest (shrink-to-size + JSON serialization of the new fields), IVMBasedMvRefreshProcessorIcebergTest#testImvSourceVersionRangeRecordedOnExtraMessage (range values + degradation to {} for unresolvable snapshots, plus the empty base-Table resolution), #testImvSourceTimestampRangeRecordedOnExtraMessage (commit times against real Iceberg snapshots on the mock catalog), and #testImvSourceRangesNotRecordedOnPctFallback (no stale ranges when hybrid falls back to PCT).
SQL: test_ivm_task_run_source_range — PCT baseline run records nothing; incremental run records both ranges.

Fixes #issue: N/A — observability enhancement, no tracking issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
- This pr needs auto generate documentation
This is a backport pr

Bugfix cherry-pick branch check:

I have checked the version labels which the pr will be auto-backported to the target branch
- 4.1
- 4.0
- 3.5

CelerData-Reviewer · 2026-06-10T06:59:08Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ba72f7a3fa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Youngwb · 2026-06-10T08:48:23Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d04e00752c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Youngwb · 2026-06-10T09:25:43Z

@codex review

chatgpt-codex-connector · 2026-06-10T09:31:03Z

Codex Review: Didn't find any major issues. More of your lovely PRs please.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Youngwb · 2026-06-10T09:36:49Z

rerun

Youngwb · 2026-06-10T09:40:38Z

Re-run the workflow

…un EXTRA_MESSAGE Record, for each IVM (incremental) MV refresh, the TVR version range consumed from every base table and the matching Iceberg snapshot commit times into information_schema.task_runs.EXTRA_MESSAGE: - imvSourceVersionRange: {"catalog.db.tbl": {"start": "100", "end": "128"}} - imvSourceTimestampRange: same shape, values are snapshot commit times in epoch millis; endpoints that cannot be resolved (MIN/MAX sentinel versions, expired/unknown snapshots, metadata errors) are omitted, so a table degrades to {} instead of failing the refresh. The ranges are recorded right after MVIVMRefreshProcessor stages the per-table TVR deltas, through the existing best-effort updateTaskRunStatus() path, so recording can never fail a refresh. PCT runs leave both fields empty. Both setters cap entries via MvUtils.shrinkToSize like the other EXTRA_MESSAGE maps. These fields feed the IMV_SOURCE_* columns of the planned MV refresh jobs system table. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Youngwb <yangwenbo_mailbox@163.com>

github-actions · 2026-06-11T13:38:06Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-06-11T13:39:00Z

[FE Incremental Coverage Report]

✅ pass : 47 / 49 (95.92%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/connector/ConnectorMetadata.java	0	1	00.00%	[203]
🔵	com/starrocks/scheduler/mv/ivm/MVIVMRefreshProcessor.java	27	28	96.43%	[179]
🔵	com/starrocks/server/MetadataMgr.java	2	2	100.00%	[]
🔵	com/starrocks/scheduler/persist/MVTaskRunExtraMessage.java	10	10	100.00%	[]
🔵	com/starrocks/connector/iceberg/IcebergMetadata.java	2	2	100.00%	[]
🔵	com/starrocks/connector/CatalogConnectorMetadata.java	4	4	100.00%	[]
🔵	com/starrocks/connector/unified/UnifiedMetadata.java	2	2	100.00%	[]

github-actions · 2026-06-11T13:39:42Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

…un EXTRA_MESSAGE (StarRocks#74605) Signed-off-by: Youngwb <yangwenbo_mailbox@163.com> Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Oliver Layer <o.layer@celonis.com>

…un EXTRA_MESSAGE (StarRocks#74605) Signed-off-by: Youngwb <yangwenbo_mailbox@163.com> Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit 95d2d9c) Signed-off-by: Youngwb <yangwenbo_mailbox@163.com>

wanpengfei-git added the META-REVIEW label Jun 10, 2026

wanpengfei-git requested a review from a team June 10, 2026 06:58

mergify Bot assigned Youngwb Jun 10, 2026

github-actions Bot requested a review from stephen-shelby June 10, 2026 07:00

chatgpt-codex-connector Bot reviewed Jun 10, 2026

View reviewed changes

Comment thread test/sql/test_materialized_view_ivm/T/test_ivm_task_run_source_range

Youngwb force-pushed the mv-obs-ivm-source-range branch 2 times, most recently from 198cffe to d04e007 Compare June 10, 2026 07:33

github-actions Bot requested a review from HangyuanLiu June 10, 2026 08:09

chatgpt-codex-connector Bot reviewed Jun 10, 2026

View reviewed changes

Comment thread fe/fe-core/src/main/java/com/starrocks/scheduler/mv/ivm/MVIVMRefreshProcessor.java Outdated

Youngwb force-pushed the mv-obs-ivm-source-range branch from d04e007 to 8108b75 Compare June 10, 2026 09:23

Youngwb force-pushed the mv-obs-ivm-source-range branch 2 times, most recently from 41c4090 to 87c8894 Compare June 11, 2026 06:59

stephen-shelby reviewed Jun 11, 2026

View reviewed changes

Comment thread fe/fe-core/src/main/java/com/starrocks/catalog/Table.java Outdated

Youngwb force-pushed the mv-obs-ivm-source-range branch 3 times, most recently from eb31393 to 8c261bb Compare June 11, 2026 08:11

Youngwb force-pushed the mv-obs-ivm-source-range branch from 8c261bb to 9f8561d Compare June 11, 2026 12:21

stephen-shelby approved these changes Jun 12, 2026

View reviewed changes

HangyuanLiu approved these changes Jun 12, 2026

View reviewed changes

Youngwb enabled auto-merge (squash) June 12, 2026 03:25

Youngwb merged commit 73cc4db into StarRocks:main Jun 12, 2026
61 checks passed

Youngwb deleted the mv-obs-ivm-source-range branch June 12, 2026 03:28

Youngwb mentioned this pull request Jul 3, 2026

[Enhancement] Backport MV observability columns, refresh_jobs table & metrics (backport #74342) #75795

Merged

24 tasks

Navigation Menu

Uh oh!

[Enhancement] Record IVM source version/timestamp ranges in MV task run EXTRA_MESSAGE - #74605

[Enhancement] Record IVM source version/timestamp ranges in MV task run EXTRA_MESSAGE#74605
Youngwb merged 1 commit into
StarRocks:mainfrom
Youngwb:mv-obs-ivm-source-range

Youngwb commented Jun 10, 2026 •

edited

Loading

Uh oh!

CelerData-Reviewer commented Jun 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Youngwb commented Jun 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Youngwb commented Jun 10, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 10, 2026

Uh oh!

Youngwb commented Jun 10, 2026

Uh oh!

Youngwb commented Jun 10, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

Youngwb commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

CelerData-Reviewer commented Jun 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Youngwb commented Jun 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Youngwb commented Jun 10, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 10, 2026

Uh oh!

Youngwb commented Jun 10, 2026

Uh oh!

Youngwb commented Jun 10, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 11, 2026

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jun 11, 2026

[FE Incremental Coverage Report]

file detail

Uh oh!

github-actions Bot commented Jun 11, 2026

[BE Incremental Coverage Report]

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Youngwb commented Jun 10, 2026 •

edited

Loading