Ease up importer CPU: configurable aggressiveness + daily-gate non-fresh Graph imports + stagger WebJobs

[sambetts]

Closes #161.

What & why

A customer reported the importer WebJobs cause frequent 100% CPU spikes on the shared App Service plan, tripping their admin alerts. This PR makes the importers less aggressive on CPU — on both the App Service tier and the SQL Server tier (CPU/DTU) — without losing freshness on the data that needs it most (audit events and web-traffic "hits").

Root cause

• Both importer WebJobs (Office365ActivityImporter = audit + Graph, AppInsightsImporter = hits) plus the Web app share one App Service plan (B1 = 1 vCPU default, B2 in prod), AlwaysOn.
• Each WebJob loops every 10 min via a hardcoded ConsoleApp.WebjobWait (Thread.Sleep(600000)), and the two jobs are not staggered — they can peak together on the shared core.
• The audit full-report load fans out to a hardcoded 20 parallel threads (ParallelListProcessor).
• Every InsertBatch importer (audit-event persistence, Copilot, Power Platform, App Insights hits) commits its staged rows through that same hardcoded 20-thread fan-out — the SQL-commit burst driver behind the SQL CPU/DTU spikes.
• GraphUsersMetadata, GraphUserApps and GraphTeams run every 10-min cycle with no time gate, even though that data barely changes intraday (only GraphUsageReports was already daily-gated). At ~200k users this is a major, pointless CPU burst 144x/day.

What stays fresh

Audit events (ActivityLog / Copilot) and hits keep the 10-min cycle cadence at every level — only burst concurrency (and, at Gentle, the cycle pause) change. Everything else can be eased.

Changes

New admin-tunable config (all optional AppSettings, preset-driven, explicit per-knob override — same read-with-default pattern as MaxSummaryFetchConcurrency; not installer-written)

Setting	Default	Purpose
ImportAggressiveness	Balanced	Preset High | Balanced | Gentle. Drives the knobs below.
MaxAuditReportLoadConcurrency	preset (High 20 / Balanced 8 / Gentle 3)	Caps audit full-load threads (was hardcoded 20). Eases App Service CPU.
MaxSqlCommitConcurrency	preset (High 20 / Balanced 8 / Gentle 3)	Caps the InsertBatch SQL-commit thread fan-out (was hardcoded 20). Eases SQL Server CPU/DTU.
ImportCyclePauseMinutes	preset (High/Balanced 10 / Gentle 20)	Inter-cycle wait (was hardcoded 10).
GraphMetadataImportIntervalHours	24 (High 0 = every cycle)	Daily-gates user metadata / user apps.
GraphTeamsImportIntervalHours	24 (High 0 = every cycle)	Daily-gates the Teams crawl (its own knob, can be set more frequent without un-gating the static imports).
ImportStartStaggerMinutes	pause/2	One-off offset so the hits WebJob does not peak with the activity WebJob.

Explicit per-knob AppSettings override the preset; invalid / zero / negative values fall back to the preset.

Code

• Common/Entities/Config/AppConfig.cs — ImportAggressivenessLevel enum + the 7 keys + preset/override parsing.
• Common/DataUtils/ParallelListProcessor.cs — optional maxConcurrentThreads (default 20; existing callers unchanged).
• Common/DataUtils/Sql/Inserts/InsertBatchConcurrency.cs (new) — process-wide SQL-commit concurrency lever (default 20, clamps < 1 to 1).
• Common/DataUtils/Sql/Inserts/InsertBatch.cs — SaveToStagingTable fans out at InsertBatchConcurrency.MaxConcurrentThreads (the single SQL-commit choke point).
• Common/DataUtils/ConsoleApp.cs — WebjobWait(telemetry, minutes) (configurable cycle pause).
• WebJob.Office365ActivityImporter.Engine/ActivityAPI/ActivityImporter.cs — audit load uses MaxAuditReportLoadConcurrency.
• WebJob.Office365ActivityImporter.Engine/Graph/GraphImporter.cs + ImportLastRunStore.cs (new) — Redis-backed per-section last-run cadence gate for the non-fresh Graph imports (with an in-memory fallback when Redis is absent, and a ForceGraphMetadataImport one-off bypass).
• WebJob.Office365ActivityImporter/Program.cs + WebJob.AppInsightsImporter/Program.cs — set InsertBatchConcurrency.MaxConcurrentThreads from config at startup; configurable cycle pause; hits-WebJob startup stagger; startup log line reporting the active level and derived knobs (audit-load + SQL-commit threads, pause, gate interval).

Behaviour change on upgrade

Default preset Balanced (approved with @sambetts), so existing installs ease up automatically:

• audit-load concurrency 20 -> 8, SQL-commit concurrency 20 -> 8,
• the non-fresh Graph imports (user metadata / apps / Teams) move from every-cycle to once-a-day,
• audit events and hits keep the 10-min cadence (only burst concurrency lowered).

Admins wanting the old behaviour set ImportAggressiveness=High; the complaining customer can set Gentle.

Load test (the numbers behind the change)

Measured with a non-interactive LoadTestSuite harness added under Tests.FakeDataGen (reproducible benchmark). Each import area imports 100k items per setting, 3 reps, median reported, sampling both importer-process CPU and LocalDB sqlservr CPU. Caveat: 8-core dev box + LocalDB (no DTU cap) != prod 1-vCPU + Azure SQL, so absolute numbers do not transfer but relative comparisons across settings do. CPU-s = work; peak % = burst (100% = one core). Full per-rep tables are in the pinned comment on this PR.

SQL-commit fan-out, isolated (InsertBatch, the new lever) — insert-bound, strict win:

Setting (cap)	wall s	SQL CPU-s	SQL peak %
High (20)	12.5	27.0	401%
Balanced (8)	9.6	21.8 (-19%)	430%
Gentle (3)	14.5	20.6 (-24%)	310% (-23%)

Real importers — SQL CPU per 100k at the current default (cap 20), showing where the DTU goes:

Area	SQL CPU-s / 100k	SQL peak %	Bound by / lever
Hits (App Insights)	~143	~550%	InsertBatch + heavy 13-lookup MERGE — cap cuts peak (-35% at Gentle)
Sent email	~113	~143%	EF SaveChanges — off lever, sustained, no spike
Copilot	~65	~401%	InsertBatch + MERGE — cap cuts peak (-36% at Gentle)
Power Platform	~44	~482%	InsertBatch + MERGE — cap cuts peak (-32% at Gentle)
Audit-event persistence (~plain InsertBatch)	~27	~401%	InsertBatch INSERT — cap cuts work -19% @ 8
Usage reports (SqlBulkCopy / TVP)	~2	~114%	off lever, cheap
Audit load (download/parse)	~0 (CPU on App Service)	—	MaxAuditReportLoadConcurrency

Audit load fan-out (App Service CPU; in-memory fakes, so a lower bound): peak ~428% (cap 20) -> ~300-373% (8/3); 1-vCPU emulation pins at ~108-126% regardless, i.e. the cap bounds burst width.

Takeaways:

• MaxSqlCommitConcurrency=8 is a strict win on insert-bound commits (audit-event persistence, hits staging): -19% SQL CPU and -23% wall vs 20.
• On merge-bound importers (hits — the heaviest at ~143 SQL-CPU-s/100k, Copilot, Power Platform) the cap mainly cuts the SQL peak ~30-35% — i.e. it tames the spike that trips the alert — while the residual CPU is in the MERGE statements.
• Follow-up (not in this PR): to cut the merge-bound work (esp. hits), tune the merge SQL (OPTION (MAXDOP n) / smaller batches) and lean on ImportStartStaggerMinutes so hits + audit do not merge simultaneously. Sent Email (EF) is a heavy-but-non-spiky sustained load and a future EF->bulk candidate; Usage (bulk) is already cheap.

Testing

• Full solution Release build (VS18 MSBuild): 0 errors / 0 warnings.
• 58 relevant unit tests pass (Debug):
  • AppConfigDefaultsTests — 36 tests: preset defaults for every knob (incl. MaxSqlCommitConcurrency High/Balanced/Gentle = 20/8/3), case-insensitive parse, invalid/empty/zero/negative -> preset, explicit per-knob overrides, 0-disables gate.
  • ImportCadenceTests — 8 tests: the Redis-backed daily-gate for the non-fresh Graph imports.
  • DataUtilsTests — InsertBatchConcurrency_DefaultAndClamp (default 20, clamps <1 to 1) + ParallelListProcessor tests (incl. chunk-exception propagation) pass with the new optional param.
  • InsertBatchOverWidthTests + StagingColumnSqlTypeOverrideTests — the InsertBatch commit path (the line this PR changed) still skips over-width rows and emits correct staging types.
• Note: the 2 GraphUsageReportImportTests are pre-existing live-Graph integration tests that require real tenant credentials (AADSTS90002: Tenant not found in CI/local without secrets); they are unrelated to these changes.

Database changes

None. All new settings are optional appSettings; no schema/migration changes. (Labelled migration needed because it is a config change — new App Settings — per repo convention.)

Docs

Wiki updated: [Configuration Reference] gains a "Reducing importer CPU usage (aggressiveness)" section covering all knobs incl. MaxSqlCommitConcurrency, plus a cross-link from [Monitoring System Performance].

[sambetts]

Pushed �cafe80 addressing review feedback (self + code-review agent):

• Critical: the daily-gate was dead code. RunGraphSectionIfDueAsync was never called - the three sections still ran inline every cycle, so the headline feature did nothing. Now wired up (user-metadata/apps via GraphMetadataImportIntervalHours, Teams via its own interval).
• In-memory fallback (per @sambetts): new IImportLastRunStore (Redis + in-memory impls) created once in Program.cs and threaded through ProgramTasks -> GraphImporter, so the gate works without Redis (resets on restart) and persists across cycles with it.
• Fail-open: Redis read/write errors log a warning and let the import run (these imports previously had no Redis dependency).
• ForceGraphMetadataImport flag for a one-off re-import (mirrors ForceUsageReportsImport).
• ImportAggressiveness=High now truly restores legacy - non-fresh interval is preset-derived (High=0, Balanced/Gentle=24h).
• Teams split out into GraphTeamsImportIntervalHours (fresher data, own knob).
• Gate logic extracted to ImportCadenceGate.ShouldRun (unit tested) + in-memory store tests + new AppConfig tests. Release build clean; 46 targeted tests pass.

Wiki: added a Forcing a re-import (clearing the last-run flags) section (force flag + redis-cli DEL of the unprefixed keys) and the new knobs.

[sambetts]

Load test — importer aggressiveness (PR #162 / issue #161): App-Service CPU and SQL CPU/DTU

Branch under test: feature/importer-cpu-ease-up (PR #162). Harness = a new non-interactive
LoadTestSuite in Tests.FakeDataGen (local file changes only, not committed). Each area imports
100k items at each setting, 3 reps, median reported.

Now covers all import jobs — audit-load, the SQL-commit choke point, Copilot, hits, Power
Platform, Sent Email, Usage. (This supersedes the earlier note that hits was "insert-bound";
measured, hits is the heaviest SQL importer and is merge-bound.)

TL;DR

1. PR Ease up importer CPU: configurable aggressiveness + daily-gate non-fresh Graph imports + stagger WebJobs #162 throttles the right thing for App-Service CPU (the audit load fan-out) but leaves
   the SQL-side lever untouched: every InsertBatch importer still commits through
   ParallelListProcessor(20) (hardcoded). That is the SQL CPU/DTU burst driver.
2. The big SQL-CPU/spike offender is the hits importer — ~143 SQL-CPU-s per 100k, peak ~550%.
   Sent Email is the [Analytics] Fix transaction usage and nullable columns #2 SQL consumer (~113s) but via chatty EF — high sustained CPU, low peak
   (~143%), no spike, and off the cap lever entirely.
3. The commit cap helps insert-bound commits (isolated InsertBatch: 20→8 = −19% SQL CPU and
   −23% wall). For the merge-bound importers (hits, Copilot, Power Platform) it barely moves SQL
   work but cuts the SQL peak ~30–35% at Gentle — i.e. it tames the spike that trips the alert.
4. Recommendation: (a) add a MaxSqlCommitConcurrency knob (preset 20/8/3) into InsertBatch
   (Balanced=8 = strict win for insert-bound staging + lower peak everywhere); and (b) the real DTU
   cost is the merge SQL (esp. hits) — add OPTION (MAXDOP n) / smaller batches and use the WebJob
   stagger so hits + audit don't merge together. Sent Email (EF) and Usage (bulk) are off-lever.

Method & caveats

• Importer-process CPU and LocalDB sqlservr CPU sampled every 100 ms. CPU-seconds = work;
  peak %CPU = burst, 100% = one logical core.
• 8-core dev box + LocalDB (no DTU cap) ≠ prod 1-vCPU B1 + Azure SQL — absolute numbers don't
  transfer, relative comparisons do.
• Audit area uses in-memory fakes (no network/JSON parse) ⇒ its CPU gains are a lower bound.
• Copilot/hits/PP merge into growing tables; reps use rotated setting order so per-setting medians
  compare like sizes (absolute SQL-CPU is inflated by repetition).
• Current/High ≡ current dev/stable for per-import cost (caps 20/20); they differ only in
  cadence/gating (a frequency change handled separately in the PR).

Results — median of 3 reps, 100k items per area

1) Audit load fan-out (MaxAuditReportLoadConcurrency) — the lever PR #162 adds (importer CPU; ~no SQL)

Setting (cap)	wall s	importer CPU-s	importer peak %CPU
Current/High (20)	1.78	2.2	428%
Balanced (8)	1.58	2.1	300%
Gentle (3)	1.59	2.1	373%

1-vCPU emulation (pinned to 1 core): peak ≈108–126%, wall ≈1.9–2.0 s at every cap. Total importer
CPU ~flat (same work); the cap bounds the burst width.

2) SQL commit fan-out, isolated (InsertBatch, empty merge) — the lever PR #162 MISSES

Setting (cap)	wall s	SQL CPU-s	SQL peak %CPU	importer CPU-s
Current/High (20)	12.48	27.0	401%	31.4
Balanced (8)	9.60	21.8 (−19%)	430%	24.1
Gentle (3)	14.47	20.6 (−24%)	310% (−23%)	43.1

Insert-bound ⇒ the cap is a strict win at 8: −19% SQL CPU and −23% wall vs 20.

3) Real merge-bound importers (InsertBatch staging + merge scripts) — cap cuts the peak, not the work

3a) Copilot (CommitAllChanges: 3 staging + merges)

Setting (cap)	wall s	SQL CPU-s	SQL peak %CPU
Current/High (20)	52.0	65.0	401%
Balanced (8)	56.8	65.0	400%
Gentle (3)	63.1	59.5	258% (−36%)

3b) Hits (PageViewCollection.SaveToSQL: stage + 13-lookup merge) — heaviest SQL area

Setting (cap)	wall s	SQL CPU-s	SQL peak %CPU
Current/High (20)	50.6	143.0	550%
Balanced (8)	45.6	137.2 (−4%)	543%
Gentle (3)	62.0	125.5 (−12%)	358% (−35%)

3c) Power Platform (CommitAllChanges: 6 staging + merges)

Setting (cap)	wall s	SQL CPU-s	SQL peak %CPU
Current/High (20)	29.8	44.3	482%
Balanced (8)	33.5	43.1	458%
Gentle (3)	38.2	43.8	327% (−32%)

All three: SQL-CPU work ~flat across caps (merge dominates); peak drops ~30–36% at Gentle; wall
increases as the cap drops. So the cap is a spike lever for these, not a work-reduction lever.

4) Sent Email — real importer (SentEmailImporter, EF SaveChanges) — off the lever

Setting	wall s	SQL CPU-s	SQL peak %CPU	importer CPU-s
(cap-independent)	125.6	113.0	143%	17.1

Chatty EF (per-chunk SaveChanges + IN-clause existing-key lookups) ⇒ high sustained SQL CPU but
no spike (peak ~1.4 cores). Not on the InsertBatch ParallelListProcessor, so the aggressiveness cap
can't change it. Worth its own optimisation later, but it isn't a CPU spike source.

5) Usage activity — SqlBulkCopy 100k rows — off the lever, cheapest

Setting	wall s	SQL CPU-s	SQL peak %CPU
(cap-independent)	5.3	2.0	114%

SqlBulkCopy (and the prod TVP+upsert proc) are very efficient — the least SQL CPU of any area. Off-lever.

Cross-area SQL CPU per 100k (at current cap 20) — the DTU picture

Area	SQL CPU-s / 100k	SQL peak	Lever / bound by
Hits	~143	~550%	InsertBatch + heavy MERGE — cap = peak only
Sent email	~113	~143%	EF SaveChanges — off lever, sustained, no spike
Copilot	~65	~401%	InsertBatch + MERGE — cap = peak only
Power Platform	~44	~482%	InsertBatch + MERGE — cap = peak only
Audit-event persistence ≈ plain InsertBatch	~27	~401%	InsertBatch INSERT — cap = work −19% @8
Usage (SqlBulkCopy)	~2	~114%	bulk/TVP — off lever, cheap
Audit load (download/parse)	~0 (CPU on App Service)	—	audit-load cap (the PR's lever)

Per-area → which lever applies (all measured)

Import job	Persistence	Audit-load cap (PR #162)	SQL-commit cap (InsertBatch)	Dominant SQL cost
Audit events	EFInsertBatch + merge	✅ throttled	⚠️ not throttled — would benefit (insert-bound)	insert
Hits	EFInsertBatch + heavy merge	—	⚠️ peak only (merge-bound)	MERGE (biggest)
Copilot	3× InsertBatch + merges	—	⚠️ peak only (merge-bound)	merge
Power Platform	6× InsertBatch + merges	—	⚠️ peak only (merge-bound)	merge
Sent emails	EF SaveChanges	—	❌ off lever	EF (sustained, low peak)
Usage reports	TVP + upsert proc / SqlBulkCopy	—	❌ off lever	bulk (cheap)

Recommendation for PR #162 (to make it "also about SQL CPU/DTU")

1. Add MaxSqlCommitConcurrency (preset 20/8/3, explicit-override, same pattern as
   MaxAuditReportLoadConcurrency) → InsertBatch.SaveToStagingTable's ParallelListProcessor
   (single choke point; the harness branch already prototypes
   InsertBatchConcurrency.MaxConcurrentThreads). Default Balanced = 8 ⇒ less SQL CPU and
   faster for the insert-bound audit/hits staging, and a lower commit peak on every InsertBatch
   importer (hits/Copilot/PP) — directly the "100% spike" the customer alerts on.
2. Tune the merges for the real DTU win — hits (~143 SQL-CPU-s/100k) and the other merge-bound
   importers spend the CPU in the MERGE, which the insert cap doesn't touch. Consider OPTION (MAXDOP n)
   on the hits/Copilot/PP merge scripts and/or smaller commit batches, and rely on
   ImportStartStaggerMinutes so hits + audit don't merge at the same instant on the shared SQL tier.
3. Off-lever notes: Sent Email (EF) is a heavy but non-spiky sustained SQL load — out of scope
   for the cap but a candidate for a future EF→bulk rewrite; Usage (bulk) is already cheap.
4. Keep the PR framing honest: as-is it eases App-Service CPU; the SQL CPU/DTU spike — led by
   hits — needs the commit-side cap (peak) and merge tuning (work).

Per-rep CSVs: loadtest-reps.csv (audit/InsertBatch/Copilot), loadtest-hits.csv (hits),
loadtest-extra.csv (Power Platform / Sent Email / Usage).