Skip to content

Commit 28c91f1

Browse files
paulalbert1paulalbert1
committed
feat(grantrecs): grant-history join + daily opportunity projection (#1218)
Step 3 (app) — extend the reverse funding matcher with grant history: deriveGrantSignals() attaches esiEligible / yearsSinceDegree / fundingStatus per researcher (funding via the canonical isFundingActive NCE-grace rule; ESI dated from the terminal research/clinical degree, not the latest credential). Adds the ESI blurb clause, a funding-status filter, funding + 'Also in their Grants for me' row badges, and two CSV columns. The cross-ref is a cheap MySQL-only topic-affinity top-N over the open-opportunity corpus (same status/deadline gate the forward matcher uses, so closed/past-due opps can't over-claim); validated against the real forward matcher in scripts/funding-crossref-compare.ts (100% top-10 agreement, past-due guard clean). Step 2 (cdk) — standalone daily etl:dynamodb schedule (06:30 UTC) so newly-published opportunities stop 404-ing while the nightly is blocked at etl:ed (#443). Creation-gated on a new opportunityProjectionScheduleEnabled flag (staging on, prod off), modeled on the curationBackup schedule: task -> retry -> catch -> state machine -> rule -> cadence alarm. Tests: +grant-signal/ESI/funding/CSV unit coverage; cdk staging rules 8->9 + projection schedule test + prod-absence test (130 cdk tests). Operator deploys: cdk deploy Sps-Etl-staging (Step 2) + app image roll (Step 3).
1 parent 280d3fb commit 28c91f1

12 files changed

Lines changed: 1129 additions & 10 deletions

File tree

‎app/api/opportunities/[opportunityId]/researchers/route.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ export async function GET(
6262
sort: sortRaw as ResearcherSort,
6363
stageLens,
6464
limit,
65+
crossRef: true,
6566
});
6667

6768
// View-model assembly for the redesigned screen: opportunity card fields, the

‎cdk/lib/config.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,23 @@ export interface SpsEnvConfig {
207207
* docs/curation-backup-runbook.md § Prod.
208208
*/
209209
readonly curationBackupScheduleEnabled: boolean;
210+
/**
211+
* Whether the standalone daily DynamoDB→MySQL projection (`etl:dynamodb`, the
212+
* step that mirrors ReciterAI's `reciterai` table into the `opportunity` +
213+
* scholar tables) runs on its OWN schedule, separate from the nightly cadence.
214+
*
215+
* `etl:dynamodb` is already a nightly step, but it sits AFTER `etl:ed`, which
216+
* is blocked by the on-prem-routing gap (#443) — so the nightly aborts before
217+
* reaching it and freshly-published opportunities 404 until someone re-projects
218+
* by hand. This standalone schedule keeps the funding-matcher corpus fresh
219+
* independent of the blocked nightly (#1218). A DEDICATED, creation-gating flag
220+
* (like {@link curationBackupScheduleEnabled}): `true` in staging (the matcher
221+
* is live there); `false` in prod until the prod corpus is published and the
222+
* #443 fix lands — at which point the nightly reaches `etl:dynamodb` again and
223+
* this stopgap can be retired. Gating CREATION (not just the rule's Enabled
224+
* flag) keeps the extra state machine + rule + alarm out of prod entirely.
225+
*/
226+
readonly opportunityProjectionScheduleEnabled: boolean;
210227
/**
211228
* The externally-created, TGW-attached VPC that on-prem-reachable ETL tasks
212229
* run in — specifically the ED LDAP → S3 email-visibility export (#443).
@@ -367,6 +384,10 @@ const ENV_CONFIG: Record<EnvName, SpsEnvConfig> = {
367384
// #1032 — daily curated-tables logical backup; enabled in staging (the
368385
// backup is live + verified here). Read-only + tiny, so safe from launch.
369386
curationBackupScheduleEnabled: true,
387+
// #1218 — daily standalone DynamoDB projection so the funding-matcher corpus
388+
// stays fresh while the nightly is blocked at etl:ed (#443). On in staging
389+
// (matcher is live here); idempotent upsert, so safe from launch.
390+
opportunityProjectionScheduleEnabled: true,
370391
// #443 — staging runs the ED email-visibility bridge in scholars-dev, whose
371392
// on-prem LDAP reach is proven (2026-06-18: in-VPC LDAPS bind + 2440-unit
372393
// search). Only the two private `app` subnets (TGW + NAT routes) are listed.
@@ -454,6 +475,10 @@ const ENV_CONFIG: Record<EnvName, SpsEnvConfig> = {
454475
// prod (deploy + first verify run) then flip this to true. See
455476
// docs/curation-backup-runbook.md § Prod.
456477
curationBackupScheduleEnabled: false,
478+
// #1218 — standalone DynamoDB projection ships disabled on prod: the prod
479+
// opportunity corpus isn't published yet and the matcher is dark there. Flip
480+
// when the prod corpus lands (or leave off once #443 unblocks the nightly).
481+
opportunityProjectionScheduleEnabled: false,
457482
// #443 — prod's on-prem-reachable VPC is scholars-prod. Wired but NOT yet
458483
// activated: edEmailVisibilityBridgeEnabled stays false until the
459484
// scholars-prod path is verified end-to-end (the same in-VPC bind probe as

‎cdk/lib/etl-stack.ts‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1870,6 +1870,132 @@ export class EtlStack extends Stack {
18701870
curationBackupCadenceAlarm.addAlarmAction(alarmAction);
18711871
}
18721872

1873+
// ------------------------------------------------------------------
1874+
// Standalone DynamoDB projection schedule — daily `etl:dynamodb` (#1218).
1875+
//
1876+
// `etl:dynamodb` mirrors ReciterAI's `reciterai` DynamoDB table into the
1877+
// `opportunity` + scholar tables. It is already a NIGHTLY step, but it sits
1878+
// after `etl:ed`, which the on-prem-routing gap (#443) blocks — so the nightly
1879+
// aborts before reaching it and newly-published opportunities 404 until an
1880+
// operator re-projects by hand (OPERATIONS-RUNBOOK § run-task). This standalone
1881+
// daily schedule runs just that one step so the funding-matcher corpus stays
1882+
// fresh independent of the blocked nightly. Idempotent upsert, so a double-run
1883+
// with the nightly (once #443 lands) is harmless; retire the stopgap then.
1884+
//
1885+
// Like the curated-tables backup above, the WHOLE block is creation-gated on
1886+
// `opportunityProjectionScheduleEnabled` (prod ships nothing until the prod
1887+
// corpus is published), not just the rule's Enabled flag. Reuses the 8 GB ETL
1888+
// task def + SG + private-egress subnets (same placement as the nightly step).
1889+
// ------------------------------------------------------------------
1890+
if (envConfig.opportunityProjectionScheduleEnabled) {
1891+
const projectionTask = new tasks.EcsRunTask(this, "TaskOpportunityProjection", {
1892+
integrationPattern: sfn.IntegrationPattern.RUN_JOB,
1893+
cluster: ecsCluster,
1894+
taskDefinition: this.etlTaskDefinition,
1895+
launchTarget: new tasks.EcsFargateLaunchTarget({
1896+
platformVersion: ecs.FargatePlatformVersion.LATEST,
1897+
}),
1898+
assignPublicIp: false,
1899+
subnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
1900+
securityGroups: [etlSecurityGroup],
1901+
containerOverrides: [
1902+
{
1903+
containerDefinition: etlContainer,
1904+
command: ["npm", "run", "etl:dynamodb"],
1905+
},
1906+
],
1907+
// A full reciterai scan + upsert; 1h is generous over cold start + the
1908+
// 10k-scholar/opportunity projection, and well under the daily cadence so
1909+
// a wedged run can't stack on the next fire.
1910+
taskTimeout: sfn.Timeout.duration(Duration.hours(1)),
1911+
});
1912+
projectionTask.addRetry({
1913+
errors: ["States.TaskFailed", "States.Timeout"],
1914+
maxAttempts: 1,
1915+
backoffRate: 2,
1916+
interval: Duration.minutes(1),
1917+
});
1918+
projectionTask.addCatch(
1919+
new tasks.SnsPublish(this, "NotifyOpportunityProjection", {
1920+
topic: this.failureTopic,
1921+
subject: `SPS opportunity projection ${env} -- run failed`,
1922+
message: sfn.TaskInput.fromObject({
1923+
env,
1924+
step: "OpportunityProjection",
1925+
stateMachine: sfn.JsonPath.stateMachineName,
1926+
execution: sfn.JsonPath.executionName,
1927+
error: sfn.JsonPath.stringAt("$.error"),
1928+
}),
1929+
}).next(
1930+
new sfn.Fail(this, "FailOpportunityProjection", {
1931+
cause: "opportunity projection run failed",
1932+
}),
1933+
),
1934+
{ errors: ["States.ALL"], resultPath: "$.error" },
1935+
);
1936+
1937+
const projectionSmLogGroup = new logs.LogGroup(this, "OpportunityProjectionSmLogGroup", {
1938+
logGroupName: `/aws/states/opportunity-projection-${env}`,
1939+
retention: logRetention,
1940+
removalPolicy: RemovalPolicy.RETAIN,
1941+
});
1942+
const opportunityProjectionStateMachine = new sfn.StateMachine(
1943+
this,
1944+
"OpportunityProjectionStateMachine",
1945+
{
1946+
stateMachineName: `scholars-opportunity-projection-${env}`,
1947+
stateMachineType: sfn.StateMachineType.STANDARD,
1948+
definitionBody: sfn.DefinitionBody.fromChainable(projectionTask),
1949+
timeout: Duration.minutes(65),
1950+
logs: {
1951+
destination: projectionSmLogGroup,
1952+
level: sfn.LogLevel.ERROR,
1953+
includeExecutionData: false,
1954+
},
1955+
tracingEnabled: true,
1956+
},
1957+
);
1958+
1959+
// Daily at 06:30 UTC -- after the curated-tables backup (06:00), before the
1960+
// nightly window (07:00); refreshes the corpus ahead of the workday.
1961+
const projectionRule = new events.Rule(this, "OpportunityProjectionScheduleRule", {
1962+
ruleName: `sps-opportunity-projection-${env}`,
1963+
description: `SPS opportunity projection (etl:dynamodb) -- daily 06:30 UTC (${env}). #1218.`,
1964+
schedule: events.Schedule.cron({ minute: "30", hour: "6" }),
1965+
enabled: true,
1966+
});
1967+
projectionRule.addTarget(
1968+
new eventsTargets.SfnStateMachine(opportunityProjectionStateMachine, {
1969+
input: events.RuleTargetInput.fromObject({}),
1970+
retryAttempts: 0,
1971+
}),
1972+
);
1973+
1974+
// Cadence alarm -- silent schedule death (rule disabled, IAM gap) would let
1975+
// the corpus go stale unnoticed. Alarm if no execution started across two
1976+
// consecutive 1-day windows (~2 missed daily fires); the Catch above owns
1977+
// failed runs, this owns absence.
1978+
const projectionCadenceAlarm = new cloudwatch.Alarm(this, "OpportunityProjectionCadenceAlarm", {
1979+
alarmName: `sps-opportunity-projection-cadence-${env}`,
1980+
alarmDescription: `SPS opportunity projection (${env}) -- cadence missed (no execution started in ~2 days). Next: confirm the daily rule is enabled and the state-machine IAM is intact; run 'npm run etl:dynamodb' via run-task to bridge the gap.`,
1981+
metric: new cloudwatch.Metric({
1982+
namespace: "AWS/States",
1983+
metricName: "ExecutionsStarted",
1984+
statistic: cloudwatch.Stats.SUM,
1985+
period: Duration.days(1),
1986+
dimensionsMap: {
1987+
StateMachineArn: opportunityProjectionStateMachine.stateMachineArn,
1988+
},
1989+
}),
1990+
evaluationPeriods: 2,
1991+
datapointsToAlarm: 2,
1992+
threshold: 1,
1993+
comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
1994+
treatMissingData: cloudwatch.TreatMissingData.BREACHING,
1995+
});
1996+
projectionCadenceAlarm.addAlarmAction(alarmAction);
1997+
}
1998+
18731999
// ------------------------------------------------------------------
18742000
// ED email-visibility bridge — on-prem LDAP → S3 → RDS (#443).
18752001
//

0 commit comments

Comments
 (0)