Tenant pipeline: rolling code out to every tenant stack

Goal

With silo tenants, “deploy the new version” no longer means one stack. It means the pooled stack plus one stack per Platinum tenant, and that number grows with every sale. This step adds a pipeline that, whenever new code is uploaded, updates every tenant stack listed in the tenant mapping table, in waves.

How it works

The tenant pipeline: source.zip, CodePipeline and a state machine that updates tenant stacks in waves

The state machine has two nested Map states: waves run one at a time (maxConcurrency 1) and up to 5 stacks of a wave run in parallel. Each stack is updated by a CodeBuild run of server/scripts/update-tenant.sh, which downloads that exact source.zip version, runs npm ci and then CDK_PARAM_TENANT_ID=<tenant> CDK_PARAM_COMMIT_ID=<version> cdk deploy <stackName> --exclusively.

Waves come from waveNumber in the mapping table (default 1, or set CDK_PARAM_WAVE_NUMBER when a stack is created). Put a few low-risk tenants in wave 1, check the alarms, and the rest follow. If any stack in a wave fails, the Map fails and later waves never start.
Each stack’s mapping row records the code version it runs (codeCommitId). The tenant template’s custom resource updates it on every deploy.
/whoami now also returns codeVersion (env var CODE_VERSION = CDK_PARAM_COMMIT_ID), so you can see which version a tenant is on.

Compared with the original

The original reference solution defines the same idea with a hand-written ASL JSON file, a “wave iterator” Lambda and an SQS approval queue, and gives CodeBuild "*" permissions. Here the state machine is built with CDK’s Step Functions constructs (two nested Map states). The CodeBuild project deploys through the CDK bootstrap roles (sts:AssumeRole on cdk-* only), like the provisioning job in step 09. The pipeline is triggered by S3 EventBridge events instead of polling.

Files

server/cdk/lib/tenant-pipeline/tenant-pipeline-stack.ts: CodeBuild project, Lambda, state machine, pipeline, EventBridge rule.
server/src/TenantPipeline/list_tenant_stacks.py: mapping table → ordered waves (paginated scan).
server/scripts/update-tenant.sh: updates one stack.
server/cdk/lib/bootstrap-template/bootstrap-template-stack.ts: eventBridgeEnabled on the source bucket.
server/src/WhoAmI/whoami.py, server/cdk/bin/serverless-saas.ts: codeVersion, CDK_PARAM_WAVE_NUMBER, the pipeline stack.
server/tests/test_list_tenant_stacks.py: wave grouping and order (44 tests pass).
scripts/cleanup.sh: deletes the silo stacks, the client stack and all server stacks.

Deploy

./scripts/deploy.sh you@example.com

Gotcha: a new CodePipeline runs once as soon as it’s created, using whatever source.zip is currently in the bucket. That can be older than what you just deployed with cdk deploy. deploy.sh uploads the fresh source right after deploying, so the newest version wins. On the test deployment the first run also collided with a local cdk deploy of the pooled stack (Cannot delete ChangeSet in status CREATE_IN_PROGRESS) and failed harmlessly.

Verify

On the test deployment: pooled tenant acme and a new Platinum tenant initech (onboarded with the previous code version), with initech moved to wave 2:

aws dynamodb update-item --table-name <TenantMappingTable> --key '{"tenantId":{"S":"<initech id>"}}' \
  --update-expression 'set waveNumber = :w' --expression-attribute-values '{":w":{"S":"2"}}'
./scripts/upload-source.sh

	Before	After
acme `/whoami` (pooled stack)	`codeVersion: "local"` (deployed from a laptop)	`codeVersion: "YTZTkJso…"`
initech `/whoami` (silo stack)	no `codeVersion` field (older code)	`codeVersion: "YTZTkJso…"`
mapping table `codeCommitId`	`local` / `kwNcwFFv…`	`YTZTkJso…` for both

The pipeline execution was triggered by CloudWatchEvent (the S3 upload) with source revision YTZTkJso… and status Succeeded. The state machine ran the waves in order:

00:57:17  Update tenant stack  serverless-saas-tenant-template-pooled      (wave 1)
00:59:54  succeeded
00:59:54  Update tenant stack  serverless-saas-tenant-template-145d1d02…   (wave 2, started only after wave 1)
01:02:26  succeeded

Clean up

./scripts/cleanup.sh

This deletes every Platinum tenant stack listed in the mapping table, then serverless-saas-client, and then all server stacks (pipeline, pooled tenant template, bootstrap, control plane). The CDKToolkit bootstrap stack is left in place.

Where to go next

The rebuild now covers the reference solution end to end. Ideas to extend it:

Canary deployments: shift traffic on the functions’ live alias with CodeDeploy, rolled back by the error alarms from step 05.
A per-tenant CloudWatch dashboard from the tenant_id EMF dimension.
Filter inactive tenants out of GET /tenant-config (see the notes in steps 08 and 11).
Turn API Gateway scopes back on for the control plane (setAPIGWScopes: true).