Spanner Autoscaler is a Kubernetes Operator to scale Google Cloud Spanner automatically based on Cloud Spanner Instance CPU utilization like Horizontal Pod Autoscaler.
Cloud Spanner is scalable. When CPU utilization becomes high, we can reduce it by increasing compute capacity.
Spanner Autoscaler is created to reconcile Cloud Spanner compute capacity like Horizontal Pod Autoscaler by configuring a compute capacity range and targetCPUUtilization.
When CPU Utilization(High Priority) is above (or below) targetCPUUtilization, Spanner Autoscaler tries to bring it back to the threshold by calculating desired compute capacity and then increasing (or decreasing) compute capacity.
The pricing of Cloud Spanner states that any compute capacity which is provisioned will be billed for a minimum of one hour, so Spanner Autoscaler maintains the increased compute capacity for about an hour. Spanner Autoscaler has --scale-down-interval flag (default: 55min) for achieving this.
While scaling down, removing large amounts of compute capacity at once (like 10000 PU -> 1000 PU) can cause a latency increase. Therefore, Spanner Autoscaler decreases the compute capacity in steps to avoid such large disruptions. This step size can be configured with the scaledownStepSize parameter (default: 2000 PU). Similarly, scaleupStepSize limits how much capacity can be added in a single scale-up operation (default: no limit). Both parameters accept either an integer number of Processing Units or a percentage of current capacity (e.g., "10%").
Per-resource scale intervals can also be configured with scaledownInterval and scaleupInterval in the scaleConfig section, overriding the global --scale-down-interval and --scale-up-interval flags set on the controller.
If there are some batch jobs or any other compute intensive tasks which are run periodically on the Cloud Spanner, it is now possible to bump up the scaling range only for a specified duration. For example, the following SpannerAutoscaleSchedule will add an extra compute capacity of 600 Processing Units to the spanner instance every day at 2 o'clock, just for 3 hours:
apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaleSchedule
metadata:
name: spannerautoscaleschedule-sample
namespace: your-namespace
spec:
targetResource: spannerautoscaler-sample
additionalProcessingUnits: 600
schedule:
cron: "0 2 * * *"
duration: 3hThe cron field supports extended syntax (L, L-n, nW, LW, DAY#n, DAY#L) in addition to the standard 5-field format, powered by go-cron. See the Extended Syntax documentation for details and examples.
Note: When multiple schedules are active simultaneously (i.e. their windows overlap), the
additionalProcessingUnitsfrom all active schedules are summed and added to bothdesiredMinPUsanddesiredMaxPUs. For example, if schedule A adds +1,000 PU and schedule B adds +5,000 PU and both are active at the same time,desiredMinPUs = spec.processingUnits.min + 6,000.
To prevent unexpected scale down operations during business hours or critical periods, you can restrict scale down operations to specific time windows using cron expressions. This feature allows you to limit scale downs to maintenance windows or low-traffic periods (such as late night hours).
apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
spec:
scaleConfig:
# Scale down only during late night hours (2:00 AM to 4:59 AM daily)
scaledownAllowedTimes:
- "* 2-4 * * *"
# Other configuration...For time windows that cross midnight (e.g., 11:00 PM to 5:59 AM), you can specify multiple cron expressions:
scaledownAllowedTimes:
- "* 23 * * *" # 11:00 PM to 11:59 PM
- "* 0-5 * * *" # 12:00 AM to 5:59 AMAlternatively, you can prevent scale down operations during specific time periods using scaledownNotAllowedTimes. This is useful when you want to prevent scale downs during peak hours or critical business periods:
apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
spec:
scaleConfig:
# Prevent scale down during business hours (9:00 AM to 5:59 PM on weekdays)
scaledownNotAllowedTimes:
- "* 9-17 * * 1-5"
# Other configuration...For multiple forbidden periods, you can specify multiple cron expressions:
scaledownNotAllowedTimes:
- "* 12-13 * * 1-5" # Lunch time on weekdays
- "* 18-19 * * 1-5" # Evening peak on weekdaysNote: You can specify either
scaledownAllowedTimesORscaledownNotAllowedTimes, but not both. When neither is specified, scale down operations are allowed at any time (default behavior). Scale up operations are never restricted and will always be executed immediately when needed, regardless of time restrictions.
The cron expressions use the standard 5-field format: minute hour day-of-month month day-of-week. For example:
"* 2-4 * * *"- Every minute from 2:00 AM to 4:59 AM daily"0 9-17 * * 1-5"- At minute 0 (top of the hour) from 9:00 AM to 5:00 PM on weekdays
Important limitations:
- Hour-level precision only: Time ranges are limited to full-hour boundaries. You cannot specify minute-level ranges like "3:15 AM to 8:40 AM".
- Minute field applies to entire range: If you specify a minute value (e.g.,
"30 9-17 * * *"), it applies to every hour in the range (9:30, 10:30, 11:30, etc.). - Use wildcard (*) for continuous coverage: To allow scale-downs throughout an hour range, use
*in the minute field.
For complex time requirements involving specific minutes, consider using multiple separate cron expressions or adjusting your maintenance windows to align with hour boundaries.
Spanner Autoscaler can be installed using KPT by following 2 steps:
-
Deploy the operator through
kpt$ kpt pkg get https://github.com/mercari/spanner-autoscaler/config spanner-autoscaler-pkg $ kpt live init spanner-autoscaler-pkg/kpt $ kpt live install-resource-group ## Append '--dry-run' to the below line to just ## check the resources which will be created $ kustomize build spanner-autoscaler-pkg/kpt | kpt live apply - ## To uninstall, use the following $ kustomize build spanner-autoscaler-pkg/kpt | kpt live destroy -
ℹ️ TIP: Instead of
kpt, you can also usekubectldirectly to install the resources (use?ref=masterfor latest version) as follows:$ kustomize build "https://github.com/mercari/spanner-autoscaler.git/config/default?ref=v0.4.1" | kubectl apply -f -These resources can then be adopted by
kptby using the--inventory-policy=adoptflag while usingkpt live applycommand. More info. -
Create a Custom Resource for managing a spanner instance
$ kubectl apply -f spanner-autoscaler-pkg/samplesExamples of CustomResources can be found below.
For authentication using a GCP service account JSON key, follow these steps to create a k8s secret with credentials.
apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
namespace: your-namespace
spec:
targetInstance:
projectId: your-gcp-project-id
instanceId: your-spanner-instance-id
scaleConfig:
processingUnits:
min: 1000
max: 10000
scaledownStepSize: "20%" # or an integer, e.g. 2000
scaleupStepSize: "50%" # or an integer, e.g. 3000; defaults to no limit
scaledownInterval: 55m # overrides --scale-down-interval for this resource
scaleupInterval: 30s # overrides --scale-up-interval for this resource
targetCPUUtilization:
highPriority: 60Scale based on both High Priority and total CPU utilization simultaneously. The controller scales out when either metric exceeds its target (OR condition) and scales in only when both metrics are below their respective targets.
apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
namespace: your-namespace
spec:
targetInstance:
projectId: your-gcp-project-id
instanceId: your-spanner-instance-id
scaleConfig:
processingUnits:
min: 1000
max: 10000
targetCPUUtilization:
highPriority: 60 # required
total: 65 # optional; enables dual-metric scaling when setNote:
highPriorityis required.totalis optional — when omitted, scaling uses only the High Priority CPU metric. When both are specified, they use independent Cloud Monitoring metrics (spanner.googleapis.com/instance/cpu/utilization_by_priorityfor High Priority andspanner.googleapis.com/instance/cpu/utilizationfor total). The desired processing units for each metric are calculated independently (respectingscaleupStepSize/scaledownStepSize), and the maximum is applied. Scale-out fires when either metric exceeds its target; scale-in fires only when both are below their targets.
apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
namespace: your-namespace
spec:
targetInstance:
projectId: your-gcp-project-id
instanceId: your-spanner-instance-id
scaleConfig:
processingUnits:
min: 1000
max: 10000
# Allow scale down only during late night hours (2:00 AM to 4:59 AM daily)
scaledownAllowedTimes:
- "* 2-4 * * *"
targetCPUUtilization:
highPriority: 65For time windows crossing midnight:
apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
namespace: your-namespace
spec:
targetInstance:
projectId: your-gcp-project-id
instanceId: your-spanner-instance-id
scaleConfig:
processingUnits:
min: 1000
max: 5000
# Allow scale down from 11:00 PM to 5:59 AM daily (crossing midnight)
scaledownAllowedTimes:
- "* 23 * * *" # 11:00 PM to 11:59 PM
- "* 0-5 * * *" # 12:00 AM to 5:59 AM
targetCPUUtilization:
total: 70Scale down time restrictions support timezone specification using the CRON_TZ format:
apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
namespace: your-namespace
spec:
targetInstance:
projectId: your-gcp-project-id
instanceId: your-spanner-instance-id
scaleConfig:
processingUnits:
min: 1000
max: 10000
# Allow scale down only during late night hours in Tokyo timezone
scaledownAllowedTimes:
- "CRON_TZ=Asia/Tokyo * 2-4 * * *"
targetCPUUtilization:
highPriority: 65When using CRON_TZ format, times are evaluated in the specified timezone rather than the controller's local timezone. This is useful for deployments where the controller runs in a different timezone than your business hours.
apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
namespace: your-namespace
spec:
targetInstance:
projectId: your-gcp-project-id
instanceId: your-spanner-instance-id
scaleConfig:
processingUnits:
min: 1000
max: 4000
scaledownStepSize: 1000
targetCPUUtilization:
highPriority: 60 apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
namespace: your-namespace
spec:
targetInstance:
projectId: your-gcp-project-id
instanceId: your-spanner-instance-id
+ authentication:
+ iamKeySecret:
+ namespace: your-namespace
+ name: spanner-autoscaler-gcp-sa
+ key: service-account
scaleConfig:
processingUnits:
min: 1000
max: 4000
scaledownStepSize: 1000
targetCPUUtilization:
highPriority: 60 apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
name: spannerautoscaler-sample
namespace: your-namespace
spec:
targetInstance:
projectId: your-gcp-project-id
instanceId: your-spanner-instance-id
+ authentication:
+ impersonateConfig:
+ targetServiceAccount: GSA_SPANNER@TENANT_PROJECT.iam.gserviceaccount.com
scaleConfig:
processingUnits:
min: 1000
max: 4000
scaledownStepSize: 1000
targetCPUUtilization:
highPriority: 60Note:
spec.targetInstance(projectIdandinstanceId) is immutable after creation. To change the target Spanner instance, delete theSpannerAutoscalerand create a new one.
On your GCP project, you will need to enable spanner.googleapis.com and monitoring.googleapis.com APIs.
You will need to create at least one GCP service account, which will be used by the spanner-autoscaler controller to authenticate with GCP for modifying compute capacity of a Spanner instance. This service account should have the following roles:
roles/spanner.admin(on the Spanner instances)roles/monitoring.viewer(on the project)
For fine grained access control, you should create one GCP service account per Spanner instance. This way, you will be able to specify a different service account in each of SpannerAutoscaler CRD resources you create later.
Generate a JSON key for the GCP service account (created above) and put it in a Kubernetes Secret:
$ kubectl create secret generic spanner-autoscaler-gcp-sa --from-file=service-account=./service-account-key.json -n your-namespaceℹ️ By default,
spanner-autoscalerwill have read access tosecrets namedspanner-autoscaler-gcp-sain any namespace. If you wish to use a different name for your secret, then you need to explicitly create aRoleand aRoleBinding(example) in your namespace. This will providespanner-autoscalerwith read access to any secret of your choice.
You can then refer to this secret in your SpannerAutoscaler CRD resource with serviceAccountSecretRef field [example].
Following are some other advanced methods which can also be used for GCP authentication:
Details
- Enable Workload Identity on the GKE cluster - Ref.
- Let's call the Kubernetes service account of the controller (
spanner-autoscaler/spanner-autoscaler-controller-manager) asKSA_CONTROLLERand the GCP service account created above asGSA_CONTROLLER.
Now configure Workload Identity betweenKSA_CONTROLLERandGSA_CONTROLLERwith the following steps:- Allow
KSA_CONTROLLERto impersonateGSA_CONTROLLERby creating an IAM Policy binding:$ gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:PROJECT_ID.svc.id.goog[spanner-autoscaler/spanner-autoscaler-controller-manager]" GSA_CONTROLLER@PROJECT_ID.iam.gserviceaccount.com` - Add annotation
$ kubectl annotate serviceaccount --namespace spanner-autoscaler spanner-autoscaler-controller-manager iam.gke.io/gcp-service-account=GSA_CONTROLLER@PROJECT_ID.iam.gserviceaccount.com`
- Allow
Details
You can configure the controller (spanner-autoscaler-controller-manager) to use GKE Workload Identity feature for key-less GCP access. Steps to do this:
Details
The Kubernetes service account which is used for running the spanner-autoscaler controller can be bound to the GCP service account (created above) through Workload Identity. If this is done, there is no need to provide serviceAccountSecretRef or impersonateConfig authentication parameters in the spec section of the SpannerAutoscaler CRD resources.
An example for this is shown here.
GSA_SPANNER: The GCP Service Account (created above) which has the correct permissions for modifying Spanner compute capacityGSA_CONTROLLER: The GCP Service Account which is used for Workload Identity with the GKE clusterKSA_CONTROLLER: The Kubernetes Service Account which is used for running the spanner-autoscaler controller pod in the GKE
Details
In this method there are 3 service accounts involved (2 GCP service accounts and 1 Kubernetes service account):
After enabling Workload Identity between GSA_CONTROLLER and KSA_CONTROLLER, you can configure GSA_CONTROLLER as roles/iam.serviceAccountTokenCreator of the GSA_SPANNER service account as follows:
$ gcloud iam service-accounts add-iam-policy-binding $GSA_SPANNER --member=serviceAccount:$GSA_CONTROLLER --role=roles/iam.serviceAccountTokenCreatorThis will allow KSA_CONTROLLER to use GSA_CONTROLLER and impersonate (act as) GSA_SPANNER for a short period of time (by using a short-lived token). An example for this can be found here.
spanner.instances.getspanner.instances.updatemonitoring.timeSeries.list
TIP: Custom role with minimum permissions
Details
Instead of predefined roles, you can define and use a custom role with lesser privileges for Spanner Autoscaler. To scale the target Cloud Spanner instance, the weakest predefined role is roles/spanner.admin. To observe the CPU usage metric of the project of the Spanner instance, the weakest predefined role is roles/monitoring.viewer.
The custom role can be created with just the following permissions:
The controller exposes custom Prometheus metrics on the standard controller-runtime /metrics endpoint (bound to --metrics-bind-address, default 127.0.0.1:8080). All business metrics carry the four identity labels namespace, name, project_id, instance_id so they can be sliced by either the Kubernetes resource identity or the target Spanner instance.
The endpoint can be scraped by Prometheus (ServiceMonitor / PodMonitor), the Datadog Agent's OpenMetrics check, VictoriaMetrics, Grafana Cloud, or any tool that consumes the Prometheus exposition format. No vendor-specific SDK is required.
| Name | Extra labels | Description |
|---|---|---|
spanner_autoscaler_current_processing_units |
— | Current Spanner processing units. |
spanner_autoscaler_desired_processing_units |
— | Desired processing units computed in the latest reconcile. |
spanner_autoscaler_min_processing_units |
— | Configured spec.scaleConfig.processingUnits.min. |
spanner_autoscaler_max_processing_units |
— | Configured spec.scaleConfig.processingUnits.max. |
spanner_autoscaler_effective_min_processing_units |
— | Effective lower bound including additions from currently active schedules. |
spanner_autoscaler_effective_max_processing_units |
— | Effective upper bound including additions from currently active schedules. |
spanner_autoscaler_cpu_utilization |
type=high_priority|total |
Current CPU utilization percentage (0–100) per metric type. |
spanner_autoscaler_cpu_utilization_target |
type=high_priority|total |
Configured target CPU utilization percentage. |
spanner_autoscaler_instance_ready |
— | 1 when the Spanner instance state is ready, 0 otherwise. |
spanner_autoscaler_active_schedules |
— | Number of SpannerAutoscaleSchedule entries currently in effect. |
spanner_autoscaler_active_schedule_additional_pu |
— | Sum of AdditionalPU contributed by currently active schedules. |
spanner_autoscaler_last_scale_timestamp_seconds |
— | Unix timestamp of the last successful processing-units update. |
spanner_autoscaler_last_sync_timestamp_seconds |
— | Unix timestamp of the last successful Cloud Monitoring sync. |
| Name | Type | Extra labels | Description |
|---|---|---|---|
spanner_autoscaler_scale_events_total |
Counter | direction=up|down, driver=cpu_high_priority|cpu_total|schedule |
Successful processing-units updates. The driver label is a best-effort attribution to the metric (or schedule floor) that drove the decision. |
spanner_autoscaler_scale_skipped_total |
Counter | reason |
Reconciles where scaling was skipped. Reasons: same, scale_up_interval, scale_down_interval, scale_down_window, instance_not_ready, cpu_not_ready. |
spanner_autoscaler_scale_pu_delta |
Histogram | direction=up|down |
Absolute processing-units change per scale event. Buckets: 100, 200, 500, 1000, 2000, 5000, 10000, 20000. |
| Name | Extra labels | Description |
|---|---|---|
spanner_autoscaler_schedule_activations_total |
— | Number of cron firings that created an ActiveSchedule entry. |
spanner_autoscaler_schedule_deactivations_total |
reason=expired|unregistered |
Number of ActiveSchedule entries removed (by EndTime expiry or by schedule resource deletion). |
| Name | Type | Extra labels | Description |
|---|---|---|---|
spanner_autoscaler_instance_update_total |
Counter | result=success|error |
Spanner UpdateInstance API call count. |
spanner_autoscaler_instance_update_duration_seconds |
Histogram | — | Latency of UpdateInstance calls. Buckets: 0.1, 0.5, 1, 2, 5, 10, 30, 60. |
spanner_autoscaler_metrics_fetch_total |
Counter | result=success|error |
Cloud Monitoring GetInstanceMetrics call count. |
spanner_autoscaler_metrics_fetch_duration_seconds |
Histogram | — | Latency of Cloud Monitoring fetches. Buckets: 0.05, 0.1, 0.25, 0.5, 1, 2, 5. |
The standard controller-runtime metrics (controller_runtime_reconcile_*, workqueue_*) and Go runtime metrics are also exposed on the same endpoint and are not duplicated here.
# Gap between desired and current PU (scaling lag).
spanner_autoscaler_desired_processing_units - spanner_autoscaler_current_processing_units
# UpdateInstance error rate over the last 5 minutes.
sum(rate(spanner_autoscaler_instance_update_total{result="error"}[5m]))
/
sum(rate(spanner_autoscaler_instance_update_total[5m]))
# Scale-up frequency by driver over the last hour.
sum by (driver) (rate(spanner_autoscaler_scale_events_total{direction="up"}[1h]))
# Cloud Monitoring fetch p99 latency.
histogram_quantile(0.99, sum by (le) (rate(spanner_autoscaler_metrics_fetch_duration_seconds_bucket[5m])))
# Autoscalers pinned at their effective max for 5+ minutes.
max_over_time(
(spanner_autoscaler_current_processing_units
== bool spanner_autoscaler_effective_max_processing_units)[5m:1m]
) == 1
See docs/development.md and CONTRIBUTING.md respectively.
The recommended local development workflow uses Tilt with local Spanner and Cloud Monitoring emulators — no real GCP credentials required:
$ make tilt-up # creates a kind cluster and starts Tilt
$ make tilt-down # tears everything down (Tilt, emulators, kind cluster, webhook certs)Tilt automatically starts the emulators, installs cert-manager, deploys the webhook configuration, and runs the controller locally. Changes to any Go file under cmd/, api/, or internal/ trigger a live reload. Running make tilt-up after make tilt-down starts from a completely clean state.
The older version 0.3.0 (with apiVersion: spanner.mercari.com/v1alpha1) is now deprecated in favor of 0.4.0 (with apiVersion: spanner.mercari.com/v1beta1).
Version 0.4.0 is backward compatible with 0.3.0, but there is a restructuring of the SpannerAutoscaler resource definition and names of many fields have changed. Thus it is recommended to go through the SpannerAutoscaler CRD reference and replace v1alpha1 resources with v1beta1 spec definition.
Breaking change: targetCPUUtilization.highPriority is now required. Configurations that set only targetCPUUtilization.total (without highPriority) are no longer valid and will be rejected by the webhook.
To migrate:
- If you were using only
total, add ahighPrioritytarget. The controller will now scale out when either metric exceeds its target and scale in only when both are below their targets. - If you were using only
highPriority, no change is required.
The --config flag has been removed (ControllerManagerConfig was dropped upstream in controller-runtime v0.19). Deployments still passing --config=... will fail to start with flag provided but not defined: -config.
The values previously in controller_manager_config.yaml are now flag defaults in the binary (--health-probe-bind-address=:8081, --metrics-bind-address=127.0.0.1:8080, --leader-elect=true, --leader-elect-id=54b82eb3.mercari.com), so behavior is preserved. The controller_manager_config.yaml ConfigMap and the manager_config_patch.yaml kustomize patch have been deleted — remove any references in downstream overlays. To override the defaults, pass the flag in the manager Deployment's args: list (see config/manager/manager.yaml). CRDs are unchanged.
Spanner Autoscaler is released under the Apache License 2.0.
- This project is currently in active development phase and there might be some backward incompatible changes in future versions.
- Spanner Autoscaler supports scaling based on
High PriorityCPU utilization (targetCPUUtilization.highPriority, required) and optionally total CPU utilization (targetCPUUtilization.total). When both are set, the controller scales out on either metric exceeding its target and scales in only when both are below their targets. It doesn't watchLow PriorityCPU utilization or Rolling average 24-hour utilization. - It doesn't check the storage size and the number of databases as well. You must take care of these metrics by yourself.
ℹ️ More information and background of spanner-autoscaler is available on this blog!