Skip to content

HANA multi-tenant: thundering herd on Service Manager credential fetch during cold tenant pool creation #1584

Description

@xyunxylona

Description of erroneous behaviour

In a multi-tenant HANA application, the first batch of concurrent requests for a cold tenant (no existing connection pool) causes ResourceRequest timed out errors from generic-pool.

https://github.com/cap-js/cds-dbs/blob/db-service-v2.10.0/db-service/lib/common/DatabaseService.js#L63:

this.pool = this.pools[tenant] ??= new ConnectionPool(this.pools._factory, tenant)                                                                     

When the pool is new and empty, generic-pool calls factory.create() for each pending acquire(). In HANAService.js, each create() https://github.com/cap-js/cds-dbs/blob/hana-v2.7.1/hana/lib/HANAService.js#L59:

await require('@sap/cds-mtxs/lib').xt.serviceManager.get(tenant, { disableCache: false })

The credential cache (_bindings4.cached in @sap/cds-mtxs/srv/plugins/hana/srv-mgr.js) is empty for the first request, and there is no deduplication of concurrent fetches. If 10 requests arrive before the first serviceManager.get() resolves, all 10 hit the Service Manager API independently. The API responds slowly under this burst, and acquireTimeoutMillis (default 1000ms) expires before connections are
established.

Typical error output:

[ERROR] Unhandled Rejection at: Promise  ResourceRequest timed out
[ERROR] Unhandled Rejection at: Promise  ResourceRequest timed out
[ERROR] Unhandled Rejection at: Promise  ResourceRequest timed out

Coalescing concurrent serviceManager.get() calls for the same tenant into a single Promise (e.g. a pendingFetches: Map<string, Promise> in _bindings4()) would prevent the thundering herd. Alternatively, the pool could serialize the first create() before spawning the rest.

Detailed steps to reproduce

  1. Deploy a multi-tenant CAP HANA application with default pool settings
  2. Subscribe a new tenant (or restart the server so in-memory pool/cache state is cleared)
  3. Open a Fiori Elements list page that triggers 5-10+ parallel OData requests for that tenant
  4. Observe ResourceRequest timed out errors in logs

This is most reliably triggered on the very first page load after a fresh server start or new tenant subscription, when no credential cache or connection pool exists yet.

Details about your project

Your Project Name https://github.tools.sap/Application-AI/app-ai-fde-collections-assistant
OData version v4
Node.js version v24.11.0
@sap/cds 9.7.1
@sap/cds-compiler 6.7.3
@sap/cds-dk 9.8.4
@sap/cds-mtxs 3.7.1
@cap-js/hana 2.6.0
@cap-js/db-service 2.8.2

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions