Add an optional SQL Server BCP fast path for fast_executemany

[rad-pat]

Summary

This adds an optional SQL Server BCP (bulk copy) fast path for
Cursor.executemany() when fast_executemany is enabled. When turned on, an
INSERT ... VALUES (...) executemany is routed through the SQL Server bulk-copy
API (bcp_init / bcp_bind / bcp_sendrow / bcp_batch / bcp_done) instead
of parameter-array INSERTs. For large loads this is dramatically faster,
because it bypasses per-statement query processing.

It is opt-in and safe: if BCP can't be used for a given call (not a
parameterized VALUES insert, an unsupported value type, a partial column list,
a non-Microsoft driver, etc.) it transparently falls back to the existing
fast_executemany path. The BCP entry points are loaded on demand from the
in-use msodbcsql driver, so there is no new build/link dependency.

Relationship to #1486

#1486 ("Implement bulk copy support") and this PR tackle bulk copy from
different angles and are complementary, not competing:

• Implement bulk copy support #1486 adds a connection.bcp(...) method that bulk-copies between a
  data file and a table (bcp_init/bcp_readfmt/bcp_exec + a format
  file), supporting both import and export — the classic bcp utility
  exposed programmatically. You bring a file (and a format file).
• This PR accelerates in-memory cursor.executemany() /
  fast_executemany (and SQLAlchemy) inserts by streaming the Python rows
  through bcp_bind/bcp_sendrow — no data file or format file. Import only.

The only overlap is low-level plumbing: both enable SQL_COPT_SS_BCP and
dynamically load the driver's bcp_* entry points (and define the same
SQL_COPT_SS_BCP/SQL_BCP_ON constants). If #1486 lands first — or the
maintainers prefer a single BCP module — I'm happy to rebase this on top and
share that infrastructure (a common bcp_* symbol loader + constants) rather
than duplicate it.

How to use (raw pyodbc)

BCP must be enabled on the connection before connecting, via the SQL
Server-specific SQL_COPT_SS_BCP pre-connect attribute:

import pyodbc

SQL_COPT_SS_BCP = 1219
SQL_BCP_ON = 1

cnxn = pyodbc.connect(connection_string,
                      autocommit=True,
                      attrs_before={SQL_COPT_SS_BCP: SQL_BCP_ON})

cur = cnxn.cursor()
cur.fast_executemany = True      # required
cur.use_bcp_fast = True          # opt into the BCP path
cur.bcp_batch_rows = 10_000      # optional: rows per committed batch (0 = one batch)
cur.bcp_tablock = True           # optional: pass the TABLOCK hint

cur.executemany(
    "INSERT INTO dbo.target (a, b, c) VALUES (?, ?, ?)",
    rows)

New Cursor attributes:

attribute	type	meaning
use_bcp_fast	bool	route fast_executemany through BCP when possible
bcp_batch_rows	int	rows per committed batch (0 = a single batch)
bcp_tablock	bool	pass the TABLOCK hint (bulk-update lock; can enable minimal logging)

AUTOCOMMIT is recommended for BCP loads (the bulk copy is committed by
bcp_done/bcp_batch).

How to use via SQLAlchemy

Enable BCP on the connection through connect_args, and flip use_bcp_fast on
each executemany with a before_cursor_execute listener:

import sqlalchemy as sa
from sqlalchemy import event

SQL_COPT_SS_BCP = 1219
SQL_BCP_ON = 1

engine = sa.create_engine(
    "mssql+pyodbc://user:pwd@host/db?driver=ODBC+Driver+18+for+SQL+Server",
    connect_args={"attrs_before": {SQL_COPT_SS_BCP: SQL_BCP_ON}},
    fast_executemany=True,           # SQLAlchemy sets cursor.fast_executemany
    isolation_level="AUTOCOMMIT",
)

@event.listens_for(engine, "before_cursor_execute")
def _enable_bcp(conn, cursor, statement, parameters, context, executemany):
    if executemany:
        cursor.use_bcp_fast = True
        cursor.bcp_batch_rows = 10_000
        # cursor.bcp_tablock = True

with engine.begin() as conn:
    conn.execute(sa.insert(my_table), list_of_dicts)

SQLAlchemy emits an explicit column list (INSERT INTO t (cols...) VALUES (...)).
That is handled correctly (see below). When SQLAlchemy omits columns from the
list (e.g. an identity PK or a server-default column), the insert is a partial
list and falls back to fast_executemany (BCP requires every column to be
bound). Full-column bulk inserts -- the typical high-volume case -- use BCP.

Column-list handling

BCP loads by table column ordinal, not by name, so an explicit column list
is mapped to ordinals before binding:

• No column list (INSERT INTO t VALUES (?, ...)): bound positionally.
• Explicit list, any order (INSERT INTO t (c, a, b) VALUES (...)): each
  listed name is resolved to its table ordinal via SQLColumnsW, so values land
  in the right columns regardless of list order. Non-ASCII column names work.
  The lookup is pinned to the schema bcp_init targets (the declared schema, or
  SCHEMA_NAME() when omitted) to avoid cross-schema ambiguity.
• Partial list (fewer columns than the table), INSERT ... SELECT/EXEC,
  DEFAULT VALUES, duplicate column names: fall back to the normal path.

Supported value types

Host column types are deduced from the first row:

• int -> INT (or BIGINT when it doesn't fit 32 bits; out-of-range values raise)
• bool -> BIT
• float -> FLOAT
• str / bytes / bytearray -> character
• Decimal -> sent as text (non-finite values rejected)
• datetime.date / time / datetime (naive and tz-aware) -> sent as ISO text,
  letting the server convert to DATE/TIME/DATETIME2/DATETIMEOFFSET
• None -> SQL NULL

Any other value type (e.g. uuid.UUID) causes that executemany to fall back.

Behaviour / fallback

executemany falls back to the existing fast_executemany implementation (no
error) whenever BCP can't be used. bcp_init is started only after the rows
validate and the column buffers are allocated, and every error path afterwards
calls bcp_done, so a failed row never leaves the connection mid-bulk-copy.

What is not done / limitations

• SQL Server + msodbcsql only. Other drivers/databases fall back.
• Native binary temporal types are not enabled. date/time/datetime2/
  datetimeoffset are sent as ISO text and converted server-side. The native
  length-prefixed binary bind is implemented but disabled: bcp_bind does not
  carry the fractional-second scale, so the driver rejects sub-second values.
  Decimal is likewise sent as text.
• Types are deduced from the first row, so provide representative first rows
  (or leave use_bcp_fast off) for heterogeneous data.
• bytes are bound as character data, not VARBINARY.
• Reading a DATETIMEOFFSET column back is unrelated to this change but worth
  noting: pyodbc has no native decoder for SQL type -155.

Performance

Measured in CI inserting into a heap table (INT, FLOAT, VARCHAR(100), DATETIME2(7)). The first method uses fewer rows because the row-by-row path is
slow; the bulk methods use 100k rows. Database in FULL recovery:

method	rows	time	rate
executemany (row-by-row)	5,000	3.36 s	~1,500 rows/s
fast_executemany	100,000	37.3 s	~2,700 rows/s
BCP, no TABLOCK	100,000	0.367 s	~272,000 rows/s
BCP, TABLOCK	100,000	0.370 s	~270,000 rows/s

A separate run under SIMPLE recovery produced effectively the same figures.

Takeaways:

• BCP is roughly two orders of magnitude faster than fast_executemany here.
• Almost all of the gain comes from the BCP protocol itself, not minimal
  logging: TABLOCK made a negligible difference at this size, and the result
  held under both FULL and SIMPLE recovery.
• Caveat: this is a small, shared CI runner where fast_executemany is
  abnormally slow (~2,700 rows/s), which inflates the multiple. On capable
  hardware the gap will be smaller, but BCP remains clearly faster for bulk loads.

Tests

tests/sqlserver_test.py::test_bcp_fast_executemany covers all supported types,
NULLs, variable-length growth, >32/64-bit integers and batching (plain
VALUES, positional binding). test_bcp_fast_executemany_column_list covers an
explicit list that reorders columns (via BCP) and a column subset (falls back).
utils/bcp_benchmark.py is a standalone benchmark (kept under utils/ so it is
never collected by pytest, per CLAUDE.md). The full CI matrix (Python
3.10-3.14 x SQL Server 2019/2022/2025 + PostgreSQL / MySQL / SQLite) is green.

🤖 Generated with Claude Code