CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Changed

• Auto-indexing now uses a closed cardinality range [auto_index_min_threshold, auto_index_threshold] instead of a single upper bound (issue #142). Defaults bumped: auto_index_threshold 3 → 10 and the new auto_index_min_threshold defaults to 3. The new lower bound skips the useless-single-value-column case @EricSoroos flagged in #142 (a 1-value text column previously produced a 10–40MB B-tree the Postgres planner would never choose). The upper bump from 3 → 10 widens the DataTables-SearchBuilder filtering sweet spot to cover typical enum-shaped columns (status = 3–10 values, common enums under ~10). Operator-facing impact at default settings: columns with cardinality 1–2 lose their auto-index (intentional — the indexes were dead weight); columns with cardinality 4–10 gain an auto-index. No migration is required; the change re-applies on the next resubmit. To restore pre-#142 behavior (no lower floor) set ckanext.datapusher_plus.auto_index_min_threshold = 0. The setting auto_index_threshold = -1 ("index every column") still works but now hits the min_threshold = 3 floor unless paired with auto_index_min_threshold = 0.
• BREAKING Bumped MINIMUM_QSV_VERSION from 4.0.0 to 20.0.0. Operators must upgrade their qsv binary at the path configured by ckanext.datapusher_plus.qsv_bin before deploying this version — DP+ will refuse to start otherwise. See qsv 20.0.0 release notes and the migration notes below. (README install snippet updated accordingly.)
• BREAKING Bumped MINIMUM_QSV_VERSION from 20.0.0 to 20.1.0 (and bumped the qsv version installed by Dockerfile.worker, .github/workflows/ci.yml, and .github/workflows/main.yml to match). Operators must upgrade their qsv binary at the path configured by ckanext.datapusher_plus.qsv_bin before deploying this version — DP+ will refuse to start otherwise with JobError: At least qsv version 20.1.0 required. Found 20.0.0.. qsv 20.1.0 itself introduces no breaking changes against 20.0.0 (per the 20.1.0 release notes: "pipelines built on 20.0.0 will upgrade in place"), so the upgrade is binary-swap-only — no data re-ingestion is required. The reason this is still flagged as BREAKING is the minimum-version gate, not the qsv behavior. User-visible improvement justifying the floor bump: qsv-dateparser 0.14 → 0.15 in qsv 20.1.0 adds recognition for ISO 8601 T-separated datetimes without a timezone suffix (e.g. 2024-10-11T14:30:00) — qsv 20.0.0 misclassified these as String during qsv stats --infer-dates, which in DP+ surfaced as a date-typed column being demoted to text on certain CSV shapes (one of the gaps documented in the issue #173 regression test, now closed). The regression-test matrix in tests/test_issue_173_date_format_inference.py is updated to the new baseline.

qsv 20.0.0 migration notes (operator-facing)

The new minimum brings the following user-visible changes from qsv 4.0.0 → 20.0.0 that may affect existing CKAN deployments:

• Postgres column names may shift for edge-case headers. qsv's safe_header_names now enforces a 60-byte cap on the final sanitized column name (including any duplicate-disambiguation suffix like _2), snapping at UTF-8 char boundaries. Previously the cap was char-based and could produce names up to ~240 bytes, exceeding Postgres' default NAMEDATALEN of 63 bytes.
  • ASCII-only headers without duplicates: unchanged.
  • ASCII headers with duplicate names that previously landed at 61–63 chars: now 1–2 chars shorter.
  • CJK / accented / emoji headers that previously produced names > 60 bytes: now aggressively trimmed to fit.
  • Resources ingested under qsv ≤ 19.1.0 whose sanitized column names fell in either of the latter two buckets will get different column names on resubmit under qsv 20.0.0. After upgrading, re-submit affected resources (or drop the old datastore table first) so schema and dpp_suggestions stay in sync.
• qsv stats --percentiles consolidated column format changed. When --percentiles is enabled (via ckanext.datapusher_plus.summary_stats_options), the single percentiles column now emits entries as <N>: <value> joined by | (e.g. "5: 1|10: 1|40: 2|60: 3|90: 5|95: 5") instead of bare values joined by | (qsv 12.0.0). DP+ does not pass --percentiles by default (SUMMARY_STATS_OPTIONS in config.py is unset), so default deployments are unaffected. Risk surface: deployments that have set summary_stats_options = "--percentiles" AND parse the consolidated percentiles column in scheming formula: / suggest_formula: expressions. Audit those formulas before upgrading and update parsers to handle the <N>: prefix in each pipe-delimited entry.
• qsv stats cache fingerprint changed from SHA256 to BLAKE3 (qsv 9.1.0). On first run after upgrade, any qsv stats caches are silently regenerated. No action required.
• qsv stats output adds n_negative/n_zero/n_positive columns for numeric fields (qsv 11.0.2) and weighted-stats columns (qsv 12.0.0). DP+ consumes the stats CSV via csv.DictReader and only reads named columns, so the additions are transparent.
• qsv safenames verify-mode JSON output corrections (qsv 20.0.0): unsafe_headers strings now have surrounding " and whitespace trimmed; duplicate_headers is sorted alphabetically; verify counts now correctly include duplicate-suffix renames. DP+ only reads unsafe_headers length and logs the list, so these corrections are benign for the plugin.

See docs/qsv-20.0.0-upgrade-test-plan.md for the regression-test checklist used to validate the bump.

What's Changed

• Update README with some fixes by @tino097 in #178
• Druf apr2025 by @jqnatividad in #180
• Refactor upload log level by @jqnatividad in #181
• feat: zip file support by @jqnatividad in #182
• feat: shapefile support by @jqnatividad in #183
• Refactor jobs py by @jqnatividad in #184
• feat: make frequency limit configurable; move stats/freq copying to datastore from jobs.py to qsv_utils.py by @jqnatividad in #185
• Lat lon columns inferencing for use in Formulas by @jqnatividad in #188
• Configurable Date/Datetime inferencing and dataset stats by @jqnatividad in #190
• refactor: move pii-screening to a separate module by @jqnatividad in #191
• chore: add WIP geojson update by @rzmk in #186
• "smart" formula spatial functions by @jqnatividad in #192
• Jobs cleanup by @jqnatividad in #193
• Fix datastore upload log timestamps by @jqnatividad in #194
• DCAT 3 formula helpers by @jqnatividad in #195
• fix: tmp input was being wrongfully assigned by @jqnatividad in #197
• refactored SQL-enabled formulas by @jqnatividad in #199
• auto unzip one file setting by @jqnatividad in #200
• add LRU caches to potentially expensive Formula methods by @jqnatividad in #201
• feat: add dpp_suggestions.STATUS to track formulae processing progress by @jqnatividad in #202
• refactor dpp_suggestions.STATUS to sync with Suggestions UI by @jqnatividad in #203
• Refactor: remove stats & freq table save to datastore by @jqnatividad in #204

New Contributors

• @rzmk made their first contribution in #186

Full Changelog: https://github.com/dathere/datapusher-plus/compare/2.0.0...2.1.0

[2.0.0] - 2025-04-25

🎉 Data Resource Upload First (DRUF) Workflow is finally here! 🎉

A workflow that flips the old CKAN traditional data ingestion on its head.

• Instead of filling out the metadata first and then uploading the data, users upload data resources first
• In a few seconds, even for very large datasets, analysis and validation is done while precompiling statistical metadata
• This precompiled metadata are then used by Metadata Formulae defined in the scheming yaml files to either precompute other metadata fields (on both package & resource levels) or to offer metadata suggestions
• Metadata Formulae use the same powerful Jinja2 template engine that powers CKAN's templating system.
• It comes with an extensible library of Jinja2 filters/functions that can be used in Metadata Formulae ala Excel.

The DRUF reinvents CKAN data ingestion - by automatically calculating/suggesting "Automagical Metadata" - high-quality, high-resolution metadata that reflects and describes what's INSIDE the dataset (e.g. summary stats; frequency table; spatial extent, date range, outliers, etc. calculated with Metadata Formulae) in addition to metadata about the dataset FILE (e.g. last updated, size of the file, owner, format, license, etc - what's normally found in traditional data catalogs).

Future improvements planned:

• Expanded Data Dictionary

• "entry-time" Metadata Formulae In addition to the two formula types (formula to set a metadata field directly during creation/update; and suggestion_formula to suggest values using the Bootstap Popover UI), we'll add the ability to allow Data Publishers to enter formulas while they're entering metadata - fully embracing the Excel formula UI/UX aesthetic.

• DCAT3-optimized reference profiles Following implementation guidance for both DCAT-US v3 and DCAT-AP 3 scheming profiles with Metadata Formulae to compute recommended and optional properties that allow publishers to more fully take advantage of DCAT3 features and improvements - metadata properties that are often too laborious to manually compile.

• Co-Curator AI "Automagical metadata" is the perfect context for AI engines - as it summarizes even very large datasets in just a few kilobytes. It allows the Co-Curator¹ to suggest tags, descriptions, links to related data sets and chat about the corpus WHILE the Data Publisher is curating the data.

• Inline Data Validation Optional ability to infer an initial JSON Schema validation file, and then validate future updates to the dataset using it, leveraging the same blazing-fast qsv engine (validating up to 340,000 records/per second²).

• Customizable DRUF Data ingestion pipeline Currently, there are numerous configuration settings to fine-tune the DRUF data-ingestion pipeline. However, the built-in default pipeline can only be customized to a limit without customizing the code. We will expose hooks that CKAN operators can take advantage of to tailor their DRUF pipelines to meet their requirements, while preserving the ability to access the precompiled statistical metadata that DP+ maintains.

• Dynamic loading of Formula filters/functions So users can share custom Jinja2 filters and functions they developed for their Metadata Formulae.

• Inline Data Enrichment Data can be optionally enriched while it's being ingested from other reference datasets within the same CKAN instance or external sources (e.g. enriched against high value curated sources like the Census; geocoding, etc.)

• and more! It took a while for us to bake 2.0.0, but we look forward to picking up the pace and co-innovating with the CKAN ecosystem.

NOTE: To fully experience the DRUF workflow, you'll need to use scheming dataset form pages and apply some CKAN core changes. A detailed installation procedure will be published on the Wiki shortly.

---

Added

• Data Resource Upload First (DRUF) Workflow
  • Enhanced resource validation for DRUF workflow
  • Formulas for precomputing metadata/metadata sugggestions
  • Spatial file support - supports GeoJSON and Shapefiles
• Support for CKAN 2.9 compatibility in CLI operations
• Enhanced error handling and logging for resource uploads

Changed

• Updated CLI interface to work with CKAN 2.9
• Refactored resource upload process to support DRUF workflow
• Improved error messages and user feedback
• Enhanced configuration handling

Fixed

• Various bug fixes and improvements for CKAN 2.9 compatibility
• Resource upload process reliability improvements

Contributors

• @tino097
• @minhajuddin2510
• @jqnatividad

Full Changelog: https://github.com/dathere/datapusher-plus/compare/1.0.4...2.0.0

[1.0.4] - 2025-01-15

[1.0.3] - 2024-10-30

Changed

• Ensure we are always using the same token setting for datapusher
• Fix iconv
• Fix the api_token config variable and fix for default views creation
• Migration added

Contributors

• @tino097
• @avdata99

[1.0.2] - 2024-09-16

Changed

• Update README file for DP+ as extension
• Fix MANIFEST.in
• Migrate cli commands
• Fix init db command
• Config part
• Database migrations
• Update readme
• Fix yaml extension in MANIFEST.in
• Fix datefmt compatability with qsv in dev-v1.0
• Remove obsolete assets

Contributors

• @Zharktas
• @tino097
• @pdelboca

[1.0.1] - 2024-05-22

Changed

• Replace http requests with actions
• Fix calling package action for resource

Contributors

• @tino097

[1.0.0] - 2024-05-06

Added

• Convert the datapusher to work as plugin
• Feature db models
• Add migration script
• Rewrite resource URL if it differs from the defined ckan_url

Changed

• Code cleanup
• Rewrite resource url
• Sync with master

Contributors

• @jhbruhn
• @tino097
• @TomeCirun

[0.16.4] - 2024-01-23

Changed

• sync read buffer with buffer size of copyexpert

Contributors

• @jqnatividad

[0.16.3] - 2024-01-23

Changed

• make COPY_READBUFFER_SIZE a configurable parameter

Contributors

• @jqnatividad

[0.16.2] - 2024-01-23

Changed

• explicitly create a large read buffer when reading CSV when COPYing files to the datastore

[0.16.1] - 2024-01-15

Fixed

• fix utf8 encoding check, replacing NamedTemporaryFile approach, with Temporary Directory approach

Note

• Requires uchardet for the encoding check (apt-get install uchardet)

[0.16.0] - 2024-01-11

Added

• Update README.md
• Use a temporary directory to manage temporary files
• Utf8 conversion
• import syntax and ckanserviceprovider version
• upgrade container qsv to 0.118.0
• Fixed init of index_elapsed in case auto_index is off

Changed

• caught a missing variable

Contributors

• @bzar
• @categulario
• @hjhornbeck
• @EricSoroos
• @minhajuddin2510

[0.15.0] - 2023-06-26

Fixed

• removed DOWNLOAD_PREVIEW_ONLY as its unreliable with CSVs and corrupted Excel files
• removed SUMMARY_STATS_WITH_PREVIEW, which doesn't make sense without DOWNLOAD_PREVIEW_ONLY

[0.14.1] - 2023-06-26

Changed

• made a mistake publishing 0.14.0, neglecting to bump setup version
• add a note about using glibc-2.31 version of qsv if the Linux distro running it has an older version of GNU C library (e.g. Ubuntu 18.04 and Debian 11)

[0.14.0] - 2023-06-26

Added

• More robust file format detection, also now prompts the user to specify the file format if it cannot infer it from the server's content-type header or the file's extension

Changed

• Minimum QuickSilver version is now 0.108.0, which features a more robust and faster input command DP+ uses for transcoding and normalization of CSVs

Fixed

• Fixed file format detection
• Removed SNIFF_DELIMITER setting which was causing DP+ to periodically sniff non-comma delimiters even if the file was using comma delimiters

[0.13.2] - 2023-06-22

Fixed

• Added tzdata dependency and missing logging import

Contributors

• @minhajuddin2510

[0.13.1] - 2023-06-22

Changed

• Reordered imports for clarity
• Minor Download improvements to make streaming download more robust as we're doing streaming downloads when using preview rows by doing request in a with clause
• added vscode setting to use black formatter

Fixed

• added missing dependencies for pytz and python-dateutil. These new dependencies are required because of the fix in 0.13.0 that checked if a resource's metadata has been modified, allowing a DP+ job even if the file hash has not changed (e.g. when the Data Dictionary data types are changed and the user wants the resource file re-pushed to use the new data types)

[0.13.0] - 2023-06-16

Added

• Add unsafe headers configuration settings. This allows DP+ to use an alternate unsafe prefix when sanitizing column names
• Add SNIFF_DELIMITER setting. This allows DP+ to automatically infer the delimiter used by a CSV file if its not a comma
• The inferred Data Dictionary now also has a "Unit" column. Note that you'll still need to modify your CKAN theme to expose the Unit field in the Data Dictionary tab

Changed

• set minimum qsv version to 0.107.0

Fixed

• Allow url parameters. This allows DP+ to process links with URL parameters. Just be sure to specify the resource format to one of the supported DP+ formats so it will be processed
• Properly handle when there is no timezone info when checking if a resource is updated

[0.12.0] - 2023-05-19

Changed

• Use single source of configuration
• Containerfile dependencies

Fixed

• Don't crash when not given content-length header
• Use --prefer-dmy with qsv instead of --prefer_dmy
• Don't crash on missing original column name
• Allow reupload of file if resource metadata has changed
• Reset resource.preview_rows to False if existing resource falls below preview_rows threshold

Contributors

• @bluepython508

[0.11.0] - 2023-04-10

Added

• Added link to datapusher-plus docker
• Added uninstallation procedure
• Added more comments in the main jobs.py process where all the main work is done
• Added details about what qsv analysis enables

Changed

• Revamped documentation to streamline installation
• set config.py to more conservative defaults
• set minimum QSV version to 0.99.0

Fixed

• Container packaging fixes
• Fix error handling in validate
• pinned ckanserviceprovider to 1.1.0 and APScheduler to 3.9.1.post1

Contributors

• @Zharktas
• @EricSoroos
• @minhajuddin2510

[0.10.1] - 2023-02-03

Changed

• add separate AUTO_UNIQUE_INDEX setting
• improved Development Installation procedure
• improved Datapusher+ Configuration section, with heavily commented dot-env.template
• bumped qsv from 0.87.0 to 0.87.1, with improved safenames sanitizing
• added qsv version checks

[0.9.0] - 2023-01-30

Added

• Updated the readme to include locale installation
• Initial implementation of PII screening

Contributors

• @jqnatividad
• @minhajuddin2510

[0.8.0] - 2023-01-18

More detailed release notes forthcoming...

[0.7.0] - 2023-01-17

Fixed

• fix import of MutableMapping from collections.abc

Contributors

• @ctrepka

[0.6.0] - 2023-01-06

Added

• validate excel file exported CSVs as well, as they can potentially be invalid CSVs (e.g. differing column counts per row)
• support negative values for PREVIEW_ROWS to start previewing from the end of a file (e.g. -1000 = last 1000 rows)
• if an Excel file is invalid or password-protected, show additional file metadata by using the file command
• add PREFER_DMY setting for parsing dates and doing column date inferencing (otherwise, the default is YMD)
• add logic to DROP VIEWS if ALIAS_UNIQUE is false, and show warning on datastore log
• implement smart auto-indexing which is controlled by AUTO_INDEX_THRESHOLD (default: 3) and AUTO_INDEX_DATES (default: true)
• improved log messages (comma-separated formatting for numbers, context-sensitive normalizing/transcoding messages, etc.)
• applied Black formatter to jobs.py

Removed

• remove obsolete CHUNK_INSERT_ROWS setting as we now do Postgres COPY

[0.5.1] - 2023-01-05

Fixed

• Fixed "no data rows" bug

Added

• added more implementation comments and TODOS

[0.5.0] - 2023-01-04

Added

• new AUTO_ALIAS_UNIQUE setting with a default of false. This ensure the alias is stable if the resource is updated
• two-stage normalization/validation of incoming files, ensuring that we can gracefully handle corrupt files
• ensure column names are "safe" (e.g. valid postgresql column identifiers), modifying them as required - while still retaining the original "unsafe" name in the data dictionary

Changed

• updated deployment instructions

[0.4.0] - 2022-12-13

Added

• smart data dictionary
• "safe" column names handling
• uwsgi deployment fixed
• send the env file explicitly

Contributors

• @TomeCirun

[0.3.1] - 2022-12-09

Changed

• refactored log message right before qsv preprocessing starts

[0.3.0] - 2022-12-09

Changed

• spreadsheet files that are added as a link are parsed properly so long as the resource format is set
• header names are sanitized so they are valid Postgres column identifiers

Fixed

• wsgi deployment fixed

[0.2.0] - 2022-12-07

Changed

• fix UnboundLocalError
• Add datapusherplus config
• fix resource download
• delete settings.py

Contributors

• @TomeCirun

[0.1.0] - 2022-09-09

Added

• available smarter data type mapping to Postgres data types. By looking at the min/max values of a column, we can infer the best postgres data type - integer, bigint or numeric, instead of using the numeric Postgres type for all integers. This is done by changing TYPE_MAPPING of Integer from numeric to smartint.
• Add resource preview metadata fields:
  • preview - if the resource is a preview, and not the entire file, containing only the first PREVIEW_ROWS of the file (boolean)
  • preview_rows - the number of rows of the preview
  • total_record_count - the actual number of rows of the file

Changed

• change mapping of inferred Date fields to the Postgres date data type, instead of using Postgres timestamp data type for both Date (YYYY-MM-DD) and Datetime (YYYY-MM-DD HH:MM:SS TZ) columns.
• warn when duplicates are found, instead of info
• decreased default preview to 1,000 rows
• better error handling when calling qsv binary
• update instructions to use the latest qsv binary - qsv 0.67.0

Fixed

• trimmed header and column values when processing spreadsheets. As spreadsheets are more often than not, manually curated, there are often invisible whitespaces that "look" right that may cause invalid CSVs - e.g. column names with leading/trailing whitespaces that cause Postgres errors when columns are created using the Excel column name.

[0.0.23] - 2022-05-09

Changed

• use psycopg2-binary instead of psycopg2 to ease installation and eliminate need to have postgres dev files
• made logging messages auto-dedup aware if dupes are detected, by adding "unique" qualifier to record count
• pointed to the latest qsv version (0.46.1) with the excel off by 1 fix
• added note about nightly builds of qsv for maximum performance
• added note about additional DP+ supported Excel and TSV subformats
• use JOB_CONFIG consistently for setting DP+ settings
• made qsvdp the default QSV_BIN
• added note about how to install python 3.7 and above in DP+ virtual environment

Removed

• removed Hitchiker's guide quote from setup.py epilog
• removed six as DP+ requires at least python 3.7
• removed pytest step in Development installation until the tests are adapted to DP+

Fixed

• fixed development installation procedure, so no assumptions are made
• fixed production deployment procedure and made it more detailed
• fixed off by 1 error in excel export message in qsv

[0.0.21] - 2022-05-04

Added

• additional analysis & preparation steps enabled by qsv
  • blazing fast, guaranteed data type inferences with comprehensive descriptive statistics
  • configurable Excel/ODS exports (supports XLS, XLSX, ODS, XLSM, and XLSB formats)
  • automatic deduplication (#25)
  • RFC 4180 validation and UTF-8 transcoding of CSV files
  • smart date inferencing (#28)
  • automatic preview subset creation
  • DP+ optimized qsv binary (qsvdp) - which is 6x smaller than regular qsv, 3x smaller than qsvlite, and with the self-update engine removed.
• added an init_db command for initializing jobs_store db by @categulario (#3)
• automatic resource alias (aka Postgres view) creation (#26)
• added JOB_CONFIG environment variable by @categulario (#6)
• added package install by @categulario (#16)
• Postgres COPY to datastore
• additional data types inferred and changed postgres type mapping (String -> text; Float -> numeric; Integer -> numeric; Date -> timestamp; DateTime -> timestamp; NULL -> text)
• added requirement to create a Postgres application role/user with SUPERUSER privs to do native Postgres operations through psycopg2
• added more verbose logging with elapsed time per phase (Analysis & Preparation Phase, Copy Phase, Total elapsed time)
• added Containerfile reference by @categulario (#22)
• added datasize dependency for human-readable file sizes in messages
• added a Changelog using the Keep a Changelog template.
• published datapusher-plus on pypi
• added a GitHub action to publish a pypi package on release
• added a Roadmap Tracking EPIC, that will always be updated as DP+ evolves https://github.com/dathere/datapusher-plus/issue/5

Changed

• DP+ requires at least Python 3.7, as it needs CAPTURE_OUTPUT option in subprocess.run. It should still continue to work with CKAN<=2.8 as it runs in its own virtualenv.
• replaced messytables with qsv
• removed old "chunked" datastore inserts
• removed requirements.txt, requirements/dependencies are now in setup.py
• expanded documentation - rationale for DP+; addl config vars; modified installation procedure; scnshots; etc.
• improvements to install instructions by @categulario (#20)
• made DP+ a detached Datapusher fork to maintain own issue tracker, discussions, releases, etc.

Footnotes

1. Inspired by the Curator in Ready Player One ↩

2. validate_index benchmark - https://qsv.dathere.com/benchmarks ↩