tap-salesforce

Singer tap that extracts data from a Salesforce database and produces JSON-formatted data following the Singer spec.

$ mkvirtualenv -p python3 tap-salesforce
$ pip install tap-salesforce
$ tap-salesforce --config config.json --discover
$ tap-salesforce --config config.json --properties properties.json --state state.json

Quickstart

Install the tap

> pip install tap-salesforce

Create a Config file

{
  "client_id": "secret_client_id",
  "client_secret": "secret_client_secret",
  "refresh_token": "abc123",
  "start_date": "2017-11-02T00:00:00Z",
  "api_type": "BULK",
  "select_fields_by_default": true
}

The client_id and client_secret keys are your OAuth Salesforce App secrets. The refresh_token is a secret created during the OAuth flow. For more info on the Salesforce OAuth flow, visit the Salesforce documentation. Additionnaly, if the Salesforce Sandbox is to be used to run the tap, the parameter "is_sandbox": true must be passed to the config.

The start_date is used by the tap as a bound on SOQL queries when searching for records. This should be an RFC3339 formatted date-time, like "2018-01-08T00:00:00Z". For more details, see the Singer best practices for dates.

The api_type is used to switch the behavior of the tap between using Salesforce's "REST" and "BULK" APIs. When new fields are discovered in Salesforce objects, the select_fields_by_default key describes whether or not the tap will select those fields by default.

Run Discovery

To run discovery mode, execute the tap with the config file.

> tap-salesforce --config config.json --discover > properties.json

Sync Data

To sync data, select fields in the properties.json output and run the tap.

> tap-salesforce --config config.json --properties properties.json [--state state.json]

Syncing Reports

There are three ways to sync reports using this tap:

1. Contact and Lead filters

You can add the following to your config.json:

"report_ids": ["00OK...", "00O3..."]

This will filter your leads + contacts streams to only leads and contacts that are in this particular report

2. Syncing raw report payloads

You can add the following to your config.json:

"list_reports": true
"report_ids": ["00OK...", "00O3..."]
"report_names": ["Email Sends by User over Time", "Leads Report"]

During the discover process, the tap will add a stream named ReportList where each field is a different report which is queryable. If you pass the report_ids and/or report_names fields, then only configured reports will be added to the ReportList stream.

The selected fields of this stream will each come through as their own stream when running a sync.

Note that using list_reports in this way can significantly slow down the discover process on SF instances with large amounts of reports.

Additionally, SF limits the number of records per report to 2000 when syncing through this method.

3. Exporting reports via Excel

If you want to get past the 2k record limit, you can configure:

"list_reports": true,
"discover_report_fields": true,
"report_ids": ["00OK...", "00O3..."]
"report_names": ["Email Sends by User over Time", "Leads Report"]

This will:

1. Add a stream to the catalog for each configured report
2. Sync the report via an excel import which is slower, but does not have a record count limitation.

---

Development

Setup

Clone the repository and install development dependencies:

git clone <repository-url>
cd tap-salesforce
pip install -e '.'

Pre-commit Hooks

Important: After cloning, install pre-commit hooks:

pre-commit install                      # Runs tests before commits
pre-commit install --hook-type pre-push # Runs tests before pushes

This automatically runs tests before commits and pushes. See PRE_COMMIT_SETUP.md for troubleshooting.

Running Tests

# Run all tests
pytest

# Run with verbose output
pytest -v

# Run specific test file
pytest tests/test_sync_report_via_excel.py

# Run specific test
pytest tests/test_sync_report_via_excel.py::TestSyncReportViaExcel::test_basic_report_sync

Test Fixtures

Test fixtures are organized in tests/fixtures/:

• fixtures/streams/report/ - Report-specific test data
• fixtures/streams/products/ - Products-specific test data

See tests/FIXTURES_GUIDE.md for details on using and creating fixtures.

---

Copyright © 2017 Stitch