Homogenize bin scripts

.credentials.template

@@ -0,0 +1,39 @@
+# All values in this files can be overriden by environment variables
+# with the same names, or by command line arguments.
+
+# Credentials for GitHub
+########################
+
+# First, create a GitHub app as described on https://develop.sentry.dev/integrations/github/.
+# This will give you the App ID and a private key you can save as a .pem file.
+# Then install the app on your account. You'll then find the installation ID
+# in the "Configure" URL listed on https://github.com/settings/installations.
+GITHUB_APP_ID=2876513
+GITHUB_PRIVATE_KEY_PATH=github.pem
+GITHUB_INSTALLATION_ID=120833184
+
+# Credentials for GitLab
+########################
+
+GITLAB_BASE_URL=https://gitlab.com
+# GitLab access token, from https://gitlab.com/-/user_settings/personal_access_tokens.
+GITLAB_ACCESS_TOKEN=
+
+# Credentials for RPC
+#####################
+SCM_RPC_SECRET=some-other-shared-secret-for-scm-etc
+
+# Default repositories
+######################
+
+# The GitLab project ID is a numeric identifier for your repository.
+# You can find it on the project page, in the "..." menu in the top right corner.
+GITLAB_PROJECT_ID=
+
+# For GitHub, the full repository name is used for bin/scm-*-client
+GITHUB_REPOSITORY_NAME={owner}/{repo}
+# but the repository ID is required for bin/sentry-rpc-github-client.
+# You can find it in the HTML code of the repository page, as:
+# <meta name="octolytics-dimension-repository_id" content="1234567890" />
+#                                                          ^^^^^^^^^^
+GITHUB_REPOSITORY_ID=

README.md

@@ -31,6 +31,105 @@ The SCM solves all the problems you don't want to care about.
 
 We have extensive documentation both inline in the Sentry codebase and on the [Sentry developer documentation](https://develop.sentry.dev/backend/source-code-management-platform/) portal. If you're interested in expanding your SCM usage or in enabling new service-providers for a limited amount of effort take a took at the SCM platform.
 
+### Interactive testing
+
+We provide a few command-line scripts for interactive testing.
+
+You'll need to create `.credentials` according to `.credentials.template` before using them.
+
+All `bin/*-client` have the same subcommands (see [`add_commands`](src/scm/private/cli_support.py) or use the `--help` command-line option). Some subcommands will fail if attempted against a provider that doesn't implement them.
+
+Rationale for having `bin/*-github-client` and `bin/*-gitlab-client`: it lets you configure your favorite test repository in `.credentials` for each provider.
+
+Rationale for having three different kinds of clients: it lets you use the following three modes to target the SCM hosts.
+
+#### Targetting SCM hosts directly
+
+```mermaid
+flowchart LR
+    bin/direct-github-client --> GitHub
+    bin/direct-gitlab-client --> GitLab
+```
+
+Example: `bin/direct-github-client get-repository | jq .data` produces something like:
+
+```
+DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443
+DEBUG:urllib3.connectionpool:https://api.github.com:443 "POST /app/installations/120833184/access_tokens HTTP/1.1" 201 323
+DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443
+DEBUG:urllib3.connectionpool:https://api.github.com:443 "GET /repos/jacquev6/test-Sentry-Integration-Dev-jacquev6 HTTP/1.1" 200 None
+{
+  "full_name": "jacquev6/test-Sentry-Integration-Dev-jacquev6",
+  "default_branch": "main",
+  "clone_url": "https://github.com/jacquev6/test-Sentry-Integration-Dev-jacquev6.git",
+  "private": false,
+  "size": 1
+}
+```
+
+#### Using the RCP, going through `bin/scm-rpc-server`:
+
+```mermaid
+flowchart LR
+    bin/scm-rpc-github-client --> bin/scm-rpc-server
+    bin/scm-rpc-gitlab-client --> bin/scm-rpc-server
+    bin/scm-rpc-server --> GitHub
+    bin/scm-rpc-server --> GitLab
+```
+
+Example: run `bin/scm-rpc-server` in a terminal, then `bin/scm-rpc-gitlab-client get-app-installation | jq .data` in another to produce something like:
+
+```
+DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8080
+DEBUG:urllib3.connectionpool:http://localhost:8080 "GET /api/0/internal/scm-rpc/ HTTP/1.1" 200 None
+DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8080
+DEBUG:urllib3.connectionpool:http://localhost:8080 "POST /api/0/internal/scm-rpc/ HTTP/1.1" 200 None
+{
+  "has_read_access": true,
+  "has_write_access": true
+}
+```
+
+#### Using the RPC, going through a Sentry development environment:
+
+```mermaid
+flowchart LR
+    bin/sentry-rpc-github-client --> sentry
+    bin/sentry-rpc-gitlab-client --> sentry
+    sentry --> GitHub
+    sentry --> GitLab
+```
+
+Example: with your sentry development environment running in a terminal (and the target repository configured via Sentry's "integrations" GUI), run `bin/sentry-rpc-github-client get-pull-request 2 | jq .data` from another terminal to produce something like:
+
+```
+DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8000
+DEBUG:urllib3.connectionpool:http://localhost:8000 "GET /api/0/internal/scm-rpc/ HTTP/1.1" 200 217
+DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8000
+DEBUG:urllib3.connectionpool:http://localhost:8000 "POST /api/0/internal/scm-rpc/ HTTP/1.1" 200 None
+{
+  "internal_id": "3329785233",
+  "id": "2",
+  "title": "Add blah",
+  "body": null,
+  "state": "open",
+  "merged": false,
+  "html_url": "https://github.com/jacquev6/test-Sentry-Integration-Dev-jacquev6/pull/2",
+  "head": {
+    "sha": "7497e018d01503b6abc3053b7896266115e631f6",
+    "ref": "topics/blah"
+  },
+  "base": {
+    "sha": "0941ee0a9eac9914cfddf5adec7a9558a2f1c447",
+    "ref": "main"
+  },
+  "author": {
+    "id": "327146",
+    "username": "jacquev6"
+  }
+}
+```
+
 # Releasing a New Version
 
 1. On the `getsentry/scm-platform` repository page click the `Actions` tab.

bin/direct-github-client

@@ -0,0 +1,120 @@
+#!/usr/bin/env python
+import argparse
+import logging
+import sys
+
+from scm.manager import SourceCodeManager
+from scm.private.cli_support import (
+    GitHubApiClient,
+    GitHubInstallationTokenManager,
+    add_commands,
+    execute_command,
+    load_credentials,
+    resolve,
+)
+from scm.providers.github.provider import GitHubProvider
+from scm.rpc.client import NoOpRateLimitProvider
+from scm.types import Provider, Repository, RepositoryId
+
+logger = logging.getLogger(__name__)
+
+
+def fetch_repository(organization_id: int, repository_id: RepositoryId) -> Repository | None:
+    if isinstance(repository_id, (list, tuple)):
+        provider_name, external_id = repository_id
+        return {
+            "id": 1,
+            "external_id": external_id,
+            "integration_id": 1,
+            "is_active": True,
+            "name": external_id,
+            "organization_id": organization_id,
+            "provider_name": provider_name,
+        }
+    return None
+
+
+def make_fetch_provider(github_client: GitHubApiClient):
+    def fetch_provider(organization_id: int, repository: Repository) -> Provider | None:
+        return GitHubProvider(
+            client=github_client,
+            organization_id=organization_id,
+            repository=repository,
+            rate_limit_provider=NoOpRateLimitProvider(),
+        )
+
+    return fetch_provider
+
+
+def main():
+    creds = load_credentials()
+
+    parser = argparse.ArgumentParser(description="Direct client for GitHub repositories")
+
+    parser.add_argument(
+        "--github-app-id",
+        default=None,
+        help="GitHub App ID. Defaults to GITHUB_APP_ID from .credentials",
+    )
+    parser.add_argument(
+        "--github-private-key",
+        default=None,
+        help="Path to the GitHub App private key. Defaults to GITHUB_PRIVATE_KEY_PATH from .credentials",
+    )
+    parser.add_argument(
+        "--github-installation-id",
+        default=None,
+        help="GitHub App installation ID. Defaults to GITHUB_INSTALLATION_ID from .credentials",
+    )
+
+    parser.add_argument("--org-id", type=int, default=1, help="Sentry organization ID owning the repository")
+    parser.add_argument(
+        "--repo",
+        default=creds.get("GITHUB_REPOSITORY_NAME"),
+        required="GITHUB_REPOSITORY_NAME" not in creds,
+        help="GitHub repository as 'owner/repo'. Defaults to GITHUB_REPOSITORY_NAME from .credentials",
+    )
+
+    add_commands(parser)
+
+    args = parser.parse_args()
+    if not args.command:
+        parser.print_help()
+        sys.exit(1)
+
+    github_app_id = resolve("GITHUB_APP_ID", args.github_app_id, creds)
+    github_private_key_path = resolve("GITHUB_PRIVATE_KEY_PATH", args.github_private_key, creds)
+    github_installation_id = resolve("GITHUB_INSTALLATION_ID", args.github_installation_id, creds)
+
+    missing = []
+    if not github_app_id:
+        missing.append("GITHUB_APP_ID")
+    if not github_private_key_path:
+        missing.append("GITHUB_PRIVATE_KEY_PATH")
+    if not github_installation_id:
+        missing.append("GITHUB_INSTALLATION_ID")
+    if missing:
+        logger.error(f"Error: missing required config: {', '.join(missing)}")
+        sys.exit(1)
+
+    with open(github_private_key_path) as f:
+        github_private_key = f.read()
+
+    github_client = GitHubApiClient(
+        GitHubInstallationTokenManager(github_app_id, github_private_key, github_installation_id)
+    )
+
+    scm = SourceCodeManager.make_client(
+        args.org_id,
+        ("github", args.repo),
+        fetch_repository=fetch_repository,
+        fetch_provider=make_fetch_provider(github_client),
+        record_count=lambda name, value, tags: None,
+    )
+
+    execute_command(args, scm)
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.DEBUG)
+    main()

bin/direct-gitlab-client

@@ -0,0 +1,108 @@
+#!/usr/bin/env python
+import argparse
+import logging
+import sys
+from urllib.parse import urlparse
+
+from scm.manager import SourceCodeManager
+from scm.private.cli_support import (
+    GitLabApiClient,
+    add_commands,
+    execute_command,
+    load_credentials,
+    resolve,
+)
+from scm.providers.gitlab.provider import GitLabProvider
+from scm.types import Provider, Repository, RepositoryId
+
+logger = logging.getLogger(__name__)
+
+
+def fetch_repository(organization_id: int, repository_id: RepositoryId) -> Repository | None:
+    if isinstance(repository_id, (list, tuple)):
+        provider_name, external_id = repository_id
+        return {
+            "id": 1,
+            "external_id": external_id,
+            "integration_id": 1,
+            "is_active": True,
+            "name": external_id,
+            "organization_id": organization_id,
+            "provider_name": provider_name,
+        }
+    return None
+
+
+def make_fetch_provider(gitlab_client: GitLabApiClient):
+    def fetch_provider(organization_id: int, repository: Repository) -> Provider | None:
+        return GitLabProvider(
+            client=gitlab_client,
+            organization_id=organization_id,
+            repository=repository,
+        )
+
+    return fetch_provider
+
+
+def main():
+    creds = load_credentials()
+
+    parser = argparse.ArgumentParser(description="Direct client for GitLab repositories")
+
+    parser.add_argument(
+        "--gitlab-base-url",
+        default=None,
+        help="Base URL for the GitLab instance. Defaults to GITLAB_BASE_URL from .credentials",
+    )
+    parser.add_argument(
+        "--gitlab-access-token",
+        default=None,
+        help="GitLab access token. Defaults to GITLAB_ACCESS_TOKEN from .credentials",
+    )
+
+    parser.add_argument("--org-id", type=int, default=1, help="Sentry organization ID owning the repository")
+    parser.add_argument(
+        "--repo",
+        default=creds.get("GITLAB_PROJECT_ID"),
+        required="GITLAB_PROJECT_ID" not in creds,
+        help=(
+            "GitLab project ID (numeric, found in the top-right '...' menu on the GitLab project page)."
+            " Defaults to GITLAB_PROJECT_ID from .credentials"
+        ),
+    )
+
+    add_commands(parser)
+
+    args = parser.parse_args()
+    if not args.command:
+        parser.print_help()
+        sys.exit(1)
+
+    gitlab_base_url = resolve("GITLAB_BASE_URL", args.gitlab_base_url, creds)
+    gitlab_access_token = resolve("GITLAB_ACCESS_TOKEN", args.gitlab_access_token, creds)
+
+    missing = []
+    if not gitlab_access_token:
+        missing.append("GITLAB_ACCESS_TOKEN")
+    if not gitlab_base_url:
+        missing.append("GITLAB_BASE_URL")
+    if missing:
+        logger.error(f"Error: missing required config: {', '.join(missing)}")
+        sys.exit(1)
+
+    gitlab_client = GitLabApiClient(gitlab_base_url, gitlab_access_token)
+
+    scm = SourceCodeManager.make_client(
+        args.org_id,
+        ("gitlab", f"{urlparse(gitlab_base_url).netloc}:{args.repo}"),
+        fetch_repository=fetch_repository,
+        fetch_provider=make_fetch_provider(gitlab_client),
+        record_count=lambda name, value, tags: None,
+    )
+
+    execute_command(args, scm)
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.DEBUG)
+    main()