Skip to content

Document fallback field for Kafka topic name sources #211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dmzmk wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Document fallback field for Kafka topic name sources #211

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/sharing-the-cluster/queue-splitting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -474,11 +474,13 @@ spec:
- directEnvVar:
container: consumer
variable: KAFKA_TOPIC_NAME
fallback: views-topic # optional, used when the variable is absent
```

The topics consumer resource above says that:
1. It provides context for deployment `meme-app` in namespace `meme`.
2. The deployment consumes one queue. Its name is read from environment variable `KAFKA_TOPIC_NAME` in container `consumer`.
If the variable is absent, the fallback value `views-topic` is used instead.
The Kafka consumer group id is read from environment variable `KAFKA_GROUP_ID` in container `consumer`.
3. The Kafka queue can be referenced in a mirrord config under ID `views-topic`.

Expand All @@ -503,8 +505,13 @@ The queues are described in entries of the `spec.topics` list:
* `id` can be arbitrary, as it will only be [referenced](queue-splitting.md#setting-a-filter-for-a-mirrord-run) from the user's mirrord config.
* `clientConfig` stores the name of the `MirrordKafkaClientConfig` to use when making connections to the Kafka cluster.
* `nameSources` stores a list of all occurrences of the queue name in the consumer workload's pod template.
* `groupIdSources` stores a list of all occurrences of the consumer Kafka group ID in the consumer workload's pod template.
* `groupIdSources` (required) stores a list of all occurrences of the consumer Kafka group ID in the consumer workload's pod template.
The operator will use the same group ID when consuming messages from the queue.
When using `applicationIdSources` instead (Kafka Streams consumers), set this to an empty list `[]`.
* `applicationIdSources` stores a list of all occurrences of the consumer Kafka Streams application ID in the consumer workload's pod template.
Relevant only for Kafka Streams consumers. Must be empty if `groupIdSources` is non-empty.

Each source entry supports an optional `fallback` value, used when the specified variable is absent from the workload.

{% hint style="warning" %}
The mirrord operator can only read consumer's environment variables if they are either:
Expand Down