`apache-kafka-credentials.yaml`¶

The Apache Kafka® source has no automated discovery step — instead, you hand-author an apache-kafka-credentials.yaml describing how kcp should connect to your cluster(s). It is consumed by kcp scan clusters --source-type apache-kafka and any downstream kcp iac commands that target an Apache Kafka source.

This covers self-managed Apache Kafka, Confluent Platform, and other Kafka-API-compatible distributions.

For comparison, the MSK equivalent (msk-credentials.yaml) is generated for you by kcp discover — there is no MSK counterpart to this file because MSK metadata comes from the AWS APIs.

Minimal example¶

clusters:
  - id: prod-kafka
    bootstrap_servers:
      - broker1.example.com:9092
      - broker2.example.com:9092
    auth_method:
      sasl_scram:
        use: true
        username: admin
        password: changeme
        mechanism: SHA256

SASL over TLS with a private CA¶

clusters:
  - id: private-ca-kafka
    bootstrap_servers:
      - broker1.example.com:9093
    auth_method:
      sasl_scram:
        use: true
        username: admin
        password: changeme
        mechanism: SHA512
        ca_cert: /path/to/ca.pem

Full example — multiple clusters, mixed auth, metrics¶

# Apache Kafka Credentials Configuration
# Configure your Apache Kafka cluster connection details.

clusters:
  # Production cluster: SASL/SCRAM + Jolokia metrics collection.
  - id: production-kafka-us-east
    bootstrap_servers:
      - broker1.prod.example.com:9092
      - broker2.prod.example.com:9092
      - broker3.prod.example.com:9092
    auth_method:
      sasl_scram:
        use: true
        username: admin
        password: changeme
        mechanism: SHA256
    jolokia:
      endpoints:
        - http://broker1.prod.example.com:8778/jolokia
        - http://broker2.prod.example.com:8778/jolokia
        - http://broker3.prod.example.com:8778/jolokia
      auth:
        username: monitorRole
        password: monitorPass
    # Alternative: query Prometheus instead of polling Jolokia.
    # prometheus:
    #   url: http://prometheus.prod.example.com:9090
    #   auth:
    #     username: promuser
    #     password: prompass
    metadata:
      environment: production
      location: us-datacenter-1

  # Staging cluster: mTLS.
  - id: staging-kafka
    bootstrap_servers:
      - broker1.staging.example.com:9093
    auth_method:
      tls:
        use: true
        ca_cert: /path/to/ca-cert.pem
        client_cert: /path/to/client-cert.pem
        client_key: /path/to/client-key.pem
    metadata:
      environment: staging

  # Dev cluster: no auth (test environments only).
  - id: dev-kafka
    bootstrap_servers:
      - localhost:9092
    auth_method:
      unauthenticated_plaintext:
        use: true
    metadata:
      environment: development

Field reference¶

Top-level¶

Field	Type	Required	Description
`clusters`	list	yes	One or more cluster entries — `kcp` scans each one.

Per-cluster¶

Field	Type	Required	Description
`id`	string	yes	Unique identifier for the cluster. Used as the cluster key in `kcp-state.json` and as the `--cluster-id` value for downstream `kcp iac` commands.
`bootstrap_servers`	list of strings	yes	Broker `host:port` addresses.
`auth_method`	object	yes	Authentication method. Choose exactly one sub-block (see below).
`insecure_skip_tls_verify`	bool	no	Disables server certificate verification for this cluster's Kafka connection. Does not affect `jolokia` or `prometheus`, which have their own `tls.insecure_skip_verify`. With `sasl_plain` it only applies when TLS is on (`ca_cert` or `tls: true`).
`jolokia`	object	no	Jolokia HTTP endpoints for live JMX metrics. Required if you pass `--metrics jolokia` on `kcp scan clusters` or `kcp scan self-managed-connectors`. For Connect metrics, point endpoints to Connect workers rather than brokers.
`prometheus`	object	no	Prometheus HTTP API for historical metrics. Required if you pass `--metrics prometheus` on `kcp scan clusters` or `kcp scan self-managed-connectors`.
`metadata`	map	no	Free-form labels surfaced in reports and the UI (e.g. `environment`, `location`).

`auth_method` — pick one¶

Sub-block	Use case	Required fields
`sasl_scram`	SASL/SCRAM-SHA-256 or SHA-512 over TLS	`use: true`, `username`, `password`, `mechanism` (`SHA256`/`SHA512`); optional `ca_cert`
`sasl_plain`	SASL/PLAIN, plaintext or over TLS	`use: true`, `username`, `password`; optional `ca_cert` or `tls: true`
`tls`	TLS / mTLS with client certs	`use: true`, `ca_cert`, `client_cert`, `client_key`
`unauthenticated_tls`	TLS with no client authentication	`use: true`; optional `ca_cert`
`unauthenticated_plaintext`	No auth (test environments only)	`use: true`

TLS for SASL blocks

sasl_scram always connects over TLS. ca_cert names the CA to trust; without it, kcp uses the system trust store. Set it when the brokers present a certificate from a private CA.

sasl_plain connects in plaintext unless TLS is selected: set ca_cert for a private CA, or tls: true for a certificate signed by a public CA (system trust store).

SCRAM mechanism for Apache Kafka vs MSK

Apache Kafka supports both SHA256 and SHA512. SHA256 is the more common default, so kcp does not infer one for you — set mechanism explicitly.

AWS MSK only supports SHA512; kcp discover sets that automatically when generating msk-credentials.yaml.

`jolokia` — optional, for live metrics¶

jolokia:
  endpoints:
    - http://broker1:8778/jolokia
    - http://broker2:8778/jolokia
  auth:                         # optional — omit for unauthenticated Jolokia
    username: monitorRole
    password: secret
  tls:                          # optional — omit for plain HTTP
    ca_cert: /path/to/ca.pem
    insecure_skip_verify: false
  mbean_overrides:              # optional — see "Metric-name overrides" below
    BytesInPerSec: "acme.kafka:type=BrokerTopicMetrics,name=BytesInPerSec"
  connect_mbean_overrides:      # optional — for Connect worker scans; see connect-metrics-collection.md
    task-count: "acme.connect:type=connect-worker-metrics"

Field	Required	Description
`endpoints`	yes	List of Jolokia HTTP endpoints — one per broker.
`auth.username` / `password`	no	HTTP basic auth credentials.
`tls.ca_cert`	no	CA certificate for HTTPS Jolokia endpoints.
`tls.insecure_skip_verify`	no	Skip TLS verification (test environments only).
`mbean_overrides`	no	Map of logical metric label → MBean object name for agents that expose non-standard MBean names. See Metric-name overrides.
`connect_mbean_overrides`	no	Map of logical Connect-metric label → MBean object name, for `kcp scan self-managed-connectors` scans of Kafka Connect workers. See Connect metric-name overrides.

`prometheus` — optional, for historical metrics¶

prometheus:
  url: http://prometheus:9090
  auth:                         # optional — omit for unauthenticated
    username: promuser
    password: prompass
  tls:                          # optional — omit for plain HTTP
    ca_cert: /path/to/ca.pem
    insecure_skip_verify: false
  filter:                       # optional — scope queries to a specific target
    labels:
      job: confluent/kafka-jmx-exporter
  timeout: 60s                  # optional — HTTP client timeout per query (omit for 30s default; minimum 30s)
  metric_names:                 # optional — see "Metric-name overrides" below
    BytesInPerSec: acme_broker_bytesin_total
    MessagesInPerSec: acme_broker_messagesin_total
  connect_metric_names:         # optional — for Connect worker scans; see connect-metrics-collection.md
    task-count: acme_connect_task_count

Field	Required	Description
`url`	yes	Prometheus server URL.
`auth.username` / `password`	no	HTTP basic auth credentials.
`tls.ca_cert`	no	CA certificate for HTTPS Prometheus endpoints.
`tls.insecure_skip_verify`	no	Skip TLS verification (test environments only).
`filter.labels`	no	Map of Prometheus label selectors to scope queries. When set, all PromQL queries include these as `{key="value"}` filters. Useful when a single Prometheus scrapes multiple clusters.
`timeout`	no	HTTP client timeout per Prometheus query, as a Go duration (e.g. `60s`, `2m`). Omit to use the `30s` default; an explicit value must be at least `30s` (shorter values are rejected). Raise this for large `--metrics-range` queries against high-cardinality clusters that respond slowly.
`metric_names`	no	Map of logical metric label → base Prometheus series name for exporters that relabel the standard series. See Metric-name overrides.
`connect_metric_names`	no	Map of logical Connect-metric label → base Prometheus series name, for `kcp scan self-managed-connectors` scans of Kafka Connect workers. See Connect metric-name overrides.

jolokia and prometheus are mutually exclusive per scan invocation — --metrics selects which one kcp reads. You can keep both blocks in the file and switch between them by changing the flag.

Metric-name overrides¶

If your Prometheus exporter relabels the standard broker series, or your Jolokia agent exposes broker MBeans under non-standard object names, the built-in queries return empty results. Rather than patching those names in source, repoint them per cluster: prometheus.metric_names maps a logical label to the base series name your exporter exposes, and jolokia.mbean_overrides maps a logical label to the MBean object name your agent exposes.

Both key on the same seven logical labels:

BytesInPerSec, BytesOutPerSec, MessagesInPerSec, PartitionCount, GlobalPartitionCount, ClientConnectionCount, TotalLocalStorageUsage.

For example, if your Prometheus JMX Exporter publishes the byte-rate series under an acme_ prefix and drops the _total suffix, map just the labels that changed — everything you omit keeps its default:

prometheus:
  url: http://prometheus.example.com:9090
  metric_names:
    BytesInPerSec: acme_kafka_bytesin     # default: kafka_server_brokertopicmetrics_bytesinpersec_total
    BytesOutPerSec: acme_kafka_bytesout   # default: kafka_server_brokertopicmetrics_bytesoutpersec_total

kcp slots the name into its own wrapping, so it now runs sum(rate(acme_kafka_bytesin[<window>])) in place of the default. The Jolokia equivalent maps the same logical label to a full MBean object name (domain included) rather than a series name:

jolokia:
  endpoints:
    - http://broker1.example.com:8778/jolokia
  mbean_overrides:
    BytesInPerSec: "acme.kafka:type=BrokerTopicMetrics,name=BytesInPerSec"   # default domain: kafka.server

Only labels you need to change must appear; unlisted labels keep their defaults. A key with an empty value (BytesInPerSec: "") is treated as no override and silently ignored — set a real name or omit the key entirely.
Keys are validated at load time — an unknown or misspelled label (wrong case included) is a hard error listing the valid labels, not a silent no-op. Both blocks are validated whenever the file loads, regardless of which one --metrics selects, so a typo in the block you are not currently scanning with still fails the scan.
Prometheus overrides replace the base series name only. kcp keeps its own wrapping (sum(rate(<name>[<window>])), sum(<name>), the GiB conversion) and filter.labels injection, so the override is a rename, not a full-query rewrite. GlobalPartitionCount is the one exception worth calling out explicitly: on the default series (kafka_controller_kafkacontroller_value) kcp appends a {name="GlobalPartitionCount"} discriminator, because several controller values share that one series name. Once you override GlobalPartitionCount, the discriminator is dropped — your override already identifies the series on its own, and most relabelled/flattened exporters expose it with no name label at all. If your override value already carries its own selector (e.g. a job-scoped series), it is used exactly as given.
If an overridden metric still returns no data, kcp logs it at WARN (a plain missing default is logged at DEBUG) — the override was configured precisely to fix an empty result, so a still-empty result is worth surfacing.
An override is a rename, not a re-interpretation. The series or bean you point at must have the same shape as the default — a byte counter for the *PerSec rates, a gauge for the counts, raw bytes for storage. Repointing at a rate gauge or a differently-united series yields a wrong value with no error, not an empty result. See What an override does not change.

Kafka Connect worker metrics have their own analogous pair — prometheus.connect_metric_names and jolokia.connect_mbean_overrides — keyed on the ten Connect metric labels rather than the seven broker ones. See Connect metric-name overrides for the label list, worked examples, and the per-connector caveats specific to Connect.

Where to go next¶

kcp scan clusters — pass this file with --credentials-file.
Metrics collection — design notes on Jolokia vs Prometheus, the broker metrics that kcp records, and how rates are computed.
Connect metrics collection — collecting metrics from Kafka Connect workers.

apache-kafka-credentials.yaml¶