kcp scan clusters

kcp scan clusters¶

Scan Kafka clusters using the Kafka Admin API

Synopsis¶

Scan MSK or Apache Kafka clusters — including Confluent Platform and other Kafka-API-compatible distributions — to discover topics, ACLs, and other metadata via the Kafka Admin API. Results are merged into the kcp-state.json file.

Source-specific notes:

--source-type msk reads cluster connection details from the msk-credentials.yaml file produced by kcp discover. Enable exactly one authentication method per cluster in that file before scanning. SCRAM is forced to SHA-512 (the only mechanism MSK supports).
--source-type apache-kafka reads from a hand-authored apache-kafka-credentials.yaml file. SASL/SCRAM defaults to SHA-256 — set auth_method.sasl_scram.mechanism: SHA512 if your cluster requires SHA-512. The full schema and worked examples are documented at Apache Kafka configuration → Credentials.

For the step-by-step workflow from discovery to viewing the results, see Getting started.

Consumer groups:

Consumer group discovery runs by default; pass --skip-consumer-groups to turn it off. For every group KCP records its KIP-848 type (classic, consumer, share, streams), state, and coordinator, and merges them into the state file.
Reading the group type requires a broker running Kafka 3.8 or newer; against older brokers the type is reported as blank and every group is treated as classic-protocol.
Only classic groups are fully described (members and their assigned topics). consumer, share, and streams groups record type, state, and coordinator but not member-level detail (that needs the newer ConsumerGroupDescribe API); they are flagged with incomplete detail in the state file and UI.

Metrics collection (Apache Kafka only):

--metrics jolokia polls each broker's Jolokia HTTP endpoint live for the duration set by --metrics-duration (interval: --metrics-interval, default 10s).
--metrics prometheus queries a Prometheus server for historical metrics over --metrics-range (e.g. 7d, 30d).

Both backends produce the same metric shape and feed reports and the UI. See Apache Kafka configuration → Metrics collection for the metric list, the counter-based rate calculation, and authentication options.

kcp scan clusters [flags]

Examples¶

  # Scan an MSK cluster (credentials from kcp discover)
  kcp scan clusters --source-type msk --state-file kcp-state.json --credentials-file msk-credentials.yaml

  # Scan an Apache Kafka cluster (hand-authored credentials)
  kcp scan clusters --source-type apache-kafka --state-file kcp-state.json --credentials-file apache-kafka-credentials.yaml

  # Apache Kafka with live Jolokia metric collection
  kcp scan clusters --source-type apache-kafka --state-file kcp-state.json \
      --credentials-file apache-kafka-credentials.yaml \
      --metrics jolokia --metrics-duration 5m --metrics-interval 10s

  # Apache Kafka with historical Prometheus metrics
  kcp scan clusters --source-type apache-kafka --state-file kcp-state.json \
      --credentials-file apache-kafka-credentials.yaml \
      --metrics prometheus --metrics-range 30d

Options¶

      --credentials-file string   Path to credentials file (msk-credentials.yaml or apache-kafka-credentials.yaml)
  -h, --help                      help for clusters
      --metrics string            Metrics collection source: 'jolokia' or 'prometheus' (Apache Kafka only)
      --metrics-duration string   Duration to poll Jolokia (e.g. 10m, 1h). Required with --metrics jolokia.
      --metrics-interval string   Polling interval for Jolokia (e.g. 10s, 30s). Default: 10s. (default "10s")
      --metrics-range string      Day range to query from Prometheus (e.g. 7d, 30d). Required with --metrics prometheus.
      --skip-acls                 Skip ACL discovery
      --skip-consumer-groups      Skip consumer group discovery
      --skip-topics               Skip topic discovery
      --source-type string        Source type: 'msk' or 'apache-kafka' (required)
      --state-file string         Path to the KCP state file (default "kcp-state.json")

Options inherited from parent commands¶

      --verbose   Enable verbose logging to console

AWS IAM Permissions¶

Only required for --source-type msk. Apache Kafka scans use credentials from the credentials file, not AWS IAM.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "MSKClusterKafkaAccess",
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeCluster",
        "kafka-cluster:DescribeClusterDynamicConfiguration",
        "kafka-cluster:DescribeGroup",
        "kafka-cluster:DescribeTopic"
      ],
      "Resource": [
        "arn:aws:kafka:<AWS REGION>:<AWS ACCOUNT ID>:topic/<MSK CLUSTER NAME>/<MSK CLUSTER ID>/*",
        "arn:aws:kafka:<AWS REGION>:<AWS ACCOUNT ID>:group/<MSK CLUSTER NAME>/<MSK CLUSTER ID>/*",
        "arn:aws:kafka:<AWS REGION>:<AWS ACCOUNT ID>:cluster/<MSK CLUSTER NAME>/<MSK CLUSTER ID>"
      ]
    }
  ]
}

kcp scan clusters