kcp scan clusters

kcp scan clusters¶

Scan Kafka clusters using the Kafka Admin API

Synopsis¶

Scan MSK or Apache Kafka clusters to discover topics, ACLs, and other metadata via the Kafka Admin API. Results are merged into the kcp-state.json file.

Source-specific notes:

--source-type msk reads cluster connection details from the msk-credentials.yaml file produced by kcp discover. SCRAM is forced to SHA-512 (the only mechanism MSK supports).
--source-type apache-kafka reads from a hand-authored apache-kafka-credentials.yaml file. SASL/SCRAM defaults to SHA-256 — set auth_method.sasl_scram.mechanism: SHA512 if your cluster requires SHA-512. The full schema and worked examples are documented at Apache Kafka configuration → Credentials.

Metrics collection (Apache Kafka only):

--metrics jolokia polls each broker's Jolokia HTTP endpoint live for the duration set by --metrics-duration (interval: --metrics-interval, default 10s).
--metrics prometheus queries a Prometheus server for historical metrics over --metrics-range (e.g. 7d, 30d).

Both backends produce the same metric shape and feed reports and the UI. See Apache Kafka configuration → Metrics collection for the metric list, the counter-based rate calculation, and authentication options.

kcp scan clusters [flags]

Examples¶

  # Scan an MSK cluster (credentials from kcp discover)
  kcp scan clusters --source-type msk --state-file kcp-state.json --credentials-file msk-credentials.yaml

  # Scan an Apache Kafka cluster (hand-authored credentials)
  kcp scan clusters --source-type apache-kafka --state-file kcp-state.json --credentials-file apache-kafka-credentials.yaml

  # Apache Kafka with live Jolokia metric collection
  kcp scan clusters --source-type apache-kafka --state-file kcp-state.json \
      --credentials-file apache-kafka-credentials.yaml \
      --metrics jolokia --metrics-duration 5m --metrics-interval 10s

  # Apache Kafka with historical Prometheus metrics
  kcp scan clusters --source-type apache-kafka --state-file kcp-state.json \
      --credentials-file apache-kafka-credentials.yaml \
      --metrics prometheus --metrics-range 30d

Options¶

      --credentials-file string   Path to credentials file (msk-credentials.yaml or apache-kafka-credentials.yaml)
  -h, --help                      help for clusters
      --metrics string            Metrics collection source: 'jolokia' or 'prometheus' (Apache Kafka only)
      --metrics-duration string   Duration to poll Jolokia (e.g. 10m, 1h). Required with --metrics jolokia.
      --metrics-interval string   Polling interval for Jolokia (e.g. 10s, 30s). Default: 10s. (default "10s")
      --metrics-range string      Day range to query from Prometheus (e.g. 7d, 30d). Required with --metrics prometheus.
      --skip-acls                 Skip ACL discovery
      --skip-topics               Skip topic discovery
      --source-type string        Source type: 'msk' or 'apache-kafka' (required)
      --state-file string         Path to the KCP state file (default "kcp-state.json")

Options inherited from parent commands¶

      --verbose   Enable verbose logging to console

AWS IAM Permissions¶

Only required for --source-type msk. Apache Kafka scans use credentials from the credentials file, not AWS IAM.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "MSKClusterKafkaAccess",
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeCluster",
        "kafka-cluster:DescribeClusterDynamicConfiguration",
        "kafka-cluster:DescribeTopic"
      ],
      "Resource": [
        "arn:aws:kafka:<AWS REGION>:<AWS ACCOUNT ID>:topic/<MSK CLUSTER NAME>/<MSK CLUSTER ID>/*",
        "arn:aws:kafka:<AWS REGION>:<AWS ACCOUNT ID>:cluster/<MSK CLUSTER NAME>/<MSK CLUSTER ID>"
      ]
    }
  ]
}

kcp scan clusters