kcp discover

kcp discover

Multi-region, multi cluster discovery scan of AWS MSK

Synopsis

Performs a full Discovery of all MSK clusters across multiple regions, and their associated resources, costs and metrics

kcp discover [flags]

Examples

  # Scan a single region
  kcp discover --region us-east-1

  # Scan multiple regions (repeated flag or comma-separated)
  kcp discover --region us-east-1 --region eu-west-3
  kcp discover --region us-east-1,eu-west-3

  # Skip topic/cost/metric discovery for faster runs or reduced IAM scope
  kcp discover --region us-east-1 --skip-topics --skip-costs --skip-metrics


  # Specify metrics granularity (mutually exclusive with --skip-metrics)
  kcp discover --region us-east-1 --metrics-granularity 60s
  kcp discover --region us-east-1 --metrics-granularity 5m
  kcp discover --region us-east-1 --metrics-granularity 1h
  kcp discover --region us-east-1 --metrics-granularity 1d

  The maximum time range for each granularity is:
  - 60s = 15 days
  - 5m = 63 days
  - 1h = 365 days
  - 1d = 365 days

  The finer the granularity, the more detailed the metrics data, but also more data is stored in the state-file, resulting in state-file growth. Coarser granularity is recommended for averaging workloads over longer time periods, but will smooth out spikes, while finer granularity is recommended for analyzing more bursty workloads and uncovering spikes over short time periods.

  # Discover a single cluster (region inferred from the ARN); create or replace it in state
  kcp discover --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/uuid

  # Re-discover one cluster at a finer metrics granularity without touching other clusters
  kcp discover --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/uuid --metrics-granularity 60s

Options

      --cluster-arn strings          Discover only the specified MSK cluster ARN(s) (comma separated or repeated flag). Region is inferred from each ARN. Mutually exclusive with --region.
  -h, --help                         help for discover
      --metrics-granularity string   The granularity for which to query for CloudWatch metrics. Valid values: 60s, 5m, 1h, 1d. The maximum time range for each granularity is: 60s = 15 days, 5m = 63 days, 1h = 365 days, 1d = 365 days. (default "1d")
      --region strings               The AWS region(s) to scan (comma separated list or repeated flag). Mutually exclusive with --cluster-arn.
      --skip-costs                   Skips the cost discovery through the AWS Cost Explorer API
      --skip-metrics                 Skips the metrics discovery through the AWS CloudWatch API
      --skip-topics                  Skips the topic discovery through the AWS MSK API

Options inherited from parent commands

      --verbose   Enable verbose logging to console

AWS IAM Permissions

The following policy covers a full run. If you pass --skip-topics, --skip-costs, or --skip-metrics, the corresponding statements can be omitted.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "MSKScanPermissions",
      "Effect": "Allow",
      "Action": [
        "kafka:DescribeClusterV2",
        "kafka:DescribeConfigurationRevision",
        "kafka:DescribeReplicator",
        "kafka:GetBootstrapBrokers",
        "kafka:GetClusterPolicy",
        "kafka:GetCompatibleKafkaVersions",
        "kafka:ListClientVpcConnections",
        "kafka:ListClusterOperationsV2",
        "kafka:ListClustersV2",
        "kafka:ListConfigurations",
        "kafka:ListKafkaVersions",
        "kafka:ListNodes",
        "kafka:ListReplicators",
        "kafka:ListScramSecrets",
        "kafka:ListVpcConnections"
      ],
      "Resource": "*"
    },
    {
      "Sid": "MSKClusterConnect",
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeCluster"
      ],
      "Resource": "*"
    },
    {
      "Sid": "MSKTopicActions",
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:DescribeTopicDynamicConfiguration",
        "kafka:DescribeTopic",
        "kafka:ListTopics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "CostMetricsScanPermissions",
      "Effect": "Allow",
      "Action": [
        "ce:GetCostAndUsage",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:ListMetrics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "MSKNetworkingScanPermission",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeSubnets"
      ],
      "Resource": "*"
    },
    {
      "Sid": "MSKConnectScanPermissions",
      "Effect": "Allow",
      "Action": [
        "kafkaconnect:DescribeConnector",
        "kafkaconnect:ListConnectors"
      ],
      "Resource": "*"
    }
  ]
}

SEE ALSO

• kcp - A CLI tool for kafka cluster planning and migration