{"id":2734117,"date":"2023-06-19T19:19:43","date_gmt":"2023-06-19T23:19:43","guid":{"rendered":"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/plato-data\/multi-tenancy-apache-kafka-clusters-in-amazon-msk-with-iam-access-control-and-kafka-quotas-part-1-amazon-web-services\/"},"modified":"2023-06-19T19:19:43","modified_gmt":"2023-06-19T23:19:43","slug":"multi-tenancy-apache-kafka-clusters-in-amazon-msk-with-iam-access-control-and-kafka-quotas-part-1-amazon-web-services","status":"publish","type":"station","link":"https:\/\/platodata.io\/plato-data\/multi-tenancy-apache-kafka-clusters-in-amazon-msk-with-iam-access-control-and-kafka-quotas-part-1-amazon-web-services\/","title":{"rendered":"Multi-tenancy Apache Kafka clusters in Amazon MSK with IAM access control and Kafka Quotas \u2013 Part 1 | Amazon Web Services"},"content":{"rendered":"
With Amazon Managed Streaming for Apache Kafka<\/a> (Amazon MSK), you can build and run applications that use Apache Kafka to process streaming data. To process streaming data, organizations either use multiple Kafka clusters based on their application groupings, usage scenarios, compliance requirements, and other factors, or a dedicated Kafka cluster for the entire organization. It doesn\u2019t matter what pattern is used, Kafka clusters are typically multi-tenant, allowing multiple producer and consumer applications to consume and produce streaming data simultaneously.<\/p>\n With multi-tenant Kafka clusters, however, one of the challenges is to make sure that data consumer and producer applications don\u2019t overuse cluster resources. There is a possibility that a few poorly behaved applications may overuse cluster resources, affecting the well-behaved applications as a result. Therefore, teams who manage multi-tenant Kafka clusters need a mechanism to prevent applications from overconsuming cluster resources in order to avoid issues. This is where Kafka quotas come into play. Kafka quotas control the amount of resources client applications can use within a Kafka cluster.<\/p>\n In Part 1 of this two-part series, we explain the concepts of how to enforce Kafka quotas in MSK multi-tenant Kafka clusters while using AWS Identity and Access Management<\/a> (IAM) access control for authentication and authorization. In Part 2<\/a>, we cover detailed implementation steps along with sample Kafka client applications.<\/p>\n Kafka quotas control the amount of resources client applications can use within a Kafka cluster. It\u2019s possible for the multi-tenant Kafka cluster to experience performance degradation or a complete outage due to resource constraints if one or more client applications produce or consume large volumes of data or generate requests at a very high rate for a continuous period of time, monopolizing Kafka cluster\u2019s resources.<\/p>\n To prevent applications from overwhelming the cluster, Apache Kafka allows configuring quotas that determine how much traffic each client application produces and consumes per Kafka broker in a cluster. Kafka brokers throttle the client applications\u2019 requests in accordance with their allocated quotas. Kafka quotas can be configured for specific users<\/strong>, or specific client IDs<\/strong>, or both<\/strong>. The client ID<\/strong> is a logical name defined in the application code that Kafka brokers use to identify which application sent messages. The user<\/strong> represents the authenticated user principal of a client application in a secure Kafka cluster with authentication enabled.<\/p>\n There are two types of quotas supported in Kafka:<\/p>\n Depending on the business requirements, you can use either of these quota configurations. However, the use of network bandwidth quotas is common because it allows organizations to cap platform resources consumption according to the amount of data produced and consumed by applications per second.<\/p>\n Because this post uses an MSK cluster with IAM access control, we specifically discuss configuring network bandwidth quotas<\/strong> based on the applications\u2019 client IDs<\/strong> and authenticated user principals<\/strong>.<\/p>\n Keep the following in mind when working with Kafka quotas:<\/p>\n Note that client metrics are metrics exposed by clients connecting to Kafka clusters.<\/p>\n For more information about Kafka quotas, refer to Kafka documentation<\/a>.<\/p>\n Following our understanding of Kafka quotas, let\u2019s look at how to enforce them in an MSK cluster while using IAM access control for authentication and authorization. IAM access control in Amazon MSK eliminates the need for two separate mechanisms for authentication and authorization.<\/p>\n The following figure shows an MSK cluster that is configured to use IAM access control in the demo account. Each producer and consumer application has a quota that determines how much data they can produce or consume in bytes per second. For example, <\/p>\n The preceding figure illustrates how Kafka client applications ( As explained earlier, Kafka quotas can be configured for specific users, specific client IDs, or both. Let\u2019s explore client ID<\/strong> and user<\/strong> concepts and configurations required for Kafka quota allocation.<\/p>\n Client ID<\/strong><\/p>\n A client ID representing an application\u2019s logical name can be configured within an application\u2019s code. In Java applications, for example, you can set the producer\u2019s and consumer\u2019s client IDs using User<\/strong><\/p>\n The user refers to an authenticated user principal of the client application in the Kafka cluster with authentication enabled. As shown in the solution architecture, producer and consumer applications assume the For more information, refer to IAM identifiers<\/a>.<\/p>\n The role session name<\/strong> is a string identifier that uniquely identifies a session when IAM principals, federated identities, or applications assume an IAM role. In our case, The following is an example code snippet from the The following commands configure the produce<\/strong> and consume<\/strong> quotas dynamically for client applications based on their client ID<\/strong> and authenticated user<\/strong> principal in the MSK cluster configured with IAM access control.<\/p>\n The following code configures the produce quota:<\/p>\n The The following code configures the consume quota:<\/p>\n The Let\u2019s look at some example quota configuration commands for As a result of the preceding commands, the Part 2<\/a> of this series showcases the step-by-step detailed implementation of Kafka quotas configuration with IAM access control along with the sample producer and consumer client applications.<\/p>\n Kafka quotas offer teams the ability to set limits for producer and consumer applications. With Amazon MSK, Kafka quotas serve two important purposes: eliminating guesswork and preventing issues caused by poorly designed producer or consumer applications by limiting their quota, and allocating operational costs of a central streaming data platform across different cost centers and tenants (application and product teams).<\/p>\n In this post, we learned how to configure network bandwidth quotas within Amazon MSK while using IAM access control. We also covered some sample commands and code snippets to clarify how the client ID and authenticated principal are used when configuring quotas. Although we only demonstrated Kafka quotas using IAM access control, you can also configure them using other Amazon MSK-supported authentication mechanisms.<\/p>\n In Part 2<\/a> of this series, we demonstrate how to configure network bandwidth quotas with IAM access control in Amazon MSK and provide you with example producer and consumer applications so that you can see them in action.<\/p>\n Check out the following resources to learn more:<\/p>\n Vikas Bajaj<\/strong> is a Senior Manager, Solutions Architects, Financial Services at Amazon Web Services. Having worked with financial services organizations and digital native customers, he advises financial services customers in Australia on technology decisions, architectures, and product roadmaps.<\/p>\n <\/p>\nBrief introduction to Kafka quotas<\/h2>\n
\n
Considerations for Kafka quotas<\/h2>\n
\n
Producer.send()<\/code> call to be blocked.
Producer.send()<\/code> will eventually throw a
TimeoutException<\/code> if the timeout delay isn\u2019t sufficient to allow the broker to catch up to the producer application.<\/li>\n
client-id=\"marketing-producer-client\"<\/code> and
user=\"marketing-app-user\"<\/code>. In this case, all producer applications that have
marketing-producer-client<\/code> as a client ID<\/strong> and
marketing-app-user<\/code> as an authenticated user<\/strong> principal will share the 5 MB\/sec produce quota, impacting each other\u2019s throughput.<\/li>\n
produce-throttle-time-avg<\/code> and
produce-throttle-time-max<\/code>. If these are non-zero, it indicates that the destination brokers are slowing the producer down and the quotas configuration should be reviewed.<\/li>\n
fetch-throttle-time-avg<\/code> and
fetch-throttle-time-max<\/code>. If these are non-zero, it indicates that the origin brokers are slowing the consumer down and the quotas configuration should be reviewed.<\/li>\n<\/ul>\n
\n
kafka-config.sh<\/code> or the Kafka Admin API. The dynamic configuration mechanism is much more convenient and manageable because it allows quotas for the new producer and consumer applications to be configured at any time without having to restart brokers. Even while application clients are producing or consuming data, dynamic configuration changes take effect in real time.<\/li>\n
kafka-config.sh<\/code> command-line tool, you can set dynamic consume, produce, and request quotas using the following three configuration keys, respectively:
consumer_byte_rate<\/code>,
producer_byte_rate<\/code>, and
request_percentage<\/code>.<\/li>\n<\/ul>\n
Enforce network bandwidth quotas with IAM access control<\/h2>\n
ProducerApp-1<\/code> has a produce quota of 1024 bytes\/sec<\/strong>, and
ConsumerApp-1<\/code> and
ConsumerApp-2<\/code> each have a consume quota of 5120<\/strong> and 1024 bytes\/sec<\/strong>, respectively. It\u2019s important to note that Kafka quotas are set on the Kafka cluster rather than in the client applications.<\/p>\n
ProducerApp-1<\/code>,
ConsumerApp-1<\/code>, and
ConsumerApp-2<\/code>) access
Topic-B<\/code> in the MSK cluster by assuming write and read IAM roles. The workflow is as follows:<\/p>\n
\n
ProducerApp-1<\/code> (via its
ProducerApp-1-Role<\/code> IAM role) assumes the
Topic-B-Write-Role<\/code> IAM role to send messages to
Topic-B<\/code> in the MSK cluster.<\/li>\n
Topic-B-Write-Role<\/code> IAM role assumed,
ProducerApp-1<\/code> begins sending messages to
Topic-B<\/code>.<\/li>\n
ConsumerApp-1<\/code> (via its
ConsumerApp-1-Role<\/code> IAM role) and
ConsumerApp-2<\/code> (via its
ConsumerApp-2-Role<\/code> IAM role) assume the
Topic-B-Read-Role<\/code> IAM role to read messages from
Topic-B<\/code> in the MSK cluster.<\/li>\n
Topic-B-Read-Role<\/code> IAM role assumed,
ConsumerApp-1<\/code> and
ConsumerApp-2<\/code> start consuming messages from
Topic-B<\/code>.<\/li>\n<\/ul>\n
ConsumerApp-1<\/code> and
ConsumerApp-2<\/code> are two separate consumer applications. They do not belong to the same consumer group.<\/p>\n
Configuring client IDs and understanding authenticated user principal<\/h3>\n
ProducerConfig.CLIENT_ID_CONFIG<\/code> and
ConsumerConfig.CLIENT_ID_CONFIG<\/code> configurations, respectively. The following code snippet illustrates how
ProducerApp-1<\/code> sets the client ID to
this-is-me-producerapp-1<\/code> using
ProducerConfig.CLIENT_ID_CONFIG:<\/code><\/p>\n
Properties props = new Properties();\nprops.put(ProducerConfig.CLIENT_ID_CONFIG,\"this-is-me-producerapp-1\");<\/code><\/pre>\n<\/p><\/div>\n
Topic-B-Write-Role<\/code> and
Topic-B-Read-Role<\/code> IAM roles, respectively, to perform write and read operations on
Topic-B<\/code>. Therefore, their authenticated user principal will look like the following IAM identifier:<\/p>\n
arn:aws:sts::<AWS Account Id>:assumed-role\/<assumed Role Name>\/<role session name><\/code><\/pre>\n<\/p><\/div>\n
ProducerApp-1<\/code>,
ConsumerApp-1<\/code>, and
ConsumerApp-2<\/code> applications assume an IAM role using the AWS Security Token Service<\/a> (AWS STS) SDK, and provide a role session name in the AWS STS SDK call. For example, if
ProducerApp-1<\/code> assumes the
Topic-B-Write-Role<\/code> IAM role and uses
this-is-producerapp-1-role-session<\/code> as its role session name, its authenticated user principal will be as follows:<\/p>\n
arn:aws:sts::<AWS Account Id><\/span>:assumed-role\/Topic-B-Write-Role\/this-is-producerapp-1-role-session<\/code><\/pre>\n<\/p><\/div>\n
ProducerApp-1<\/code> application using
this-is-producerapp-1-role-session<\/code> as the role session name<\/strong> while assuming the
Topic-B-Write-Role<\/code> IAM role using the AWS STS SDK:<\/p>\n
StsClient stsClient = StsClient.builder().region(region).build();\nAssumeRoleRequest roleRequest = AssumeRoleRequest.builder() .roleArn(\"<Topic-B-Write-Role ARN>\") .roleSessionName(\"this-is-producerapp-1-role-session\") \/\/role-session-name string literal .build();<\/code><\/pre>\n<\/p><\/div>\n
Configure network bandwidth (produce and consume) quotas<\/h2>\n
kafka-configs.sh --bootstrap-server <MSK cluster bootstrap servers IAM endpoint><\/span> --command-config config_iam.properties --alter --add-config \"producer_byte_rate=<number of bytes per second><\/span>\" --entity-type clients --entity-name <ProducerApp client Id><\/span> --entity-type users --entity-name <ProducerApp user principal><\/span><\/code><\/pre>\n<\/p><\/div>\n
producer_byes_rate<\/code> refers to the number of messages, in bytes, that a producer client identified by client ID<\/strong> and user<\/strong> is allowed to produce to a single broker per second. The option
--command-config<\/code> points to
config_iam.properties<\/code>, which contains the properties required for IAM access control.<\/p>\n
kafka-configs.sh --bootstrap-server <MSK cluster bootstrap servers IAM endpoint><\/span> --command-config config_iam.properties --alter --add-config \"consumer_byte_rate=<number of bytes per second><\/span>\" --entity-type clients --entity-name <ConsumerApp client Id><\/span> --entity-type users --entity-name <ConsumerApp user principal><\/span><\/code><\/pre>\n<\/p><\/div>\n
consumer_bytes_rate<\/code> refers to the number of messages, in bytes, that a consumer client identified by client ID<\/strong> and user<\/strong> allowed to consume from a single broker per second.<\/p>\n
ProducerApp-1<\/code>,
ConsumerApp-1<\/code>, and
ConsumerApp-2<\/code> client applications:<\/p>\n
\n
ProducerApp-1<\/code> has
this-is-me-producerapp-1<\/code> configured as the client ID in the application code and uses
this-is-producerapp-1-role-session<\/code> as the role session name when assuming the
Topic-B-Write-Role<\/code> IAM role. The following command sets the produce quota for
ProducerApp-1<\/code> to 1024 bytes per second:<\/li>\n<\/ul>\n
kafka-configs.sh --bootstrap-server <MSK Cluster Bootstrap servers IAM endpoint><\/span> --command-config config_iam.properties --alter --add-config \"producer_byte_rate=1024\" --entity-type clients --entity-name this-is-me-producerapp-1 --entity-type users --entity-name arn:aws:sts::<AWS Account Id><\/span>:assumed-role\/Topic-B-Write-Role\/this-is-producerapp-1-role-session<\/code><\/pre>\n<\/p><\/div>\n
\n
ConsumerApp-1<\/code> has
this-is-me-consumerapp-1<\/code> configured as the client ID in the application code and uses
this-is-consumerapp-1-role-session<\/code> as the role session name when assuming the
Topic-B-Read-Role<\/code> IAM role. The following command sets the consume quota for
ConsumerApp-1<\/code> to 5120 bytes per second:<\/li>\n<\/ul>\n
kafka-configs.sh --bootstrap-server <MSK Cluster Bootstrap servers IAM endpoint><\/span> --command-config config_iam.properties --alter --add-config \"consumer_byte_rate=5120\" --entity-type clients --entity-name this-is-me-consumerapp-1 --entity-type users --entity-name arn:aws:sts::<AWS Account Id><\/span>:assumed-role\/Topic-B-Read-Role\/this-is-consumerapp-1-role-session<\/code><\/pre>\n<\/p><\/div>\n
ConsumerApp-2 consume quota configuration<\/strong> \u2013 Let\u2019s assume ConsumerApp-2<\/code> has
this-is-me-consumerapp-2<\/code> configured as the client ID in the application code and uses
this-is-consumerapp-2-role-session<\/code> as the role session name when assuming the
Topic-B-Read-Rol<\/code>e IAM role. The following command sets the consume quota for
ConsumerApp-2<\/code> to 1024 bytes per second per broker:<\/p>\n
kafka-configs.sh --bootstrap-server <MSK Cluster Bootstrap servers IAM endpoint><\/span> --command-config config_iam.properties --alter --add-config \"consumer_byte_rate=1024\" --entity-type clients --entity-name this-is-me-consumerapp-2 --entity-type users --entity-name arn:aws:sts::<AWS Account Id><\/span>:assumed-role\/Topic-B-Read-Role\/this-is-consumerapp-2-role-session<\/code><\/pre>\n<\/p><\/div>\n
ProducerApp-1<\/code>,
ConsumerApp-1<\/code>, and
ConsumerApp-2<\/code> client applications will be throttled by the MSK cluster using IAM access control if they exceed their assigned produce and consume quotas, respectively.<\/p>\n
Implement the solution<\/h2>\n
Conclusion<\/h2>\n
\nAbout the Author<\/h3>\n
\n