{"id":2734117,"date":"2023-06-19T19:19:43","date_gmt":"2023-06-19T23:19:43","guid":{"rendered":"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/plato-data\/multi-tenancy-apache-kafka-clusters-in-amazon-msk-with-iam-access-control-and-kafka-quotas-part-1-amazon-web-services\/"},"modified":"2023-06-19T19:19:43","modified_gmt":"2023-06-19T23:19:43","slug":"multi-tenancy-apache-kafka-clusters-in-amazon-msk-with-iam-access-control-and-kafka-quotas-part-1-amazon-web-services","status":"publish","type":"station","link":"https:\/\/platodata.io\/plato-data\/multi-tenancy-apache-kafka-clusters-in-amazon-msk-with-iam-access-control-and-kafka-quotas-part-1-amazon-web-services\/","title":{"rendered":"Multi-tenancy Apache Kafka clusters in Amazon MSK with IAM access control and Kafka Quotas \u2013 Part 1 | Amazon Web Services"},"content":{"rendered":"

With Amazon Managed Streaming for Apache Kafka<\/a> (Amazon MSK), you can build and run applications that use Apache Kafka to process streaming data. To process streaming data, organizations either use multiple Kafka clusters based on their application groupings, usage scenarios, compliance requirements, and other factors, or a dedicated Kafka cluster for the entire organization. It doesn\u2019t matter what pattern is used, Kafka clusters are typically multi-tenant, allowing multiple producer and consumer applications to consume and produce streaming data simultaneously.<\/p>\n

With multi-tenant Kafka clusters, however, one of the challenges is to make sure that data consumer and producer applications don\u2019t overuse cluster resources. There is a possibility that a few poorly behaved applications may overuse cluster resources, affecting the well-behaved applications as a result. Therefore, teams who manage multi-tenant Kafka clusters need a mechanism to prevent applications from overconsuming cluster resources in order to avoid issues. This is where Kafka quotas come into play. Kafka quotas control the amount of resources client applications can use within a Kafka cluster.<\/p>\n

In Part 1 of this two-part series, we explain the concepts of how to enforce Kafka quotas in MSK multi-tenant Kafka clusters while using AWS Identity and Access Management<\/a> (IAM) access control for authentication and authorization. In Part 2<\/a>, we cover detailed implementation steps along with sample Kafka client applications.<\/p>\n

Brief introduction to Kafka quotas<\/h2>\n

Kafka quotas control the amount of resources client applications can use within a Kafka cluster. It\u2019s possible for the multi-tenant Kafka cluster to experience performance degradation or a complete outage due to resource constraints if one or more client applications produce or consume large volumes of data or generate requests at a very high rate for a continuous period of time, monopolizing Kafka cluster\u2019s resources.<\/p>\n

To prevent applications from overwhelming the cluster, Apache Kafka allows configuring quotas that determine how much traffic each client application produces and consumes per Kafka broker in a cluster. Kafka brokers throttle the client applications\u2019 requests in accordance with their allocated quotas. Kafka quotas can be configured for specific users<\/strong>, or specific client IDs<\/strong>, or both<\/strong>. The client ID<\/strong> is a logical name defined in the application code that Kafka brokers use to identify which application sent messages. The user<\/strong> represents the authenticated user principal of a client application in a secure Kafka cluster with authentication enabled.<\/p>\n

There are two types of quotas supported in Kafka:<\/p>\n