kafka 2.8.x学习笔记-多租户、安全和其它_weixin_40455124_kafka多租户

网络投稿 02-07 4282

多租户实现

kafka没有体系化的租户管理，但可以通过一些手段进行用户和topic管理实现部分租户能力

Creating user spaces for tenants (sometimes called namespaces)

通过ACL控制用户只能访问指定前缀的topic通过自定义CreateTopicPolicy/AlterConfigPolicy (cf. KIP-108 and the setting create.topic.policy.class.name)强制用户只能创建符合前缀要求的topic关闭topic自动创建

Configuring topics with data retention policies and more：设置topic级别的 retention.bytes (size) and retention.ms，实现数据生命周期管理

Securing topics and clusters with encryption, authentication, and authorization

Encryption :data transferred between Kafka brokers and Kafka clients, between brokers, between brokers and ZooKeeper nodes, and between brokers and other, optional tools.AuthenticationAuthorization 限制用户对topic的访问,使用以下类似脚本 bin/kafka-acls.sh –bootstrap-server broker1:9092 –add --allow-principal User:Alice –producer –resource-pattern-type prefixed --topic acme.infosec.

Isolating tenants with quotas and rate limits

Client quotas：Kafka supports different types of (per-user principal) client quotas. 2020引入controller_mutations_rate ，使用token机制控制topic变化频率，参考urlhttps://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+OperationsNetwork Bandwidth Quotas：By default, each unique client group receives a fixed quota in bytes/sec as configured by the cluster.在用户或者client 级别进行限流，参考：https://kafka.apache.org/28/documentation.html#quotas bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type clients --entity-name clientA

Server quotas

限制连接数量

限制每个IP的连接数量 max.connections.per.ip.overrides comma-separated list of per-ip or hostname overrides to the default maximum number of connections. An example value is “hostName:100,127.0.0.1:200”max.connections.per.ip 默认每个ip最大连接数限制每个broker总的连接数量：max.connections新连接创建rate max.connection.creation.rate

listener（端口）级别连接限制

listener.name.internal.max.connectionslistener.name.internal.max.connection.creation.rate listener 参考urlhttps://blog.csdn.net/lidelin10/article/details/105316252

Monitoring and metering：按租户要求自定义metric收集、发送和告警

Inter-cluster data sharing (cf. geo-replication)

Security Overview Authentication of connections to brokers from clients (producers and consumers), other brokers and tools, using either SSL or SASL. Kafka supports the following SASL mechanisms: SASL/OAUTHBEARER - starting at version 2.0SASL/SCRAM-SHA-256 and SASL/SCRAM-SHA-512 - starting at version 0.10.2.0SASL/PLAIN - starting at version 0.10.0.0SASL/GSSAPI (Kerberos) - starting at version 0.9.0.0 Authentication of connections from brokers to ZooKeeper Encryption of data transferred between brokers and clients, between brokers, or between brokers and tools using SSL (Note that there is a performance degradation when SSL is enabled, the magnitude of which depends on the CPU type and the JVM implementation.)Authorization of read / write operations by clients 参考url [https://·blogs.com/rexcheny/articles/12884990.html)

Authorization is pluggable and integration with external authorization services is supported 自定义认证 authorizer.class.name=kafka.security.authorizer.AclAuthorizer 其它

kafka connect

用途，与其它系统集成，进行数据导入导出，参考https://blog.csdn.net/wjandy0211/article/details/93642257

features

A common framework for Kafka connectorsDistributed and standalone modesREST interfaceAutomatic offset managementDistributed and scalable by default :builds on the existing group management protocolStreaming/batch integration

支持的sink、resource-connectors

https://www.confluent.io/hub/?_ga=2.208833618.1729877910.1646979664-160366992.1641452267

https://docs.confluent.io/home/connect/self-managed/supported.html kafka STREAMS 测试框架-Trogdor

Workloads

ConsumeBenchRoundTripWorkloadProduceBench

Faults

ProcessStopFaultNetworkPartitionFault:implemented using iptables.

External Processes

ExternalCommandWorker