【python kafka】python 连接kafka速成

Python Kafka Client有三种方式：pykafka、kafka-python 和 confluent-kafka，具体介绍见官网：
pykafka： https://pypi.org/project/pykafka/
kafka-python： https://pypi.org/project/kafka-python/
confluent_kafka： https://pypi.org/project/confluent-kafka/

安装包：

pip install pykafka
pip install kafka-python
pip install confluent_kafka

下载kafka：
https://www.cnblogs.com/darendu/p/11395546.html

启动kafka

# Start ZooKeeper Server
bin/zookeeper-server-start.sh config/zookeeper.properties

# Start Kafka Server
bin/kafka-server-start.sh config/server.properties

方法一：kafka-python

创建消费者(consumer.py)

from kafka import KafkaConsumer
consumer = KafkaConsumer('sample')
for message in consumer:
    print (message)

创建生产者(producer.py)

有一个消费者正在订阅我们的消息流，因此我们要创建一个生产者，发布消息到Kafka：

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('sample', b'Hello, World!')
producer.send('sample', key=b'message-two', value=b'This is Kafka-Python')

方法2：pykafka

https://pykafka.readthedocs.io/en/latest/index.ht

# -* coding:utf8 *-  
  from pykafka import KafkaClient  
  host = 'IP:9092, IP:9092, IP:9092'
  client = KafkaClient(hosts = host)  
  # 生产者  
  topicdocu = client.topics['my-topic']  
  producer = topicdocu.get_producer()  
  for i in range(100):  
      print i  
      producer.produce('test message ' + str(i ** 2))  
  producer.stop()



封装：
class KafkaProduct():

      def __init__(self,hosts,topic):
          """
          初始化实例
          :param hosts: 连接地址
          :param topic:
          """
          self.__client = KafkaClient(hosts=hosts)
          self.__topic = self.__client.topics[topic.encode()]
  
      def __set_topic(self, topic):
          self.__topic = self.__client.topics[topic.encode()]
  
      def set_topic(self, topic):
          """
          设置topic
          :param topic:
          :return:
          """
          self.__set_topic(topic)
  
      def get_topics(self):
          """
          获取当前所有topic
          :return:
          """
          return self.__client.topics
  
      def get_topic(self):
          """
          获取当前topic
          :return:
          """
          return self.__topic
  
      def Producer(self):
          """
          生产者对象
          :return:
          """
          with self.__topic.get_producer(delivery_reports=True) as producer:
              next_data = ''
              while True:
                  if next_data:
                      producer.produce(str(next_data).encode())
                  next_data = yield True
  
      def send_data(self,datas):
          """
          发送数据
          :param datas:需要传入的可迭代对象
          :return:
          """
          c = self.Producer()
          next(c)
          for i in datas:
              c.send(i)

if __name__ == '__main__':

    hosts = "1.2.3.4:9999,2.3.4.5:9090" #连接hosts
    topic = "test_523"
    K = KafkaProduct(hosts=hosts, topic=topic)  #
    #K.set_topic("test")  #切换设置新的topic
    K.get_topic()  #获取当前设置的topic
    #K.get_topics() #获取所有topic
    data = range(10000)  #要发送的可迭代对象
    K.send_data(data)

方法3：confluent_kafka

https://docs.confluent.io/platform/current/clients/confluent-kafka-python/index.html