Hi,
I have some problems during hermes installation on my ubuntu 14.04.02 which works natively. After following Your tutorial I have created group and topic, then subscription for this topic. But unfortunately I cannot publish any message and still receives {"message":"Broker seems to be down, cause: Failed to update metadata after 500 ms.","code":"INTERNAL_ERROR"}
. Inspecting logs with docker-compose logs
shows that one exception is still being thrown
consumers_1 | 2015-07-28 14:38:25.878 WARN kafka.client.ClientUtils$ - Fetching topic metadata with correlation id 54 for topics [Set(G1.T1)] from broker [id:9092,host:192.168.24.169,port:9092] failed
consumers_1 | java.nio.channels.ClosedChannelException: null
consumers_1 | at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) ~[kafka_2.10-0.8.2.0.jar:na]
consumers_1 | at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) ~[kafka_2.10-0.8.2.0.jar:na]
consumers_1 | at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) ~[kafka_2.10-0.8.2.0.jar:na]
consumers_1 | at kafka.producer.SyncProducer.send(SyncProducer.scala:113) ~[kafka_2.10-0.8.2.0.jar:na]
consumers_1 | at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) [kafka_2.10-0.8.2.0.jar:na]
consumers_1 | at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) [kafka_2.10-0.8.2.0.jar:na]
consumers_1 | at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) [kafka_2.10-0.8.2.0.jar:na]
consumers_1 | at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) [kafka_2.10-0.8.2.0.jar:na]
Few days ago I've found here similar issue https://github.com/allegro/hermes/issues/33 but this solution didn't worke for me. I've checked many possible values for KAFKA_ADVERTISED_HOST_NAME
: eth0 from ifconfig
, docker0 form ifconfig
, localhost from ifconfig
but issue remains. Before each modification of hermes settings I cleared whole docker environment to check if applied changes works.
To provide You more informations about my case I'm attaching here logs
docker_kafka_1
[2015-07-28 14:10:48,428] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,452] INFO Property advertised.host.name is overridden to 192.168.24.169 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,452] INFO Property advertised.port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property broker.id is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.dirs is overridden to /kafka/kafka-logs-9092 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property zookeeper.connect is overridden to 172.17.0.6:2181 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,482] INFO [Kafka Server 9092], starting (kafka.server.KafkaServer)
[2015-07-28 14:10:48,483] INFO [Kafka Server 9092], Connecting to zookeeper on 172.17.0.6:2181 (kafka.server.KafkaServer)
[2015-07-28 14:10:48,490] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-07-28 14:10:48,495] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:host.name=c2dc4a71f97b (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.version=1.6.0_34 (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.vendor=Sun Microsystems Inc. (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.class.path=:/opt/kafka_2.10-0.8.2.0/bin/../core/build/dependant-libs-2.10.4*/*.jar:/opt/kafka_2.10-0.8.2.0/bin/../examples/build/libs//kafka-examples*.jar:/opt/kafka_2.10-0.8.2.0/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/opt/kafka_2.10-0.8.2.0/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/opt/kafka_2.10-0.8.2.0/bin/../clients/build/libs/kafka-clients*.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/jopt-simple-3.2.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka-clients-0.8.2.0.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0-javadoc.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0-scaladoc.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0-sources.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0-test.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/log4j-1.2.16.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/lz4-1.2.0.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/scala-library-2.10.4.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/slf4j-api-1.7.6.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/slf4j-log4j12-1.6.1.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/snappy-java-1.1.1.6.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/zkclient-0.3.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/zookeeper-3.4.6.jar:/opt/kafka_2.10-0.8.2.0/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk-amd64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:os.version=3.16.0-45-generic (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,496] INFO Initiating client connection, connectString=172.17.0.6:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@4b704006 (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,515] INFO Opening socket connection to server 172.17.0.6/172.17.0.6:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-07-28 14:10:48,519] INFO Socket connection established to 172.17.0.6/172.17.0.6:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-07-28 14:10:48,540] INFO Session establishment complete on server 172.17.0.6/172.17.0.6:2181, sessionid = 0x14ed5005e750000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2015-07-28 14:10:48,541] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2015-07-28 14:10:48,677] INFO Log directory '/kafka/kafka-logs-9092' not found, creating it. (kafka.log.LogManager)
[2015-07-28 14:10:48,685] INFO Loading logs. (kafka.log.LogManager)
[2015-07-28 14:10:48,690] INFO Logs loading complete. (kafka.log.LogManager)
[2015-07-28 14:10:48,691] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2015-07-28 14:10:48,694] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2015-07-28 14:10:48,732] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2015-07-28 14:10:48,733] INFO [Socket Server on Broker 9092], Started (kafka.network.SocketServer)
[2015-07-28 14:10:48,819] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2015-07-28 14:10:48,904] INFO conflict in /brokers/ids/9092 data: {"jmx_port":-1,"timestamp":"1438092648877","host":"192.168.24.169","version":1,"port":9092} stored data: {"jmx_port":-1,"timestamp":"1438082684810","host":"192.168.24.169","version":1,"port":9092} (kafka.utils.ZkUtils$)
[2015-07-28 14:10:48,928] INFO I wrote this conflicted ephemeral node [{"jmx_port":-1,"timestamp":"1438092648877","host":"192.168.24.169","version":1,"port":9092}] at /brokers/ids/9092 a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-07-28 14:10:55,087] INFO conflict in /brokers/ids/9092 data: {"jmx_port":-1,"timestamp":"1438092648877","host":"192.168.24.169","version":1,"port":9092} stored data: {"jmx_port":-1,"timestamp":"1438082684810","host":"192.168.24.169","version":1,"port":9092} (kafka.utils.ZkUtils$)
[2015-07-28 14:10:55,113] INFO I wrote this conflicted ephemeral node [{"jmx_port":-1,"timestamp":"1438092648877","host":"192.168.24.169","version":1,"port":9092}] at /brokers/ids/9092 a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-07-28 14:10:56,827] ERROR Error handling event ZkEvent[Data of /controller changed sent to kafka.server.ZookeeperLeaderElector$LeaderChangeListener@39fc58f3] (org.I0Itec.zkclient.ZkEventThread)
java.lang.IllegalStateException: Kafka scheduler has not been started
at kafka.utils.KafkaScheduler.ensureStarted(KafkaScheduler.scala:114)
at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:86)
at kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:350)
at kafka.controller.KafkaController$$anonfun$2.apply$mcV$sp(KafkaController.scala:162)
at kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply$mcZ$sp(ZookeeperLeaderElector.scala:138)
at kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:134)
at kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:134)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:134)
at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:549)
at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
[2015-07-28 14:11:01,487] INFO Registered broker 9092 at path /brokers/ids/9092 with address 192.168.24.169:9092. (kafka.utils.ZkUtils$)
[2015-07-28 14:11:01,751] INFO [Kafka Server 9092], started (kafka.server.KafkaServer)
docker_consumers_1
[2015-07-28 14:10:48,428] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,452] INFO Property advertised.host.name is overridden to 192.168.24.169 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,452] INFO Property advertised.port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property broker.id is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.dirs is overridden to /kafka/kafka-logs-9092 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,453] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property zookeeper.connect is overridden to 172.17.0.6:2181 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,454] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
[2015-07-28 14:10:48,482] INFO [Kafka Server 9092], starting (kafka.server.KafkaServer)
[2015-07-28 14:10:48,483] INFO [Kafka Server 9092], Connecting to zookeeper on 172.17.0.6:2181 (kafka.server.KafkaServer)
[2015-07-28 14:10:48,490] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-07-28 14:10:48,495] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:host.name=c2dc4a71f97b (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.version=1.6.0_34 (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.vendor=Sun Microsystems Inc. (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.class.path=:/opt/kafka_2.10-0.8.2.0/bin/../core/build/dependant-libs-2.10.4*/*.jar:/opt/kafka_2.10-0.8.2.0/bin/../examples/build/libs//kafka-examples*.jar:/opt/kafka_2.10-0.8.2.0/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/opt/kafka_2.10-0.8.2.0/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/opt/kafka_2.10-0.8.2.0/bin/../clients/build/libs/kafka-clients*.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/jopt-simple-3.2.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka-clients-0.8.2.0.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0-javadoc.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0-scaladoc.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0-sources.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0-test.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/kafka_2.10-0.8.2.0.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/log4j-1.2.16.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/lz4-1.2.0.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/scala-library-2.10.4.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/slf4j-api-1.7.6.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/slf4j-log4j12-1.6.1.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/snappy-java-1.1.1.6.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/zkclient-0.3.jar:/opt/kafka_2.10-0.8.2.0/bin/../libs/zookeeper-3.4.6.jar:/opt/kafka_2.10-0.8.2.0/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk-amd64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:os.version=3.16.0-45-generic (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,495] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,496] INFO Initiating client connection, connectString=172.17.0.6:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@4b704006 (org.apache.zookeeper.ZooKeeper)
[2015-07-28 14:10:48,515] INFO Opening socket connection to server 172.17.0.6/172.17.0.6:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-07-28 14:10:48,519] INFO Socket connection established to 172.17.0.6/172.17.0.6:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-07-28 14:10:48,540] INFO Session establishment complete on server 172.17.0.6/172.17.0.6:2181, sessionid = 0x14ed5005e750000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2015-07-28 14:10:48,541] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2015-07-28 14:10:48,677] INFO Log directory '/kafka/kafka-logs-9092' not found, creating it. (kafka.log.LogManager)
[2015-07-28 14:10:48,685] INFO Loading logs. (kafka.log.LogManager)
[2015-07-28 14:10:48,690] INFO Logs loading complete. (kafka.log.LogManager)
[2015-07-28 14:10:48,691] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2015-07-28 14:10:48,694] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2015-07-28 14:10:48,732] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2015-07-28 14:10:48,733] INFO [Socket Server on Broker 9092], Started (kafka.network.SocketServer)
[2015-07-28 14:10:48,819] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2015-07-28 14:10:48,904] INFO conflict in /brokers/ids/9092 data: {"jmx_port":-1,"timestamp":"1438092648877","host":"192.168.24.169","version":1,"port":9092} stored data: {"jmx_port":-1,"timestamp":"1438082684810","host":"192.168.24.169","version":1,"port":9092} (kafka.utils.ZkUtils$)
[2015-07-28 14:10:48,928] INFO I wrote this conflicted ephemeral node [{"jmx_port":-1,"timestamp":"1438092648877","host":"192.168.24.169","version":1,"port":9092}] at /brokers/ids/9092 a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-07-28 14:10:55,087] INFO conflict in /brokers/ids/9092 data: {"jmx_port":-1,"timestamp":"1438092648877","host":"192.168.24.169","version":1,"port":9092} stored data: {"jmx_port":-1,"timestamp":"1438082684810","host":"192.168.24.169","version":1,"port":9092} (kafka.utils.ZkUtils$)
[2015-07-28 14:10:55,113] INFO I wrote this conflicted ephemeral node [{"jmx_port":-1,"timestamp":"1438092648877","host":"192.168.24.169","version":1,"port":9092}] at /brokers/ids/9092 a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-07-28 14:10:56,827] ERROR Error handling event ZkEvent[Data of /controller changed sent to kafka.server.ZookeeperLeaderElector$LeaderChangeListener@39fc58f3] (org.I0Itec.zkclient.ZkEventThread)
java.lang.IllegalStateException: Kafka scheduler has not been started
at kafka.utils.KafkaScheduler.ensureStarted(KafkaScheduler.scala:114)
at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:86)
at kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:350)
at kafka.controller.KafkaController$$anonfun$2.apply$mcV$sp(KafkaController.scala:162)
at kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply$mcZ$sp(ZookeeperLeaderElector.scala:138)
at kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:134)
at kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:134)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:134)
at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:549)
at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
[2015-07-28 14:11:01,487] INFO Registered broker 9092 at path /brokers/ids/9092 with address 192.168.24.169:9092. (kafka.utils.ZkUtils$)
[2015-07-28 14:11:01,751] INFO [Kafka Server 9092], started (kafka.server.KafkaServer)
ppiorkowski@ppiorkowski:~/Development/brain/tools$ docker logs docker_consumers_1
2015-07-28 14:10:48.921 INFO c.n.c.sources.URLConfigurationSource - URLs to be used as dynamic configuration source: [file:/etc/hermes/consumers.properties]
2015-07-28 14:10:48.955 INFO c.n.config.DynamicPropertyFactory - DynamicPropertyFactory is initialized with configuration sources: com.netflix.config.ConcurrentCompositeConfiguration@402e37bc
2015-07-28 14:10:49.013 INFO o.a.c.f.imps.CuratorFrameworkImpl - Starting
2015-07-28 14:10:49.093 INFO o.a.c.f.state.ConnectionStateManager - State change: CONNECTED
2015-07-28 14:10:49.384 INFO o.a.c.f.imps.CuratorFrameworkImpl - Starting
2015-07-28 14:10:49.390 INFO o.a.c.f.state.ConnectionStateManager - State change: CONNECTED
2015-07-28 14:10:49.451 INFO p.a.t.h.c.cache.zookeeper.NodeCache - Got entry change event for path /run/hermes/groups/G1
2015-07-28 14:10:49.462 INFO p.a.t.h.c.cache.zookeeper.NodeCache - Got entry change event for path /run/hermes/groups/G1/topics/T1
2015-07-28 14:10:49.579 INFO p.a.t.h.c.s.c.z.SubscriptionsNodeCache - Got subscription change event for path /run/hermes/groups/G1/topics/T1/subscriptions/S1 type CHILD_ADDED
2015-07-28 14:10:49.877 INFO org.I0Itec.zkclient.ZkEventThread - Starting ZkClient event thread.
2015-07-28 14:10:50.290 INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2015-07-28 14:11:13.587 WARN kafka.client.ClientUtils$ - Fetching topic metadata with correlation id 0 for topics [Set(G1.T1)] from broker [id:9092,host:192.168.24.169,port:9092] failed
java.nio.channels.ClosedChannelException: null
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) ~[kafka_2.10-0.8.2.0.jar:na]
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) ~[kafka_2.10-0.8.2.0.jar:na]
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) ~[kafka_2.10-0.8.2.0.jar:na]
at kafka.producer.SyncProducer.send(SyncProducer.scala:113) ~[kafka_2.10-0.8.2.0.jar:na]
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) [kafka_2.10-0.8.2.0.jar:na]
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) [kafka_2.10-0.8.2.0.jar:na]
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) [kafka_2.10-0.8.2.0.jar:na]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) [kafka_2.10-0.8.2.0.jar:na]
2015-07-28 14:11:43.701 WARN kafka.client.ClientUtils$ - Fetching topic metadata with correlation id 1 for topics [Set(G1.T1)] from broker [id:9092,host:192.168.24.169,port:9092] failed
java.nio.channels.ClosedChannelException: null
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) ~[kafka_2.10-0.8.2.0.jar:na]
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) ~[kafka_2.10-0.8.2.0.jar:na]
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) ~[kafka_2.10-0.8.2.0.jar:na]
at kafka.producer.SyncProducer.send(SyncProducer.scala:113) ~[kafka_2.10-0.8.2.0.jar:na]
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) [kafka_2.10-0.8.2.0.jar:na]
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) [kafka_2.10-0.8.2.0.jar:na]
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) [kafka_2.10-0.8.2.0.jar:na]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) [kafka_2.10-0.8.2.0.jar:na]
2015-07-28 14:11:43.704 WARN k.c.ConsumerFetcherManager$LeaderFinderThread - [G1_T1_S1_e0ed93125dda-1438092649872-d639c205-leader-finder-thread], Failed to find leader for Set([G1.T1,8], [G1.T1,2], [G1.T1,3], [G1.T1,0], [G1.T1,4], [G1.T1,6], [G1.T1,7], [G1.T1,1], [G1.T1,5], [G1.T1,9])
docker_zookeeper_1
JMX enabled by default
Using config: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
2015-07-28 14:10:47,960 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
2015-07-28 14:10:47,963 [myid:] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2015-07-28 14:10:47,963 [myid:] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 1
2015-07-28 14:10:47,964 [myid:] - WARN [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running in standalone mode
2015-07-28 14:10:47,964 [myid:] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2015-07-28 14:10:47,973 [myid:] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
2015-07-28 14:10:47,974 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
2015-07-28 14:10:47,974 [myid:] - INFO [main:ZooKeeperServerMain@95] - Starting server
2015-07-28 14:10:47,979 [myid:] - INFO [main:Environment@100] - Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-07-28 14:10:47,980 [myid:] - INFO [main:Environment@100] - Server environment:host.name=4675d9977d60
2015-07-28 14:10:47,980 [myid:] - INFO [main:Environment@100] - Server environment:java.version=1.7.0_65
2015-07-28 14:10:47,980 [myid:] - INFO [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
2015-07-28 14:10:47,980 [myid:] - INFO [main:Environment@100] - Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-07-28 14:10:47,980 [myid:] - INFO [main:Environment@100] - Server environment:java.class.path=/opt/zookeeper-3.4.6/bin/../build/classes:/opt/zookeeper-3.4.6/bin/../build/lib/*.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/opt/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/opt/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.6/bin/../conf:
2015-07-28 14:10:47,980 [myid:] - INFO [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2015-07-28 14:10:47,980 [myid:] - INFO [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
2015-07-28 14:10:47,982 [myid:] - INFO [main:Environment@100] - Server environment:java.compiler=<NA>
2015-07-28 14:10:47,982 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Linux
2015-07-28 14:10:47,982 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64
2015-07-28 14:10:47,982 [myid:] - INFO [main:Environment@100] - Server environment:os.version=3.16.0-45-generic
2015-07-28 14:10:47,982 [myid:] - INFO [main:Environment@100] - Server environment:user.name=root
2015-07-28 14:10:47,982 [myid:] - INFO [main:Environment@100] - Server environment:user.home=/root
2015-07-28 14:10:47,982 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/opt/zookeeper-3.4.6
2015-07-28 14:10:47,983 [myid:] - INFO [main:ZooKeeperServer@755] - tickTime set to 2000
2015-07-28 14:10:47,983 [myid:] - INFO [main:ZooKeeperServer@764] - minSessionTimeout set to -1
2015-07-28 14:10:47,983 [myid:] - INFO [main:ZooKeeperServer@773] - maxSessionTimeout set to -1
2015-07-28 14:10:47,991 [myid:] - INFO [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
2015-07-28 14:10:48,520 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.7:60189
2015-07-28 14:10:48,525 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.17.0.7:60189
2015-07-28 14:10:48,526 [myid:] - INFO [SyncThread:0:FileTxnLog@199] - Creating new log file: log.1b7
2015-07-28 14:10:48,538 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14ed5005e750000 with negotiated timeout 6000 for client /172.17.0.7:60189
2015-07-28 14:10:48,895 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14ed5005e750000 type:create cxid:0xf zxid:0x1b8 txntype:-1 reqpath:n/a Error Path:/brokers/ids/9092 Error:KeeperErrorCode = NodeExists for /brokers/ids/9092
2015-07-28 14:10:49,076 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.8:54176
2015-07-28 14:10:49,078 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.17.0.8:54176
2015-07-28 14:10:49,086 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14ed5005e750001 with negotiated timeout 7000 for client /172.17.0.8:54176
2015-07-28 14:10:49,385 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.8:54178
2015-07-28 14:10:49,386 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.17.0.8:54178
2015-07-28 14:10:49,389 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14ed5005e750002 with negotiated timeout 7000 for client /172.17.0.8:54178
2015-07-28 14:10:49,549 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.9:54530
2015-07-28 14:10:49,553 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.17.0.9:54530
2015-07-28 14:10:49,556 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14ed5005e750003 with negotiated timeout 10000 for client /172.17.0.9:54530
2015-07-28 14:10:49,878 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.8:54180
2015-07-28 14:10:49,878 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.17.0.8:54180
2015-07-28 14:10:50,290 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14ed5005e750004 with negotiated timeout 7000 for client /172.17.0.8:54180
2015-07-28 14:10:52,263 [myid:] - WARN [SyncThread:0:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:0 took 1907ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2015-07-28 14:10:52,805 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14ed5005e750004 type:create cxid:0x15 zxid:0x1be txntype:-1 reqpath:n/a Error Path:/consumers/G1_T1_S1/owners/G1.T1/6 Error:KeeperErrorCode = NodeExists for /consumers/G1_T1_S1/owners/G1.T1/6
2015-07-28 14:10:53,178 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14ed5005e750004 type:create cxid:0x17 zxid:0x1bf txntype:-1 reqpath:n/a Error Path:/consumers/G1_T1_S1/owners/G1.T1/9 Error:KeeperErrorCode = NodeExists for /consumers/G1_T1_S1/owners/G1.T1/9
2015-07-28 14:10:53,277 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.10:53206
2015-07-28 14:10:53,280 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.17.0.10:53206
2015-07-28 14:10:53,346 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14ed5005e750005 with negotiated timeout 10000 for client /172.17.0.10:53206
2015-07-28 14:10:53,347 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14ed5005e750004 type:create cxid:0x19 zxid:0x1c1 txntype:-1 reqpath:n/a Error Path:/consumers/G1_T1_S1/owners/G1.T1/8 Error:KeeperErrorCode = NodeExists for /consumers/G1_T1_S1/owners/G1.T1/8
2015-07-28 14:10:53,357 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14ed5005e750004 type:create cxid:0x1b zxid:0x1c2 txntype:-1 reqpath:n/a Error Path:/consumers/G1_T1_S1/owners/G1.T1/5 Error:KeeperErrorCode = NodeExists for /consumers/G1_T1_S1/owners/G1.T1/5
2015-07-28 14:10:53,388 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14ed5005e750004 type:create cxid:0x1d zxid:0x1c3 txntype:-1 reqpath:n/a Error Path:/consumers/G1_T1_S1/owners/G1.T1/7 Error:KeeperErrorCode = NodeExists for /consumers/G1_T1_S1/owners/G1.T1/7
2015-07-28 14:10:53,509 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.10:53209
2015-07-28 14:10:53,510 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.17.0.10:53209
2015-07-28 14:10:53,519 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.10:53210
2015-07-28 14:10:53,519 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.17.0.10:53210
2015-07-28 14:10:53,542 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14ed5005e750006 with negotiated timeout 40000 for client /172.17.0.10:53209
2015-07-28 14:10:53,628 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14ed5005e750007 with negotiated timeout 10000 for client /172.17.0.10:53210
2015-07-28 14:10:54,929 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14ed5005e750000 type:create cxid:0x12 zxid:0x1c6 txntype:-1 reqpath:n/a Error Path:/brokers/ids/9092 Error:KeeperErrorCode = NodeExists for /brokers/ids/9092
2015-07-28 14:10:56,000 [myid:] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14ed467dd920001, timeout of 7000ms exceeded
2015-07-28 14:10:56,000 [myid:] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14ed467dd920002, timeout of 7000ms exceeded
2015-07-28 14:10:56,000 [myid:] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14ed467dd920004, timeout of 7000ms exceeded
2015-07-28 14:10:56,000 [myid:] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14ed467dd920000, timeout of 6000ms exceeded
2015-07-28 14:10:56,000 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14ed467dd920001
2015-07-28 14:10:56,001 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14ed467dd920002
2015-07-28 14:10:56,001 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14ed467dd920004
2015-07-28 14:10:56,001 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14ed467dd920000
2015-07-28 14:11:00,000 [myid:] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14ed467dd920003, timeout of 10000ms exceeded
2015-07-28 14:11:00,001 [myid:] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14ed467dd920007, timeout of 10000ms exceeded
2015-07-28 14:11:00,001 [myid:] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14ed467dd920005, timeout of 10000ms exceeded
2015-07-28 14:11:00,001 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14ed467dd920003
2015-07-28 14:11:00,001 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14ed467dd920007
2015-07-28 14:11:00,001 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14ed467dd920005
2015-07-28 14:11:30,000 [myid:] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14ed467dd920006, timeout of 40000ms exceeded
2015-07-28 14:11:30,000 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14ed467dd920006