Apache ZooKeeper

Overview

Apache ZooKeeper GitHub Actions CI Travis CI Maven Central License

alt text

For the latest information about Apache ZooKeeper, please visit our website at:

https://zookeeper.apache.org

and our wiki, at:

https://cwiki.apache.org/confluence/display/ZOOKEEPER

Packaging/release artifacts

Either downloaded from https://zookeeper.apache.org/releases.html or found in zookeeper-assembly/target directory after building the project with maven.

apache-zookeeper-[version].tar.gz

    Contains all the source files which can be built by running:
    mvn clean install

    To generate an aggregated apidocs for zookeeper-server and zookeeper-jute:
    mvn javadoc:aggregate
    (generated files will be at target/site/apidocs)

apache-zookeeper-[version]-bin.tar.gz

    Contains all the jar files required to run ZooKeeper
    Full documentation can also be found in the docs folder

As of version 3.5.5, the parent, zookeeper and zookeeper-jute artifacts are deployed to the central repository after the release is voted on and approved by the Apache ZooKeeper PMC:

https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper

Java 8

If you are going to compile with Java 1.8, you should use a recent release at u211 or above.

Contributing

We always welcome new contributors to the project! See How to Contribute for details on how to submit patch through pull request and our contribution workflow.

Comments
  • ZOOKEEPER-3204: Reconfig tests are constantly failing on 3.5 after applying Java 11 fix

    ZOOKEEPER-3204: Reconfig tests are constantly failing on 3.5 after applying Java 11 fix

    Looks like that Netty 3 also having problems with correctly closing sockets and we keep getting "Bind failed" exceptions on the 3.5 branch.

    In this patch I'm trying to cherry-pick the upgrade-to-Netty4 commit by @ivmaykov from master branch and it looks like it solves the problem.

    I know this change is quite radical at this time of 3.5 release, but here're my 2 cents:

    • Netty 3 is rather old, would be a lot nicer to release 3.5 with a shiny new Netty lib support which will become stable in this release anyway (Netty is optional in 3.4)
    • After digging the net, I have absolutely no idea how to fix the Java11 binding issue with Netty 3. I suspect that the lib is as old as nobody bothers with the problem.

    Please take your time for a careful review.

    opened by anmolnar 50
  • ZOOKEEPER-3173: Quorum TLS - support PEM trust/key stores

    ZOOKEEPER-3173: Quorum TLS - support PEM trust/key stores

    Add support for loading key and trust stores from PEM files.

    Unfortunately, this PR includes 2 JIRAs, because it was quite difficult to untangle the two features as they were developed at the same time originally:

    • ZOOKEEPER-3173: Quorum TLS - support PEM trust/key stores
    • ZOOKEEPER-3175: Quorum TLS - test improvements

    Added support for PEM formatted key stores and trust stores

    • key store and trust store files can now be in PEM format as well as JKS.
    • Added config properties to tell ZK what type of trust/key store to load:
    • zookeeper.ssl.keyStore.type and zookeeper.ssl.trustStore.type for ClientX509Util
    • zookeeper.ssl.quorum.keyStore.type and zookeeper.ssl.quorum.trustStore.type for QuorumX509Util
    • store type properties could have the values "JKS", "PEM", or not set
    • leaving the type properties unset will cause auto-detection of the store type based on the file extension (".jks" or ".pem")

    Added test utilities for easily creating X509 certs and using them in unit tests

    • added new class X509TestContext and its friend, X509TestHelpers
    • rewrote some existing unit tests to use these classes, and added new tests that use them
    • some existing tests (i.e. QuorumSSLTest) should probably be ported to use this as well, haven't got around to it yet
    opened by ivmaykov 48
  • [ZOOKEEPER-1416] Persistent Recursive Watch

    [ZOOKEEPER-1416] Persistent Recursive Watch

    Here is a completed implementation for a persistent, recursive watch addition for ZK. These watches are set via a new method, addPersistentWatch() and are removed via the existing watcher removal methods. Persistent, recursive watches have these characteristics:

    • Once set, they do not auto-remove when triggered
    • They trigger for all event types (child, data, etc.) on the node they are registered for and any child znode recursively.
    • They are efficiently implemented by using the existing watch internals. A new class PathIterator walks up the path parent-by-parent when checking if a watcher applies.

    Persistent watcher specific tests are in PersistentWatcherTest.java. I'd appreciated feedback on other additional tests that should be added.

    opened by Randgalt 42
  • ZOOKEEPER-2122: add openssl to zookeeper c client to support ssl

    ZOOKEEPER-2122: add openssl to zookeeper c client to support ssl

    I have some problem with cmake, I was intend to find openssl as dependency using pkg_search_module, but it didn't work. so I have to set the include and lib directory in /opt/local, can anyone help me on this ? thanks.

    I made a wrong pull request to branch-3.5 #625, this one is a correction.

    opened by roodkcab 41
  • ZOOKEEPER-236: SSL Support for Atomic Broadcast protocol

    ZOOKEEPER-236: SSL Support for Atomic Broadcast protocol

    This is a work in progress, I wanted to get some feedback from the community while I worked on this. Please do not merge yet. Tests, documentation, and some cleanup still coming.

    This is a first pass at ssl support for the zookeeper quorum. It supports encrypting both leader election and normal operation.

    Rolling upgrades are supported via port unification (portUnification=true). This should only be used while performing a rolling upgrade.

    Some open questions:

    • Anyone have any ideas for better names for the configuration options (sslQuorum and portUnification currently).
    • I am using the same configuration that points to the truststore/keystore used for server <-> client ssl. Do they need to be separate?
    • Is port unification the correct approach for rolling upgrades? Is the impact from the use of BufferedSockets during the upgrade acceptable? See: http://stackoverflow.com/questions/25637039/detecting-ssl-connection-and-converting-socket-to-sslsocket http://stackoverflow.com/questions/6559859/is-it-possible-to-change-plain-socket-to-sslsocket
    • server <-> client ssl is implemented with netty. I did not feel that rewriting our server <-> server logic with netty was necessary given how easy ssl was to implement with standard java SSLSockets. Any arguments to the contrary?

    Thanks, Abe

    opened by afine 39
  • ZOOKEEPER-3152: Port ZK netty stack to netty4

    ZOOKEEPER-3152: Port ZK netty stack to netty4

    Summary: Ported the client connection netty stack from netty3 to netty4. This includes both the server side (NettyServerCnxn and friends) and the client side (ClientCnxnSocketNetty).

    Test Plan: Modified FourLetterWordsTest and NettyServerCnxnTest, plus manual testing on a regional ensemble.

    FB Reviewers: nixon

    opened by ivmaykov 38
  • ZOOKEEPER-4455: Move to https://reload4j.qos.ch/ (remove log4j1)

    ZOOKEEPER-4455: Move to https://reload4j.qos.ch/ (remove log4j1)

    • Drop Apache Log4j1
    • Replace with https://reload4j.qos.ch/, that is 100% compatible (same package names, it is actually a fork)
    • Remove logging implementation dependency from zookeeper server module (that is also used by clients)

    Client applications that upgrade to 3.6.4 or 3.7.1 won't have surprises regarding Maven exclusions, they were used to exclude log4j1, this is no more needed, but even if they forget to drop that exclusion it will be harmless, and they will never find reload4j as transitive dependency

    opened by eolivelli 36
  • ZOOKEEPER-3174: Quorum TLS - support reloading trust/key store

    ZOOKEEPER-3174: Quorum TLS - support reloading trust/key store

    Allow reloading SSL trust stores and key stores from disk when the files on disk change.

    Added support for reloading key/trust stores when the file on disk changes

    • new property sslQuorumReloadCertFiles which controls the behavior for reloading the key and trust store files for QuorumX509Util. Reloading of key and trust store for ClientX509Util is not in this PR but could be added easily
    • this allows a ZK server to keep running on a machine that uses short-lived certs that refresh frequently without having to restart the ZK process.
    opened by ivmaykov 35
  • ZOOKEEPER-3188: Improve resilience to network

    ZOOKEEPER-3188: Improve resilience to network

    This PR is the rebase of the previous pull request, so all the kudos should go to the original authors...

    In ZOOKEEPER-3188 we add ability to specify several addresses for quorum operations. Also added reconnection attempts if connection to leader lost.

    In this PR I rebased the changes on the current master, resolving some minor conflicts with:

    • ZOOKEEPER-3296: Explicitly closing the sslsocket when it failed handshake to prevent issue where peers cannot join quorum
    • ZOOKEEPER-3320: Leader election port stop listen when hostname unresolvable for some time
    • ZOOKEEPER-3385: Add admin command to display leader
    • ZOOKEEPER-3386: Add admin command to display voting view
    • ZOOKEEPER-3398: Learner.connectToLeader() may take too long to time-out

    I still want to test the feature manually (e.g. using docker containers with multiple virtual networks / interfaces). The steps to the manual test could be recorded in the google docs as well.

    Also I think we could add a few more unit tests where we are using multiple addresses. The current tests are using a single address only.

    Also the Zookeeper documentation needs to be changed (e.g. by a follow-up Jira?) to promote the new feature and the new config format (possibly including also the admin command documentation in relation with ZOOKEEPER-3386 and ZOOKEEPER-3461)

    opened by symat 31
  • ZOOKEEPER-3172: Quorum TLS - fix port unification to allow rolling upgrades

    ZOOKEEPER-3172: Quorum TLS - fix port unification to allow rolling upgrades

    Fix numerous problems with UnifiedServerSocket, such as hanging the accept() thread when the client doesn't send any data or crashing if less than 5 bytes are read from the socket in the initial read.

    Re-enable the "portUnification" config option.

    Fixed networking issues/bugs in UnifiedServerSocket

    • don't crash the accept() thread if the client closes the connection without sending any data
    • don't corrupt the connection if the client sends fewer than 5 bytes for the initial read
    • delay the detection of TLS vs. plaintext mode until a socket stream is read from or written to. This prevents the accept() thread from getting blocked on a read() operation from the newly connected socket.
    • prepending 5 bytes to PrependableSocket and then trying to read >5 bytes would only return the first 5 bytes, even if more bytes were available. This is fixed.
    opened by ivmaykov 31
  • ZOOKEEPER-3301:Enforce the quota limit

    ZOOKEEPER-3301:Enforce the quota limit

    • Thanks for the original work from ZOOKEEPER-1383, ZOOKEEPER-2593, ZOOKEEPER-451, especially the work from ZOOKEEPER-1383 contributed by Thawan Kooburat(I also sign off his name in the commit message) which also implemented the very valuable throughput quota.In the further, we will also do this.

    • zookeeper.enforeQuota. When enabled and the client exceeds the total bytes or children count hard quota under a znode, the server will reject the request and reply the client a QuotaExceededException by force. The default value is: false.

    • the checkQuota involves the create() and setData() api, not including the delete().

    • When users set the quota which's less than the existing stats, we give a thoughtful warning info.

    • the following code in the StatsTrack has a bad augmentability:

              if (split.length != 2) {
                  throw new IllegalArgumentException("invalid string " + stats);
              }
      

      we do a trick to solve it for the expansibility, but we will get a little strange quota info(Output quota for /c2 count=-1,bytes=-1=;byteHardLimit=-1;countHardLimit=5) when using listquota. some UTs has covered it.

    • the logic about checkQuota should be put in the PrepRequestProcessor, other than DataTree. we will get the following two negative effects if putting checkQuota in the DataTree:

        1. When the write request has exceeded the quota, the corresponding transaction log will load into disk successfully.It's not good, although it has any data inconsistency issue, because when the server restart, so long as the transaction logs are applied in the same order, the exceeded nodes will not be applied into the state machine.
        1. the client will be blocking and waiting for the response, because when throwing QuotaExceededException in the the DataTree, therc.stat will be null and BinaryOutputArchive#writeRecord will throw NPE.
        1. Overall, the pre-check about the write request should be done in the PrepRequestProcessor(at least before SyncRequestProcessor)(Look at an example from checkACL())
    • more detail in the ZOOKEEPER-3301.

    • [Added in 2020-02-25] use RateLogger to replace LOG to avoid quota exceed logs flooding the disk

    • A TODO improvement is: only users have admin permission can write to /zookeeper/quota(just like /zookeeper/config) to avoid some users' misoperation

    opened by maoling 30
  • ZOOKEEPER-4649: Upgrade netty to 4.1.86 because of CVE-2022-41915

    ZOOKEEPER-4649: Upgrade netty to 4.1.86 because of CVE-2022-41915

    Yesterday a new netty version was released fixing CVE-2022-41915 See https://nvd.nist.gov/vuln/detail/CVE-2022-41915 We need to upgrade the netty version.

    opened by symat 1
  • ZOOKEEPER-4649: Upgrade netty to 4.1.86 because of CVE-2022-41915

    ZOOKEEPER-4649: Upgrade netty to 4.1.86 because of CVE-2022-41915

    Yesterday a new netty version was released fixing CVE-2022-41915. See https://nvd.nist.gov/vuln/detail/CVE-2022-41915 We need to upgrade the netty version.

    opened by symat 2
  • ZOOKEEPER-4648 Add audit log for request process result or response.

    ZOOKEEPER-4648 Add audit log for request process result or response.

    ZOOKEEPER-4648 FinalRequestProcessor addAuditLog before the process of request and make failedTxn=false. But I think failedTxn should be true if the request can not pass the checkACL and throw KeeperException or other exceptions, since the err code after request processing is also important for audit.

    @param failedTxn whether audit is being done failed transaction for normal transaction
    
    public void processRequest(Request request) {
            ......
            Code err = Code.OK;
            try {
                ......
                AuditHelper.addAuditLog(request, rc);
    
                switch (request.type) {
                ......
                case OpCode.getAllChildrenNumber: {
                    lastOp = "GETACN";
                    ......
                    zks.checkACL(
                        request.cnxn,
                        zks.getZKDatabase().aclForNode(n),
                        ZooDefs.Perms.READ,
                        request.authInfo,
                        path,
                        null);
                    ......
                    break;
                }
                ......
                }
            } catch (SessionMovedException e) {
                ......
            } catch (KeeperException e) {
                err = e.code();
            } catch (Exception e) {
                ......
            }
    

    if the failedTxn == true or the rc.err != Code.OK, the log result will be FAILURE:

        private static Result getResult(ProcessTxnResult rc, boolean failedTxn) {
            if (failedTxn) {
                return Result.FAILURE;
            } else {
                return rc.err == KeeperException.Code.OK.intValue() ? Result.SUCCESS : Result.FAILURE;
            }
        }
    

    So we could add audit log after request processing and record the err code like below, the log info maybe more accurate.

            Code err = Code.OK;
            try { 
                 ......
            } catch (SessionMovedException e) {
                ......
            } catch (KeeperException e) {
                err = e.code();
            } catch (Exception e) {
                ......
            }
            rc.err = err.intValue();
            AuditHelper.addAuditLog(request, rc);
    
    opened by curie71 1
  • [ZOOKEEPER-4647] Tests don't pass on JDK20 because we try to mock InetAddress - upgrade Mockito

    [ZOOKEEPER-4647] Tests don't pass on JDK20 because we try to mock InetAddress - upgrade Mockito

    Some tests do not pass on latest JDK20 (and also in JDK19) because we are trying to mock InetAddress.

    More context here https://issues.apache.org/jira/browse/ZOOKEEPER-4647

    Unfortunately upgrading Mockito doesn't help

    opened by eolivelli 0
Owner
The Apache Software Foundation
The Apache Software Foundation
Mirror of Apache Storm

Master Branch: Storm is a distributed realtime computation system. Similar to how Hadoop provides a set of general primitives for doing batch processi

The Apache Software Foundation 6.4k Dec 26, 2022
Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora, and other frameworks on a dynamically shared pool of nodes.

The Apache Software Foundation 5k Dec 31, 2022
Apache ZooKeeper

Apache ZooKeeper For the latest information about Apache ZooKeeper, please visit our website at: https://zookeeper.apache.org and our wiki, at: https:

The Apache Software Foundation 11k Jan 3, 2023
😉PrettyZoo is a GUI for Zookeeper created by JavaFX and Apache Curator Framework.

?? Pretty nice Zookeeper GUI, Support Win / Mac / Linux Platform

vran 2.4k Jan 5, 2023
Highly-available version-controlled service configuration repository based on Git, ZooKeeper and HTTP/2

Visit the official web site for more information. Central Dogma Central Dogma is an open-source, highly-available and version-controlled service confi

LINE 503 Dec 20, 2022
⭐⭐⭐SpringBoot+Zookeeper+Dubbo打造分布式高并发商品秒杀系统

分布式高并发商品秒杀系统 介绍 快速启动 TODO 压测结果 Q&A 介绍 本项目为另一个项目dis-seckill的压力测试版本,重点工作为优化秒杀接口性能,提高单机系统并发瓶颈。 完整项目dis-seckill扩展包括 项目基础技术点和流程图介绍; 接口安全优化; 系统限流与降级服务; Ngin

mata 21 Sep 2, 2022
⭐⭐⭐⭐SpringBoot+Zookeeper+Dubbo打造分布式高并发商品秒杀系统

分布式高并发商品秒杀系统 介绍 快速启动 项目架构图 项目入门 TODO Q&A 参考资料 介绍 本项目是在dis-seckill上改进,项目名含义为分布式秒杀系统。采用微服务思想,意在提高秒杀系统的整体性能。 改进点: 优化秒杀流程,提高单机系统性能瓶颈。dis-seckill-test 接口安全

mata 190 Jan 2, 2023
A distributed lock that supports the use of Redis and Zookeeper, out of the box, fast and easy to use

lock-spring-boot-starter A distributed lock that supports the use of Redis and Zookeeper, out of the box, fast and easy to use 一款基于 Redis 和 Zookeeper

Pear Stack 9 Oct 15, 2022
Using this library, and writing a few lines of code, you can manage your own domain objects in ZooKeeper

Using this library, and writing a few lines of code, you can manage your own domain objects in ZooKeeper. It provides CRUD operations and change notifications out of the box.

Sahab 4 Oct 26, 2022
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

Oryx 2 is a realization of the lambda architecture built on Apache Spark and Apache Kafka, but with specialization for real-time large scale machine l

Oryx Project 1.8k Dec 28, 2022
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

Oryx 2 is a realization of the lambda architecture built on Apache Spark and Apache Kafka, but with specialization for real-time large scale machine l

Oryx Project 1.7k Mar 12, 2021
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

Oryx 2 is a realization of the lambda architecture built on Apache Spark and Apache Kafka, but with specialization for real-time large scale machine l

Oryx Project 1.8k Dec 28, 2022
Equivalent Exchange 3 Apache 2 Equivalent Exchange 3 pahimar Equivalent-Exchange-3. Mods for Minecraft. License: Apache 2 , .

Welcome to Equivalent Exchange 3! All versions are available here Minecraft Forums page Compiling EE3 - For those that want the latest unreleased feat

Rob Davis 709 Dec 15, 2022
Apache Solr is an enterprise search platform written in Java and using Apache Lucene.

Apache Solr is an enterprise search platform written in Java and using Apache Lucene. Major features include full-text search, index replication and sharding, and result faceting and highlighting.

The Apache Software Foundation 630 Dec 28, 2022
FLiP: StreamNative: Cloud-Native: Streaming Analytics Using Apache Flink SQL on Apache Pulsar

StreamingAnalyticsUsingFlinkSQL FLiP: StreamNative: Cloud-Native: Streaming Analytics Using Apache Flink SQL on Apache Pulsar Running on NVIDIA XAVIER

Timothy Spann 5 Dec 19, 2021
Apache Cayenne is an open source persistence framework licensed under the Apache License

Apache Cayenne is an open source persistence framework licensed under the Apache License, providing object-relational mapping (ORM) and remoting services.

The Apache Software Foundation 284 Dec 31, 2022
Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

Sparkler A web crawler is a bot program that fetches resources from the web for the sake of building applications like search engines, knowledge bases

USC Information Retrieval & Data Science 396 Dec 17, 2022
Apache ORC - the smallest, fastest columnar storage for Hadoop workloads

Apache ORC ORC is a self-describing type-aware columnar file format designed for Hadoop workloads. It is optimized for large streaming reads, but with

The Apache Software Foundation 576 Jan 2, 2023