Apache Pulsar - distributed pub-sub messaging system

Related tags

Messaging streaming messaging pubsub pulsar queuing event-streaming
Overview

logo

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.

Learn more about Pulsar at https://pulsar.apache.org

Main features

  • Horizontally scalable (Millions of independent topics and millions of messages published per second)
  • Strong ordering and consistency guarantees
  • Low latency durable storage
  • Topic and queue semantics
  • Load balancer
  • Designed for being deployed as a hosted service:
    • Multi-tenant
    • Authentication
    • Authorization
    • Quotas
    • Support mixing very different workloads
    • Optional hardware isolation
  • Keeps track of consumer cursor position
  • REST API for provisioning, admin and stats
  • Geo replication
  • Transparent handling of partitioned topics
  • Transparent batching of messages

Repositories

This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for components in the Pulsar ecosystem, including connectors, adapters, and other language clients.

Helm Chart

Ecosystem

Clients

Dashboard & Management Tools

Documentation

CI/CD

Build Pulsar

Requirements:

  • Java 8 JDK (for building Pulsar)
    • When building Pulsar on a higher version (higher than Java 8), the resulting artifacts are not compatible with Java 8 runtime because of some issues, such as issue 8445.
  • Maven 3.6.1+

Compile and install:

$ mvn install -DskipTests

Minimal build (This skips most of external connectors and tiered storage handlers)

mvn install -Pcore-modules

Run Unit Tests:

$ mvn test

Run Individual Unit Test:

$ cd module-name (e.g: pulsar-client)
$ mvn test -Dtest=unit-test-name (e.g: ConsumerBuilderImplTest)

Run Selected Test packages:

$ cd module-name (e.g: pulsar-broker)
$ mvn test -pl module-name -Dinclude=org/apache/pulsar/**/*.java

Start standalone Pulsar service:

$ bin/pulsar standalone

Check https://pulsar.apache.org for documentation and examples.

Setting up your IDE

Apache Pulsar is using lombok so you have to ensure your IDE setup with required plugins.

Intellij

Configure annotation processing in IntelliJ

  1. Open Annotation Processors Settings dialog box by going to Settings -> Build, Execution, Deployment -> Compiler -> Annotation Processors.

  2. Select the following buttons:

    1. "Enable annotation processing"
    2. "Obtain processors from project classpath"
    3. "Store generated sources relative to: Module content root"

  3. Set the generated source directories to be equal to the Maven directories:

    1. Set "Production sources directory:" to "target/generated-sources/annotations".
    2. Set "Test sources directory:" to "target/generated-test-sources/test-annotations".

  4. Click "OK".

  5. Install the lombok plugin in intellij.

Further configuration in IntelliJ

  • When working on the Pulsar core modules in IntelliJ, reduce the number of active projects in IntelliJ to speed up IDE actions and reduce unrelated IDE warnings.

    • In IntelliJ's Maven UI's tree view under "Profiles"
      • Activate "core-modules" Maven profile
      • De-activate "main" Maven profile
      • Run the "Reload All Maven Projects" action from the Maven UI toolbar. You can also find the action by the name in the IntelliJ "Search Everywhere" window that gets activated by pressing the Shift key twice.

  • Run the "Generate Sources and Update Folders For All Projects" action from the Maven UI toolbar. You can also find the action by the name in the IntelliJ "Search Everywhere" window that gets activated by pressing the Shift key twice. Running the action takes about 10 minutes for all projects. This is faster when the "core-modules" profile is the only active profile.

IntelliJ usage tips

  • In the case of compilation errors with missing Protobuf classes, ensure to run the "Generate Sources and Update Folders For All Projects" action.

  • All of the Pulsar source code doesn't compile properly in IntelliJ and there are compilation errors.

    • Use the "core-modules" profile if working on the Pulsar core modules since the source code for those modules can be compiled in IntelliJ.
    • Sometimes it might help to mark a specific project ignored in IntelliJ Maven UI by right-clicking the project name and select Ignore Projects from the menu.
    • Currently, it is not always possible to run unit tests directly from the IDE because of the compilation issues. As a workaround, individual test classes can be run by using the mvn test -Dtest=TestClassName command.

Eclipse

Follow the instructions here to configure your Eclipse setup.

Build Pulsar docs

Refer to the docs README.

Contact

Mailing lists
Name Scope
[email protected] User-related discussions Subscribe Unsubscribe Archives
[email protected] Development-related discussions Subscribe Unsubscribe Archives
Slack

Pulsar slack channel at https://apache-pulsar.slack.com/

You can self-register at https://apache-pulsar.herokuapp.com/

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Crypto Notice

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software: Pulsar uses the SSL library from Bouncy Castle written by http://www.bouncycastle.org.

Comments
  • [improve][broker] Add dynamic configuration for UniformLoadShedder

    [improve][broker] Add dynamic configuration for UniformLoadShedder

    Motivation

    Add dynamic configuration to UniformLoadShedder.

    Documentation

    Check the box below or label this PR directly.

    Need to update docs?

    • [ ] doc-required (Your PR needs to update docs and you will update later)

    • [x] doc-not-needed (Please explain why)

    • [ ] doc (Your PR contains doc changes)

    • [ ] doc-complete (Docs have been already added)

    type/enhancement component/broker doc-not-needed 
    opened by lordcheng10 117
  • [fix][broker] fix npe when doCacheEviction

    [fix][broker] fix npe when doCacheEviction

    Motivation

    fix npe when doCacheEviction: image

    Documentation

    Check the box below or label this PR directly.

    Need to update docs?

    • [ ] doc-required (Your PR needs to update docs and you will update later)

    • [x] no-need-doc (Please explain why)

    • [ ] doc (Your PR contains doc changes)

    • [ ] doc-added (Docs have been already added)

    • [x] doc-not-needed

    type/bug component/broker doc-not-needed 
    opened by lordcheng10 117
  • Catastrophic frequent random subscription freezes, especially on high-traffic topics.

    Catastrophic frequent random subscription freezes, especially on high-traffic topics.

    Describe the bug Topics randomly freeze, causing catastrophic topic outages on a weekly (or more frequent) basis. This has been an issue as long as my team has used Pulsar, and it's been communicated to a number of folks on the Pulsar PMC committee.

    (I thought an issue was already created for this bug, but I couldn't find it anywhere.)

    To Reproduce We have not figured out how to reproduce the issue. It's random (seems to be non-deterministic) and doesn't seem to have any clues in the broker logs.

    Expected behavior Topics should never just randomly stop working to where the only resolution is restarting the problem broker.

    Steps to Diagnose and Temporarily Resolve image Step 2: Check the rate out on the topic. (click on the topic in the dashboard, or do a stats on the topic and look at the "msgRateOut")

    If the rate out is 0 this is likely a frozen topic, but to verify do the following:

    In the pulsar dashboard, click on the broker that topic is living on. If you see that there are multiple topic that have a rate out of 0, then proceed to the next step, if not it could potentially be another issue. Investigate further. image

    image

    Step 3: Stop the broker on the server that the topic is living on. pulsar-broker stop .

    Step 4: Wait for the backlog to be consumed and all the functions to be rescheduled. (typically wait for about 5-10 mins)

    Environment:

    Docker on bare metal running: `apachepulsar/pulsar-all:2.4.0`
on CentOS.
Brokers are the function workers.

    This has been an issue with previous versions of Pulsar as well.

    Additional context

    Problem was MUCH worse with Pulsar 2.4.2, so our team needed to roll back to 2.4.0 (which has the problem, but it's less frequent). This is preventing the team from progressing in the use of Pulsar, and it's causing SLA problems with those who use our service.

    type/bug 
    opened by devinbost 94
  • [fix][broker] Fixed that newly added brokers cannot be assigned bundles for UniformLoadShedder

    [fix][broker] Fixed that newly added brokers cannot be assigned bundles for UniformLoadShedder

    Motivation

    According to the current logic, if the number of bundles on a broker is 0, the broker will not participate in bundle allocation, so that the newly started broker will not be allocated to the bundle: https://github.com/apache/pulsar/blob/da325fa64399642700d35b7d538a92b65c604bf5/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java#L78-L82

    Documentation

    Check the box below or label this PR directly.

    Need to update docs?

    • [ ] doc-required (Your PR needs to update docs and you will update later)

    • [x] doc-not-needed (Please explain why)

    • [ ] doc (Your PR contains doc changes)

    • [ ] doc-complete (Docs have been already added)

    doc-not-needed 
    opened by lordcheng10 86
  • [fix][broker] Support loadBalancerSheddingIntervalMinutes dynamic configuration

    [fix][broker] Support loadBalancerSheddingIntervalMinutes dynamic configuration

    Motivation

    Although the configuration loadBalancerSheddingIntervalMinutes is marked as dynamic=true, it cannot be changed dynamically. When loadSheddingTask is initialized, the shed interval is fixed: https://github.com/apache/pulsar/blob/4d64e2e66689381ebbb94fbfc03eb4e1dfba0405/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java#L1051-L1053

    https://github.com/apache/pulsar/blob/4d64e2e66689381ebbb94fbfc03eb4e1dfba0405/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L2087-L2093

    Documentation

    Check the box below or label this PR directly.

    Need to update docs?

    • [ ] doc-required (Your PR needs to update docs and you will update later)

    • [x] doc-not-needed (Please explain why)

    • [ ] doc (Your PR contains doc changes)

    • [ ] doc-complete (Docs have been already added)

    type/bug component/broker doc-not-needed cherry-picked/branch-2.10 release/2.10.2 cherry-picked/branch-2.11 
    opened by lordcheng10 81
  • Expose bookkeeper expose explicit lac configuration in broker.conf

    Expose bookkeeper expose explicit lac configuration in broker.conf

    Motivation

    Expose bookkeeper expose explicit lac configuration in broker.conf It's related to #3828 #4976, some Pulsar SQL users need to enable the explicitLacInterval, so that they can get the last message in Pulsar SQL.

    Verifying this change

    Added unit tests.

    Does this pull request potentially affect one of the following parts:

    If yes was chosen, please highlight the changes

    • Dependencies (does it add or upgrade a dependency): (no)
    • The public API: (no)
    • The schema: (no)
    • The default values of configurations: (yes)
    • The wire protocol: (no)
    • The rest endpoints: (no)
    • The admin cli options: (no)
    • Anything that affects deployment: (no)

    Documentation

    • Does this pull request introduce a new feature? (no)
    component/config release/2.5.1 
    opened by codelipenghui 59
  • [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)

    [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)

    Fixes #8407 Fixes #13865

    Motivation

    Current broker prometheus metrics are not grouped by metric type which causes issues in systems that read these metrics (e.g. DataDog).

    Prometheus docs states "All lines for a given metric must be provided as one single group" - https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#grouping-and-sorting

    Modifications Updated the namespace and topic prometheus metric generators to group the metrics under the appropriate type header. Updated function worker stats to include TYPE headers

    Verifying this change

    • [x] Make sure that the change passes the CI checks.

    This change added tests and can be verified as follows:

    Added unit test to verify all metrics are grouped under correct type header

    Does this pull request potentially affect one of the following parts:

    • Dependencies (does it add or upgrade a dependency): (no)
    • The public API: (no)
    • The schema: (no)
    • The default values of configurations: (no)
    • The wire protocol: (no)
    • The rest endpoints: (no)
    • The admin cli options: (no)
    • Anything that affects deployment: (no)

    Documentation

    Need to update docs?

    • [X] doc-not-needed Changes to match prometheus spec
    type/bug component/metrics doc-not-needed cherry-picked/branch-2.9 release/2.9.4 release/2.10.3 
    opened by marksilcox 57
  • [PIP-38] Support batch receive in java client.

    [PIP-38] Support batch receive in java client.

    Motivation

    Support messages batch receiving, some application scenarios can be made simpler. Users often increase application throughput through batch operations. For example, batch insert or update database.

    At present, we provide the ability to receive a single message. If users want to take advantage of batch operating advantages, need to implement a message collector him self. So this proposal aims to provide a universal interface and mechanism for batch receiving messages.

    For example:

    Messages messages = consumer.batchReceive();
insertToDB(messages);
consumer.acknowledge(messages);

    Verifying this change

    Added new UT to verify this change.

    Does this pull request potentially affect one of the following parts:

    If yes was chosen, please highlight the changes

    • Dependencies (does it add or upgrade a dependency): (no)
    • The public API: (yes)
    • The schema: (no)
    • The default values of configurations: (no)
    • The wire protocol: (no)
    • The rest endpoints: (no)
    • The admin cli options: (no)
    • Anything that affects deployment: (no)

    Documentation

    • Does this pull request introduce a new feature? (yes)
    • If yes, how is the feature documented? (docs and JavaDocs)
    component/client type/feature 
    opened by codelipenghui 56
  • [Issues 5709]remove the namespace checking

    [Issues 5709]remove the namespace checking

    [Issues 5709] remove the namespace checking Support multiple topic subscriptions across multiple namespace

    Fixes #5709

    Motivation

    Support multiple topic subscriptions across multiple namespace

    Modifications

    remove the namespace checking

    component/client 
    opened by huangdx0726 54
  • [Issue 7903][proxy]Enable AutoTopicCreationType partitioned through proxy

    [Issue 7903][proxy]Enable AutoTopicCreationType partitioned through proxy

    (If this PR fixes a github issue, please add Fixes #<xyz>.)

    Fixes #7903

    (or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)

    Master Issue: #7903

    Motivation

    proxy should always fetch topic metadata from broker

    Modifications

    • Proxy handle PartitionMetadataReqeust by selecting and fetching from an available broker instead of using current ZK metadata
    • Fix an issue(incorrect topic to receive message leads consumer blocking) on ProxyTest#testRegexSubscription unit test

    Verifying this change

    This change added tests and can be verified as follows:

    • run ProxyTest#testAutoCreateTopic

    Does this pull request potentially affect one of the following parts:

    If yes was chosen, please highlight the changes

    • Dependencies (does it add or upgrade a dependency): (no)
    • The public API: (no)
    • The schema: ( no )
    • The default values of configurations: (no)
    • The wire protocol: (no)
    • The rest endpoints: (no)
    • The admin cli options: (no)
    • Anything that affects deployment: (no)
    type/bug component/proxy cherry-picked/branch-2.7 release/2.7.3 
    opened by rudy2steiner 52
  • Fixed avro schema decode error in functions

    Fixed avro schema decode error in functions

    Fixes #5503

    Motivation

    In functions, it will encounter ClassCastException when using the Avro schema for topics.

    Exception in thread "main" java.lang.ClassCastException: org.apache.pulsar.shade.org.apache.avro.generic.GenericData$Record cannot be cast to io.streamnative.KeyValueSchemaTest$Foo2
	at io.streamnative.KeyValueSchemaTest.testConsumerByPythonProduce(KeyValueSchemaTest.java:412)
	at io.streamnative.KeyValueSchemaTest.main(KeyValueSchemaTest.java:305)

    Modifications

    In functions, when using Avro schema specific the ClassLoader for ReflectDatumReader.

    type/bug component/schemaregistry release/2.5.1 
    opened by gaoran10 52
  • Improve getManagedLedgerOffloader equality check for OffloadPoliciesImpl

    Improve getManagedLedgerOffloader equality check for OffloadPoliciesImpl

    Search before asking

    • [X] I searched in the issues and found nothing similar.

    Motivation

    The equality check on the following line does not return true based on my testing from a couple years ago :) (I meant to open this issue a while ago, but I just found the reference today, and since the less than optimal code is present, now is as good a time as ever.)

    https://github.com/apache/pulsar/blob/82feb13949e9387098e93d0484e1c0f622b7e0d5/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java#L1346

    The problem comes in how the two offloadPolicies are instantiated. Reading through the code may reveal the issue. I discovered it by running through the code with a debugger. Then, I noticed that certain fields evaluate to null while others are "" depending on how they the policy object is initialized.

    Solution

    • Fix how the objects are initialized.
    • Also, we might want to explore if the ledgerOffloaderMap has a leak of offloaders. I don't see any interactions that remove entries from the ledgerOffloaderMap map.

    Alternatives

    Fix how the object equality is checked. This could have unintended consequences. Correcting how the objects are deserialized seems like the better approach to me.

    Anything else?

    I will try to submit a PR, but I might not have time.

    Are you willing to submit a PR?

    • [ ] I'm willing to submit a PR!
    type/enhancement component/tieredstorage 
    opened by michaeljmarshall 2
  • [improve][broker] PIP-192 Moved the common broker load data feature(weightedMaxEMA) to BrokerLoadData

    [improve][broker] PIP-192 Moved the common broker load data feature(weightedMaxEMA) to BrokerLoadData

    Master Issue: https://github.com/apache/pulsar/issues/16691

    Motivation

    We will start raising PRs to implement PIP-192, https://github.com/apache/pulsar/issues/16691

    Modifications

    This PR moved the common broker load data feature(weightedMaxEMA) to BrokerLoadData.

    As weightedMaxEMA is available in the BrokerLoadData by this change, this PR

    • removed brokerAvgResourceUsageWithWeight in LeastResourceUsageWithWeight -- we don't need to separately compute and store weightedMaxEMA in LeastResourceUsageWithWeight and TransferShedder.

    • updated the unit tests.

    Verifying this change

    • [x] Make sure that the change passes the CI checks.

    This change added tests and can be verified as follows:

    • Updated unit tests.

    Does this pull request potentially affect one of the following parts:

    If the box was checked, please highlight the changes

    • [ ] Dependencies (add or upgrade a dependency)
    • [ ] The public API
    • [ ] The schema
    • [ ] The default values of configurations
    • [ ] The threading model
    • [ ] The binary protocol
    • [ ] The REST endpoints
    • [ ] The admin CLI options
    • [ ] Anything that affects deployment

    Documentation

    • [ ] doc
    • [ ] doc-required
    • [x] doc-not-needed
    • [ ] doc-complete

    We will have separate PRs to update the Doc later.

    Matching PR in forked repository

    PR in forked repository: https://github.com/heesung-sn/pulsar/pull/20

    doc-not-needed 
    opened by heesung-sn 0
  • [improve] added TopicEventListener / topic events for the BrokerService

    [improve] added TopicEventListener / topic events for the BrokerService

    Motivation

    Topic events for KOP

    Modifications

    Added TopicEventsListener interface, topic events dispatching for the BrokerService

    Verifying this change

    • [ ] Make sure that the change passes the CI checks.

    This change added unit tests

    Does this pull request potentially affect one of the following parts:

    If the box was checked, please highlight the changes

    • [ ] Dependencies (add or upgrade a dependency)
    • [ ] The public API
    • [ ] The schema
    • [ ] The default values of configurations
    • [ ] The threading model
    • [ ] The binary protocol
    • [ ] The REST endpoints
    • [ ] The admin CLI options
    • [ ] The metrics
    • [ ] Anything that affects deployment

    Documentation

    Added applicable JavaDoc

    • [ ] doc
    • [ ] doc-required
    • [x] doc-not-needed
    • [ ] doc-complete

    Matching PR in forked repository

    PR in forked repository: https://github.com/dlg99/pulsar/pull/7

    doc-not-needed ready-to-test 
    opened by dlg99 0
  • [revert]

    [revert] "[fix][broker] change name limitTime to limitTimeInSec (#19053)"

    This reverts commit fd5037ddeab4746202539509818f2b7977698537.

    Motivation

    As described in https://github.com/apache/pulsar/pull/19053#pullrequestreview-1234929550, the commit to be reverted introduced a breaking change in a non-backwards compatible way. This is especially important because the datastructure in question is used for ser/de of zk metadata. Instead of making one-off changes to variable names, I would prefer to first establish a standard/spec, and then apply it generally to all names. Pulsar certainly has room to improve names, but we must prioritize the user experience when changing these names. We also need to make sure that we cover upgrade and downgrade scenarios so that we do not break environments.

    Modifications

    • Revert #19053.

    Verifying this change

    This should be trivial.

    Does this pull request potentially affect one of the following parts:

    This reverts a breaking change.

    Documentation

    • [x] doc-not-needed

    I'll send an email to the mailing list to discuss how we should handle these in the future.

    type/cleanup doc-not-needed ready-to-test 
    opened by michaeljmarshall 2
  • [improve][broker] Add parameter check for create/update cluster.

    [improve][broker] Add parameter check for create/update cluster.

    Motivation

    In the current implementation, we didn't have any cluster data check for create/update cluster. it may cause some problems when the user sets the wrong URL.

    Modifications

    • Add parameters check.

    Verifying this change

    • [x] Make sure that the change passes the CI checks.

    Does this pull request potentially affect one of the following parts:

    • [x] The REST endpoints

    Documentation

    • [x] doc-not-needed
    component/broker doc-not-needed ready-to-test 
    opened by mattisonchao 2
  • [Bug] pulsar-2.10.2 Please redo the lookup. Request is denied: namespace=public/default

    [Bug] pulsar-2.10.2 Please redo the lookup. Request is denied: namespace=public/default

    Search before asking

    • [X] I searched in the issues and found nothing similar.

    Version

    2023-01-06T16:32:02,267+0800 [pulsar-ph-mqtt-48-14] WARN org.apache.pulsar.broker.service.BrokerService - Namespace bundle for topic (persistent://public/default/test-13-1) not served by this instance. Please redo the lookup. Request is denied: namespace=public/default 2023-01-06T16:32:02,241+0800 [pulsar-ph-mqtt-48-1] ERROR io.streamnative.pulsar.handlers.mqtt.support.MQTTBrokerProtocolMethodProcessor - [Publish] [persistent://public/default/test-13-1] Write MqttPublishMessage[fixedHeader=MqttFixedHeader[messageType=PUBLISH, isDup=false, qosLevel=AT_MOST_ONCE, isRetain=false, remainingLength=475], variableHeader=MqttPublishVariableHeader[topicName=persistent://public/default/test-13-1, packetId=-1], payload=PooledSlicedByteBuf(ridx: 0, widx: 436, cap: 436/436, unwrapped: PooledUnsafeDirectByteBuf(ridx: 24814, widx: 131448, cap: 262144))] to Pulsar topic failed. org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://public/default/test-13-1) not served by this instance. Please redo the lookup. Request is denied: namespace=public/default at org.apache.pulsar.broker.service.BrokerService.lambda$checkTopicNsOwnership$84(BrokerService.java:1855) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_131] at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_131] at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1847) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.broker.service.BrokerService.loadOrCreatePersistentTopic(BrokerService.java:1346) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$27(BrokerService.java:974) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.put(ConcurrentOpenHashMap.java:404) ~[org.apache.pulsar-pulsar-common-2.10.2.jar:2.10.2] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.computeIfAbsent(ConcurrentOpenHashMap.java:238) ~[org.apache.pulsar-pulsar-common-2.10.2.jar:2.10.2] at org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:973) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:938) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at io.streamnative.pulsar.handlers.mqtt.utils.PulsarTopicUtils.lambda$getTopicReference$4(PulsarTopicUtils.java:88) ~[?:?] at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_131] at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_131] at io.streamnative.pulsar.handlers.mqtt.utils.PulsarTopicUtils.getTopicReference(PulsarTopicUtils.java:88) ~[?:?] at io.streamnative.pulsar.handlers.mqtt.utils.PulsarTopicUtils.lambda$getTopicReference$1(PulsarTopicUtils.java:72) ~[?:?] at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_131] at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_131] at io.streamnative.pulsar.handlers.mqtt.utils.PulsarTopicUtils.lambda$getTopicReference$2(PulsarTopicUtils.java:71) ~[?:?] at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_131] at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_131] at io.streamnative.pulsar.handlers.mqtt.utils.PulsarTopicUtils.getTopicReference(PulsarTopicUtils.java:58) ~[?:?] at io.streamnative.pulsar.handlers.mqtt.AbstractQosPublishHandler.getTopicReference(AbstractQosPublishHandler.java:52) ~[?:?] at io.streamnative.pulsar.handlers.mqtt.AbstractQosPublishHandler.writeToPulsarTopic(AbstractQosPublishHandler.java:92) ~[?:?] at io.streamnative.pulsar.handlers.mqtt.AbstractQosPublishHandler.writeToPulsarTopic(AbstractQosPublishHandler.java:59) ~[?:?] at io.streamnative.pulsar.handlers.mqtt.support.Qos0PublishHandler.publish(Qos0PublishHandler.java:40) ~[?:?] at io.streamnative.pulsar.handlers.mqtt.support.MQTTBrokerProtocolMethodProcessor.doPublish(MQTTBrokerProtocolMethodProcessor.java:225) ~[?:?] at io.streamnative.pulsar.handlers.mqtt.support.MQTTBrokerProtocolMethodProcessor.processPublish(MQTTBrokerProtocolMethodProcessor.java:187) ~[?:?] at io.streamnative.pulsar.handlers.mqtt.MQTTCommonInboundHandler.channelRead(MQTTCommonInboundHandler.java:79) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.streamnative.pulsar.handlers.mqtt.adapter.CombineHandler.channelRead(CombineHandler.java:31) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final] at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:349) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) ~[io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) ~[io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]

    Minimal reproduce step

    org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://public/default/test-13-1) not served by this instance. Please redo the lookup. Request is denied: namespace=public/default

    What did you expect to see?

    org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://public/default/test-13-1) not served by this instance. Please redo the lookup. Request is denied: namespace=public/default

    What did you see instead?

    org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://public/default/test-13-1) not served by this instance. Please redo the lookup. Request is denied: namespace=public/default

    Anything else?

    org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://public/default/test-13-1) not served by this instance. Please redo the lookup. Request is denied: namespace=public/default

    Are you willing to submit a PR?

    • [X] I'm willing to submit a PR!
    type/bug 
    opened by BaoHuaWang 0
Releases(v2.10.3)

  • v2.10.3(Jan 5, 2023)

    Broker

    • [improve][broker] Omit making a copy of CommandAck when there are no broker interceptors #18997
    • [fix][broker] Copy proto command fields into final variables in ServerCnx #18987
    • [improve][broker]Add a cache eviction policy：Evicting cache data by the slowest markDeletedPosition #14985
    • [fix][broker] Fix broker cache eviction of entries read by active cursors #17273
    • [enh][broker] Add metrics for entry cache insertion, eviction #17248
    • [cleanup][broker] Extracted interface for EntryCacheManager #15933
    • [improve][broker] Support dynamic update cache config #13679
    • [improve][broker] refactor ManagedLedger cacheEvictionTask implement #14488
    • [improve][broker] Add logs for why namespace bundle been split (#19003)
    • [fix][broker] Fix deadlock in PendingAckHandleImpl (#18989)
    • [fix][broker] Branch-2.10 Avoid endless blocking call. (#18914)
    • [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865) #15558
    • [fix][broker]Update interceptor handler exception (#18940)
    • [fix][broker] Fix incorrect bundle split count metric (#17970)
    • [fix][broker] Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer in PrometheusMetricsGeneratorUtils (#18747)
    • [improve][broker] Make Consumer#equals more effective (#18662)
    • [fix][broker] In the trimDeletedEntries method, release the removed entry (#18305)
    • [improve][broker] Remove locallyAcquiredLock when removeOwnership (#18197)
    • [improve][broker] Avoid unnecessary creation of BitSetRecyclable objects #17998
    • [fix][broker] add return for PersistentMessageExpiryMonitor#findEntryFailed #17816
    • [fix][ml] Persist correct markDeletePosition to prevent message loss (#18237)
    • [improve][broker] Support setting forceDeleteTenantAllowed dynamically (#18192)
    • Make BookieId work with PulsarRegistrationDriver (second take) #17922
    • [fix][broker] Extract additional servlets to the default directory by… (#17477)
    • [improve][broker]Improve PersistentMessageExpiryMonitor expire speed when ledger not existed (#17842)
    • [fix][broker]Fix mutex never released when trimming (#17911)
    • [fix][broker] Fix if dynamicConfig item in ZK do not exist in broker cause NPE. #17705
    • [fix][broker] Fix system service namespace create internal event topic. (#17867)
    • [bugfix] Prevent Automatic Topic Creation during namespace deletion #17609
    • [fix][broker]Cache invalidation due to concurrent access (#18076)
    • [fix][broker]add test case for deleting namespace fail when has partitioned system topic (#17338)
    • Skip creating a subscription replication snapshot if no messages have been published after the topic gets activated on a broker #16618
    • [fix][broker] Fix uncompleted future when get the topic policies of a deleted topic (#18824)
    • [fix][broker] Fix delete system topic clean topic policy (#18823)
    • [improve][broker][PIP-149]Make getList async #16221
    • [fix][flaky-test]ManagedCursorMetricsTest.testManagedCursorMetrics #16878
    • [improve][broker] System topic writer/reader connection not counted (#18603)
    • [improve][broker] System topic writer/reader connection not counted. (#18369)
    • [fix][broker]unify time unit at dropping the backlog on a topic #17957[fix][broker]unify time unit at dropping the backlog on a topic #17957
    • [improve][broker] Support setting ForceDeleteNamespaceAllowed dynamically #18181
    • [fix][broker] Fix getPositionAfterN infinite loop. (#17971)
    • [fix][broker] Update the log print content of createSubscriptions (#18024)
    • [Broker] Make PersistentTopicsBase#internalGetPartitionedMetadata async #14153
    • [fix][broker] Fix duplicated schemas creation #18701
    • [improve][broker] Using handle instead of handleAsync to avoid using common pool thread (#17403)
    • [fix][broker] Fix dispatch duplicated messages with Exclusive mode. (#17237)
    • [fix][broker] Fix the order of resource close in the InMemoryDelayedDeliveryTracker (#18000)
    • [improve][broker] reduce code duplication to avoid endless wait CompletableFuture (#14853)
    • [improve][ML] Print log when delete empty ledger. (#17859)
    • [improve] clean the empty topicAuthenticationMap in zk when revoke permission (#16815)
    • [fix][broker] fix can not revoke permission after update topic partition (#17393)
    • [fix][broker] Fix Npe thrown by splitBundle (#17370)
    • [fix][broker] Fix executeWithRetry result is null (#17694)
    • [fix][broker] Fix SystemTopicBasedTopicPoliciesService NPE issue (#17602)
    • [fix][broker] Fix broker cache eviction of entries read by active cursors #17273
    • [fix][broker] Fix namespace can not be deleted by force (#18686)
    • [fix][broker] Create replicated subscriptions for new partitions when needed (#18659)
    • [fix][broker] DnsResolverUtil.TTL should be greater than zero (#18565)
    • [fix][broker] Correctly set byte and message out totals per subscription (#18451)
    • [fix][broker] fix delete_when_subscriptions_caught_up doesn't work while have active consumers (#18320)
    • [improve][broker]consumer backlog eviction policy should not reset read position for consumer (#18037)
    • [fix][broker] Fix NPE when ResourceGroupService execute scheduled task. #17840
    • [fix][storage] Autorecovery default reppDnsResolverClass to ZkBookieRackAffinityMapping (#15640)
    • [broker] Fixed delayed delivery after read operation error (#18098)
    • [fix][broker] Fix issue where leader broker information isn't available after 10 minutes (#17401)
    • [fix][broker] Make full async call in PulsarAuthorizationProvider (#18050)

    Transaction

    • [fix][txn] Transaction buffer recover blocked by readNext #18833
    • [fix][txn] transaction pending ack store future not completely problem (#18943)
    • [fix][txn] Fix PendingAckHandleImpl when pendingAckStoreProvider.checkInitializedBefore failed (#18859)
    • [improve][txn] Add getState in transaction for client API (#17423)
    • [improve][txn] Implementation of Delayed Transaction Messages (#17548)

    Pulsar IO and Pulsar Functions

    • [fix][io] Only bundle kafka schema registry client (#18931)
    • [fix][fn] Typo in method name (#18844)
    • [fix][function] Fix invalid metric type gauge (#18129)
    • [fix][fn] fix function failed to start if no typeClassName provided in FunctionDetails (#18111)

    Tiered Storage

    • [fix][offload] Fix numerical overflow bug while reading data from tiered storage (#18595)
    • [fix][offload] Fix memory leak while Offloading ledgers (#18500)

    Pulsar SQL

    • [fix][sql] Fix message without schema issue. (#18745)

    CLI

    • [fix][cli] Fix CLI client produce don't able to use multiple -m send multiple messages (#18238)
    • [fix][cli] Check numMessages after incrementing counter #17826

    Admin

    • [improve][admin] Fix NPE in admin-CLI topic stats command (#18326)
    • [improve][admin] add topic name and sub name for NotFound error message (#15606)
    • PIP-105: new API to get subscription properties #16095
    • [fix][admin] returns 4xx error when pulsar-worker-service is disabled and trying to access it (#17901)
    • Pulsar Admin: grab contextual stacktrace for sync methods (#14620)
    • [fix][admin] Fix NPE when get OffloadThreshold on namespace (#18061)

    Security

    • [fix][sec] Upgrade scala-library to get rid of CVE-2022-36944 (#18021)
    • [Improve][Auth]Update authentication failed metrics report (#17787)
    • [fix][sec] Upgrade jackson-databind to 2.13.4.2 to get rid of CVE-2022-42003 (#18394)
    • [fix][sec] Upgrade protobuf to 3.19.6 to get rid of CVE-2022-3171 (#18086)
    • [fix][sec] File tiered storage: upgrade jettison to get rid of CVE-2022-40149 (#18022)
    • [fix][sec] Upgrade JacksonXML to 2.13.4 (#18020)

    Dependency & Library updates

    • Bump jackson version from 2.13.2 to 2.13.3 #16508
    • Remove invalid Netty system property which never was valid for Netty 4.1.x (#13563)
    • [improve][misc] Upgrade Netty to 4.1.86.Final and Netty Tcnative to 2.0.54.Final (#18599)
    • [improve][build] Remove versions that are handled by netty-bom (#18629)

    CI & Test

    • [improve][test] force initialize field to avoid polluted by mocks (#17022)
    • [fix][flaky-test]ManagedCursorMetricsTest.testManagedCursorMetrics (#16878)
    • [improve][test] remove WhiteBox on MockZooKeeper #17579
    • [improve][test] try remove whitebox on MockZooKeeper (#17579)
    • [improve][test] Add test case for system topic schema not compatible bug. (#17992)
    • [fix][flaky-test]BatchMessageWithBatchIndexLevelTest.testBatchMessageAck (#17436)
    • [fix][test]The pulsarMessageOverhead in 2.10 is different (#18704)
    • [improve][test] Add subscribing regex topic test for delete_when_subscriptions_caught_up. (#18368)

    Java Client

    • [fix][client] For exclusive subscriptions, if two consumers are created repeatedly, the second consumer will block #18633
    • [fix][client] Fixes batch_size not checked in MessageId#fromByteArrayWithTopic (#18405)
    • [fix][client] Fix possible npe (#18406)
    • [fix][client] Fix exception when calling loadConf on a ConsumerBuilder that has a KeySharedPolicy (#18345)
    • [fix][client] Support LocalDateTime Conversion (#18334)
    • [fix] [pulsar-client] Fix pendingLookupRequestSemaphore leak when Ser… (#18219)
    • [fix][client] Fix failover/exclusive consumer with batch cumulate ack issue. (#18454)
    • [improve][java-client]Add init capacity for messages in BatchMessageContainerImpl (#17822)
    • [fix] [pulsar-client] Fix pendingLookupRequestSemaphore leak when channel inactive (#17856)
    • [fix][client] Unwrap completion exception for Lookup Services (#17717)
    • [fix][client] Avoid redelivering duplicated messages when batching is enabled #18486
    • [fix][client] Fix multi-topic consumer stuck after redeliver messages (#18491)
    • [fix][client] Fix IllegalThreadStateException when using newThread in ExecutorProvider.ExtendedThreadFactory
    • [fix][client] Fix NPE of MultiTopicsConsumerImpl due to race condition (#18287)

    Others

    • [fix][metrics]wrong metrics text generated when label_cluster specified #17704
    • [fix][meta] fix getChildren in MemoryMetadataStore and EtcdMetadataStore (#18172)
    • [improve][schema] Change update schema auth from tenant to produce (#18074)
    • [fix][schema]ledger handle leak when update schema (#17283)
    • [fix][metrics] fixed ProxyStats to use common.stats.JvmMetrics (#15692)
    • [refactor][java] Improve docs and code quality about KeyValueSchema usages (#17256)
    • Allow to configure and disable the size of lookahead for detecting fixed delays in messages #17907
    • [refactor][java] Unify the acknowledge process for batch and non-batch message IDs #17833
    • [fix][common] Fix parsing partitionedKey with Base64 encode issue. (#17687)
    • [fix][schema] Fix cherry-pick issue from #18283 (#18555)
    • [fix][monitor] fix metrics string encoding (#18138)
    Source code(tar.gz)
    Source code(zip)

  • v2.9.4(Jan 5, 2023)

    What's Changed

    • [branch-2.9]add message ack rate by @tjiuming in https://github.com/apache/pulsar/pull/16146
    • [branch-2.9] Fix cherry-pick issues. by @mattisonchao in https://github.com/apache/pulsar/pull/16346
    • [Branch-2.9][Cherry-pick] fix bug: fail to expose managed ledger client stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true by @HQebupt in https://github.com/apache/pulsar/pull/16343
    • [Branch 2.9] Fix compaction subscription acknowledge Marker msg issue. by @mattisonchao in https://github.com/apache/pulsar/pull/16348
    • [Branch-2.9] Fix passing incorrect authentication data by @mattisonchao in https://github.com/apache/pulsar/pull/16347
    • [Branch-2.9][transaction] Cmd-Subscribe and Cmd-Producer will not succeed even after 100 retries. by @mattisonchao in https://github.com/apache/pulsar/pull/16349
    • [Branch-2.9][Cherry-pick] Fix get non-persistent topics issue in Namespaces. by @Technoboy- in https://github.com/apache/pulsar/pull/16517
    • [branch-2.9] Fix MockStatic method caused the exception by @mattisonchao in https://github.com/apache/pulsar/pull/16628
    • [branch-2.9] Fix compatible issue. by @mattisonchao in https://github.com/apache/pulsar/pull/16779
    • [branch-2.9]Add back FutureUtil#waitForAll and FutureUtil#waitForAny methods with List parameter by @mattisonchao in https://github.com/apache/pulsar/pull/16794
    • [broker][monitoring][fix] fix pulsar_subscription_msg_ack_rate by @tjiuming in https://github.com/apache/pulsar/pull/16866
    • [Branch-2.9][fix][client] Fix client memory limit currentUsage leak and semaphore release duplicated in ProducerImpl. by @Nicklee007 in https://github.com/apache/pulsar/pull/16971
    • [improve][broker] Optimise msgOutCounter and bytesOutCounter (#16214) (#16286) by @nahguam in https://github.com/apache/pulsar/pull/16771
    • [Branch-2.9][fix][client] Fix MaxQueueSize semaphore release leak in createOpSendMsg. by @Nicklee007 in https://github.com/apache/pulsar/pull/16958
    • [Branch 2.9][fix][broker] Upgrade log4j2 version to 2.18.0 (#16884) by @liudezhi2098 in https://github.com/apache/pulsar/pull/16995
    • [improve][authentication][branch-2.9] Support for get token from HTTP params by @nodece in https://github.com/apache/pulsar/pull/16987
    • [Branch-2.9]Upgrade rocksdb version to 6.16.4 to keep sync with bookkeeper by @hangc0276 in https://github.com/apache/pulsar/pull/17246
    • [branch-2.9] Support mock final class with PowerMockito by @mattisonchao in https://github.com/apache/pulsar/pull/17327
    • [branch-2.9][fix][load-balancer] skip mis-configured resource usage(>100%) in loa… by @heesung-sn in https://github.com/apache/pulsar/pull/17285
    • [fix][broker][branch-2.9] Fix calculate avg message per entry by @coderzc in https://github.com/apache/pulsar/pull/17281
    • [branch-2.9] Rename test file name from *Test2.java to *Test.java to run all tests correctly by @equanz in https://github.com/apache/pulsar/pull/17048
    • [fix][c++][branch-2.9]Fix wrong consumers size: execute callback before executing readerCreatedCallback_ by @coderzc in https://github.com/apache/pulsar/pull/17629
    • [branch-2.9][fix][broker]Persist cursor info error when cursor close by @poorbarcode in https://github.com/apache/pulsar/pull/17604
    • [branch-2.9][cherry-pick] Fix producer/consume permission can’t get schema. by @Technoboy- in https://github.com/apache/pulsar/pull/17730
    • [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865) by @marksilcox in https://github.com/apache/pulsar/pull/17618
    • [branch-2.9] Group prometheus metrics. by @tjiuming in https://github.com/apache/pulsar/pull/17852
    • [branch-2.9] Fix cherry-pick issue by @codelipenghui in https://github.com/apache/pulsar/pull/17894
    • [fix][broker][branch-2.9] Fix system topic schema not compatible bug by @dragonls in https://github.com/apache/pulsar/pull/17986
    • [improve][authentication][branch-2.9] Improve get the basic authentication config by @nodece in https://github.com/apache/pulsar/pull/16944
    • [fix][client][txn] Use PulsarClient HashWheelTimer to schedule producer batch trigger task by @lifepuzzlefun in https://github.com/apache/pulsar/pull/18058
    • [fix][sec] File tiered storage: upgrade jettison to get rid of CVE-2022-40149 by @mattisonchao in https://github.com/apache/pulsar/pull/18105
    • [branch-2.9] Fixed key-shared delivery of messages with interleaved delays. by @mattisonchao in https://github.com/apache/pulsar/pull/18108
    • [branch-2.9] Fix license check issue by @Technoboy- in https://github.com/apache/pulsar/pull/18117
    • [fix][cpp][branch-2.9] Fix zlib dowload path. by @shibd in https://github.com/apache/pulsar/pull/18127
    • [fix][sec][branch-2.9] Updating dependencies to get rid of CVEs brought in with kafka and lo… by @alpreu in https://github.com/apache/pulsar/pull/18304
    • [cherry-pick][branch-2.9] Fix NPE when get OffloadThreshold on namespace. by @congbobo184 in https://github.com/apache/pulsar/pull/18638
    • [cherry-pick][branch-2.9] cherry-pick #17736 problem and DnsResolverTest license header by @congbobo184 in https://github.com/apache/pulsar/pull/18640
    • [cherry-pick][branch-2.9] Fix NPE when ResourceGroupService execute scheduled task. by @congbobo184 in https://github.com/apache/pulsar/pull/18685
    • [cherry-pick][branch-2.9] fix cherry-pick #17592 missing code and flaky test testFailoverSingleAckedPartitionedTopic by @congbobo184 in https://github.com/apache/pulsar/pull/18702
    • [branch-2.9] Fix no durable cursor leak problem by @gaozhangmin in https://github.com/apache/pulsar/pull/18710
    • [branch-2.9][fix][broker] Fix duplicated schemas creation (#18701) by @codelipenghui in https://github.com/apache/pulsar/pull/18762
    • [cherry-pick][branch-2.9] fix cherry-pick #17609 #17957 import by @congbobo184 in https://github.com/apache/pulsar/pull/18793
    • [cherry-pick][branch-2.9] make getList async #16221 by @congbobo184 in https://github.com/apache/pulsar/pull/18811
    • [cherry-pick][branch-2.9] Cherry-pick #18307 (Fix can not delete namespace by force) by @liangyepianzhou in https://github.com/apache/pulsar/pull/18803
    • [cherry-pick][branch-2.9] change proto license version by @congbobo184 in https://github.com/apache/pulsar/pull/18821
    • [cherry-pick][branch-2.9] Fix delete system topic clean topic policy (#18823) by @congbobo184 in https://github.com/apache/pulsar/pull/18831

    Full Changelog: https://github.com/apache/pulsar/compare/v2.9.3...v2.9.4

    Source code(tar.gz)
    Source code(zip)

  • v2.10.2(Oct 28, 2022)

    Important notice

    • [fix][admin] Fix get non-persistent topics issue in Namespaces. #16170

    Broker

    • [fix][broker] ManagedLedger: move to FENCED state in case of BadVersionException #17736
    • [fix][broker] Issue 17588: Allow deletion of a namespace that was left in deleted status #17592
    • [fix][broker] Fix BookKeeper packages npe #17291
    • [fix][broker] Fix rack awareness cache expiration data race #16825
    • [fix][broker] Fix stats-internal with option -m cause active ledger recover then close #16662
    • [fix][broker] Fix deadlock in key-shared dispatcher #16660
    • [fix][broker] PulsarLedgerManager: add missed return statement #16607
    • [fix][broker] Ensure the ack-timeout task gets re-scheduled when there is an exception in the final stage #16337
    • [fix][broker] Fix select broker is none #16316
    • [fix][broker] Fixed deadlock when checking topic ownership #16310
    • [fix][broker] Fix thread safety issues in accessing ManagedCursorContainer.heap ArrayList #16049
    • [fix][broker] Avoid IllegalStateException while client_version is not set #16788
    • [fix][broker] BadVersionException when splitting bundles, delay 100ms and try again. #16612
    • [fix][broker] Expose timestamp field for SchemaData&SchemaInfo #16380
    • [fix][broker] Fix NPE when drop backlog for time limit. #16235
    • [fix][broker] Fix NPE when get /admin/v2/namespaces/public/default/maxTopicsPerNamespace #16076
    • [fix][broker] Fix RawReader out of order #16390
    • [fix][broker] Fix calculate avg message per entry #17046
    • [fix][broker] Fix compaction subscription acknowledge Marker msg issue. #16205
    • [fix][broker] Fix consumer does not abide by the max unacks limitation for Key_Shared subscription #16718
    • [fix][broker] Fix consumer does not abide by the max unacks limitation for Shared subscription #16670
    • [fix][broker] Fix create client with TLS config #16014
    • [fix][broker] Fix namespace backlog quota check with retention. #17706
    • [fix][broker] Fix out of order data replication #17154
    • [fix][broker] Fix passing incorrect authentication data #16201
    • [fix][broker] Fix potential exception causing the reader to be unable to continue reading #17556
    • [fix][broker] Fix pulsarLedgerIdGenerator can't delete index path when zk metadata store config rootPath. #17192
    • [fix][broker] Fix schema does not replicate successfully #17049
    • [fix][broker] Fix the replicator unnecessary get schema request for BYTES schema #17523
    • [fix][broker] Fixed error when delayed messages trackers state grows to >1.5GB #16490
    • [fix][broker] Increment topic stats outbound message counters and update rate after messages have been written to the TCP/IP connection #17043
    • [fix][broker] ManagedCursor: mark delete no callback when create meta-ledger fail #16841
    • [fix][broker] Multiple consumer dispatcher stuck when unackedMessages greater than maxUnackedMessages #17483
    • [fix][broker] PulsarLedgerManager to pass correct error code to BK client #16857
    • [fix][broker] Release the entry in getEarliestMessagePublishTimeOfPos. #16386
    • [fix][broker] Retry to delete the namespace if new topics are created during the namespace deletion #16676
    • [fix][broker] Retry when DistributedIdGenerator has BadVersion error #16491
    • [fix][broker] Skip reading more entries for a pending read with no more entries #16400
    • [fix][broker] Support loadBalancerSheddingIntervalMinutes dynamic configuration #16408
    • [fix][broker] The configuration loadBalancerNamespaceMaximumBundles is invalid #16552
    • [fix][broker] Topic policy reader can't recover when getting any exception. #17562
    • [fix][broker] Unregister topic policy listener if managed ledger close failed #17652
    • [fix][broker] Upgrade log4j2 version to 2.18.0 #16884
    • [fix][broker] fix No such ledger exception #16420
    • [fix][broker] fix broker unackmessages become a negative number #17003
    • [fix][broker] remove exception log when access status.html #17025
    • [fix][broker] Consumer can't consume messages because there are two same topics in one broker #17526
    • [fix][broker] Fix getInternalStats occasional lack of LeaderInfo again #16238
    • [fix][broker] Fix subscribe dispatcher limiter not be initialized #16175
    • [fix][broker] Fix topic policies update not check message expiry #15941
    • [fix][broker] Fix topic-level replicator rate limiter not init #15825
    • [fix][broker] ManagedLedger metrics fail cause of zero period #17257
    • [fix][broker] Prevent StackOverFlowException in SHARED subscription #16968
    • [fix][broker] Fix message ack rate #16108
    • [fix][broker] Fix bug: fail to expose managed ledger client stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true #16219
    • [fix][broker] Fix npe when invoking replaceBookie. #16239
    • [fix][broker] Fail to update partition meta of topic due to ConflictException: subscription already exists for topic #17488
    • [fix][broker] Fix NPE when ledger id not found in OpReadEntry #15837
    • [Fix][broker] Fix topic dispatch rate limiter not init on broker-level #16084
    • [opt][broker] Optimize topic policy with HierarchyTopicPolicies about replicatorDispatchRate #14161
    • [fix][broker] Fix the broker close hanged issue. #17689
    • [fix][broker] Fix topic dispatch rate limiter not init on broker-level #16084 #17000
    • [fix][broker] Prevent StackOverFlowException in SHARED subscription #17053
    • [fix][broker] skip mis-configured resource usage(>100%) in load balancer #16937
    • [fix][broker] Fix wrong unit of NIC speed on Linux #15304
    • [fix][broker] Fix RawReader hasMessageAvailable returns true when no messages #16443
    • [improve][broker] Expose topic level storage write and read rate metrics #16855
    • [improve][broker] Broker/EntryFilter: make the delay for RESCHEDULED messages configurable (dispatcherFilterRescheduledMessageDelay) #16602
    • [improve][broker] Use OrderedExecutor instead of OrderedScheduler for consumer dispatch #16115
    • [improve][broker] Avoid going through all the consumers to get the message ack owner #16245
    • [improve][broker] Avoid reconnection when a partitioned topic was created concurrently #16043
    • [improve][broker] Cancel the loadShedding task when closing pulsar service #17632
    • [improve][broker] Consolidate Netty channel flushes to mitigate syscall overhead #16361
    • [improve][broker] Improve cursor.getNumberOfEntries if isUnackedRangesOpenCacheSetEnabled=true #17465
    • [improve][broker] Make MessageRedeliveryController work more efficiently #17804
    • [improve][broker] Make PulsarWebResource#getOwnerFromPeerClusterList async. #15940
    • [improve][broker] Optimise msgOutCounter and bytesOutCounter (#16214) #16286
    • [improve][broker] Recycle OpReadEntry in some corner cases #16399
    • [improve][broker] Reduce the consumers list sort by priority level #16243
    • [improve][broker] Reduce the re-schedule message read operation for PersistentDispatcherMultipleConsumers #16241
    • [improve][broker] Use LinkedHashSet for config items of type Set to preserve elements order #16138
    • [improve][broker] Use shared broker client scheduled executor provider #16338
    • [improve][broker] Remove unnecessary lock on the stats thread #16983
    • [improve][broker] Tidy update subscriptions dispatcher rate-limiter #16778
    • [improve][broker] PIP-149: Make resetCursor async (#16355) #16774
    • [improve][broker] added loadBalancerReportUpdateMinIntervalMillis and ignores memory usage in getMaxResourceUsage() #17598
    • [cleanup][broker] Follow up on #16968 to restore some behavior in PersistentDispatcherMultipleConsumers class #17018

    Metadata & Storage

    • [fix][metadata] Fix LockTimeout when storePut on the same key concurrently in RocksdbMetadataStore #16005
    • [fix][metadata] Cleanup state when lock revalidation gets LockBusyException #17700
    • [fix][metadata] Don't execute Bookkeeper metadata callbacks on Zookeeper event thread #17620
    • [fix][metadata] Set revalidateAfterReconnection true for certain failures #17664
    • [improve][metadata] Update namespace policies would cause metadata-store thread waiting too long #16438
    • [fix][storage] Fix offload read handle NPE. #17478
    • [fix][storage] Fix OpAddEntry release error when exception in ManagedLedgerInterceptor #17394
    • [fix][tableview] Fixed ack failure in ReaderImpl due to null messageId #17828

    Transaction

    • [fix][txn] Cmd-Subscribe and Cmd-Producer will not succeed even after 100 retries #16248
    • [fix][txn] Set TC state is Ready after open MLTransactionMetadataStore completely. #13957
    • [fix][txn] Ack the same batch message different batchIndex with transaction #16032
    • [fix][txn] Allow producer to enable send timeout in transaction #16519
    • [fix][txn] Fix NPE when ack message with transaction at cnx = null #16142
    • [fix][txn] Fix TopicTransactionBuffer ledger apend marker throw ManagedLedgerAlreadyClosedException #16265
    • [fix][txn] Fix append txn message is lower than lowWaterMark decrease PendingWriteOps #16266
    • [fix][txn] Fix ack with txn compute ackedCount error #17016
    • [fix][txn] Fix pattern sub filter transaction system topic #16533
    • [fix][txn] Fix race in TransactionMetaStoreHandler #16147

    Pulsar IO and Pulsar Functions

    • [fix][connector] Fix Kafka source config for pulsar-io-kafka when consumerConfigProperties="" #16731
    • [fix][connector] IOConfigUtils support required and defaultValue annotations. #16785
    • [improve][connector] Add reader config to pulsar-io-debezium and pulsar-io-kafka-connect-adaptor #16675
    • [fix][function] Ensure bytes is a well-formed UTF-8 byte sequence when decode the FunctionState bytes to string #16199
    • [fix][function] Fix python instance not process zip file correctly #16697
    • [fix][function] Fixed error when user starts with the pulsar functions local runner #16565
    • [fix][function] Ensure InternalConfigurationData data model is compatible across different versions #17690

    Tiered Storage

    • [fix][tiered-storage] Don't cleanup data when offload met Metastore exception #17512
    • [fix][tiered-storage] Fix the wrong secret key name get from env #15814
    • [fix][tiered-storage] move the state check forward #17020
    • [fix][common] Fix setManagedLedgerOffloadedReadPriority not work. #16436

    Pulsar SQL

    • [fix][sql] Fix messageQueue release message issue. #16155
    • [fix][sql] Fix Presto SQL Avro decode error when publishing non-batched msgs #17093

    Pulsar Proxy

    • [fix][proxy] Do not preserve host when forwarding admin requests. #16342
    • [fix][proxy] Fix client service URL #16834
    • [improve][proxy] Update proxy lookup throw exception type #17600

    CLI/Admin

    • [fix][admin] Fix reach max tenants error if the tenant already exists #15932
    • [fix][admin] Fix get non-persistent topics issue in Namespaces. #16514
    • [fix][admin] Add SNI header when tlsHostnameVerification is not enabled #17543
    • [fix][admin] Fix missing response type in swagger definitions #16022
    • [fix][admin] Fix typo in validation message #16021
    • [fix][admin] Fix unWrap Exception when getPoliciesAsync #17249
    • [cleanup][admin] Update/fix Swagger Annotation for param: authoritative #16222

    Security

    • [fix][sec] Avoid AuthenticationDataSource mutation for subscription name #16065
    • [fix][sec] Upgrade Netty Reactive Streams to 2.0.6 #15990
    • [fix][sec] Bump dependency check and spring version to avoid potential FP #15408
    • [fix][sec] Bump snakeyaml to 1.32 for CVE-2022-38752 #17779
    • [fix][sec] Bump snakeyaml to 1.31 fix CVE-2022-25857 #17457
    • [fix][sec] Bump PostgreSQL version to 42.4.1 #17066
    • [fix][sec] Upgrade reload4j in file-system offloader #17716
    • [fix][sec] Upgrade to Jetty to 9.4.48.v20220622 to get rid of CVE-2022-2047 #16520
    • [improve][sec] Suppress CVE-2021-3563 of openstack-keystone-2.5.0 #17458
    • [improve][sec] Add load multiple certificates support #14798
    • [improve][sec] Upgrade aws-java-sdk-s3 to 1.12.261 #16684
    • [fix][authorization] Fix multiple roles authorization #16645
    • [improve][authentication] Adapt basic authentication configuration with prefix #16935
    • [improve][authentication] Improve get the basic authentication config #16526
    • [improve][authentication] Support for get token from HTTP params #16986

    Java Client

    • [fix][java] Fix PatternTopicsChangedListener blocked when topic removed #16842
    • [fix][java] Forget to update memory usage when invalid message #16835
    • [fix][java] Send CloseConsumer on client timeout #16616
    • [fix][java] Fix ReconsumeLater will hang up if retryLetterProducer exception #16655
    • [fix][java] Fix load trust certificate #16789
    • [fix][java] Fix reach redeliverCount client can't send batch messags #17317
    • [fix][java] Fix reach redeliverCount client can't send messages to DLQ #17287
    • [fix][java] Fix the message present in incoming queue after go to DLQ #17326
    • [fix][java] Fix the startMessageId can't be respected as the ChunkMessageID #16154
    • [fix][java] Release semaphore before discarding messages in batchMessageContainer #17019
    • [fix][java] Remove consumer when close consumer command is received #15761
    • [fix][java] Remove producer when close producer command is received #16028
    • [fix][java] Remove redundant check for chunked message TotalChunkMsgSize in ConsumerImpl #16797
    • [fix][java] Fix MaxQueueSize semaphore release leak in createOpSendMsg #16915
    • [fix][java] Fix auto cluster failover can't resolve host bug #16152
    • [fix][java] Fix client memory limit currentUsage leak and semaphore release duplicated in ProducerImpl #16837
    • [fix][java] Fix newLookup TooManyRequestsException message #16594
    • [fix][java] Fix scheduledExecutorProvider not shutdown #17527
    • [fix][java] Fixed cnx channel Inactive causing the request fail to time out and fail to return #17051
    • [fix][java] Fix thread safety issue of LastCumulativeAck #16072
    • [fix][java] Make DeadLetterPolicy deserializable #16513
    • [improve][java] Improve performance of multi-topic consumer with more than one IO thread #16336
    • [improve][java] Only trigger the batch receive timeout when having pending batch receives requests #16160
    • [improve][java] Replace ScheduledExecutor to improve performance of message consumption #16236
    • [improve][java] Support passing existing scheduled executor providers to the client #16334
    • [improve][java] Add classLoader field for SchemaDefinition #15915
    • [improve][java] Add message key if exists to deadLetter messages #16615
    • [improve][java] Refactor SchemaHash to reduce call of hashFunction in SchemaHash #17948

    C++ Client

    • [fix][c++] Rename function name: pulsar_producer_configuration_set_crypto_failure_action #16031
    • [fix][c++] Fix the close of Client might stuck or return a wrong result #16285
    • [fix][c++] Fix libcurl build failure when building deb package #17614
    • [fix][c++] Fix multi-topics consumer close segmentation fault #17239
    • [fix][c++] Fix potential segfault when resending messages #17395
    • [improve][c++] Reset havePendingPingRequest flag for any data received from broker #17658
    • [improve][c++] Use an atomic state_ instead of the lock to improve performance #16940
    • [improve][c++] Upgrade OpenSSL to version 1.1.1n #17538
    • [improve][c++] Delete PartitionedConsumerImpl, use MultiTopicsConsumerImpl instead #16969
    • [cleanup][c++] Clean up C++ client curl configuration #16064

    Dependency & Library updates

    • [fix][dependency] Exclude the Netty Reactive Stream from asynchttpclient #16312
    • [fix][dependency] Removing log4j-1.2-api from dependencies #15991
    • [cleanup][dependency] Remove redundant pulsar-zookeeper-utils module #16258

    CI & Test

    • [fix][ci] Remove post-commit trigger for old release branches (2.10 and previous) #17570
    • [fix][ci] Fix some OWASP dependency problems. #16260
    • [fix][test] Fix flaky-test RackAwareTest.testRackUpdate #16071
    • [fix][test] Catch exception when updating data in mockZookeeper #16473
    • [fix][test] Fix jvm oom on Unit Test broker group 1 #16542
    • [fix][test] Fix ManagedLedgerTest#avoidUseSameOpAddEntryBetweenDifferentLedger #16720
    • [fix][test] Fix test TransactionEndToEndTest#testSendTxnMessageTimeout (only release branches) #16570
    • [fix][test] Fix Flaky-test: PartitionedProducerConsumerTest.testPartitionedTopicInterceptor #16809
    • [fix][test] TieredStorageConfigurationTests - clear system properties #15957
    • [fix][test] Fix Flaky-test: ManagedLedgerBkTest.asyncMarkDeleteAndClose. #16730
    • [fix][test] Fix flaky C++ ClientTest.testWrongListener #16510
    • [improve][test] Reduce the time consumption of BacklogQuotaManagerTest #16550
    • [improve][test] Verify the authentication data in the authorization provider #16900
    • [improve][test] Improved flaky test runs #16011
    • [fix][flaky-test] BrokerInterceptorTest.testProducerCreation #16742
    • [fix][flaky-test] ElasticSearchClientTests.testBulkBlocking #16920
    • [fix][flaky-test] Fix BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuotaWithEmptyLedge #16735
    • [fix][flaky-test] Fix ClassCastException: BrokerService cannot be cast to class PulsarResources #16821
    • [fix][flaky-test] Fix DefaultMessageFormatter.formatMessage #17104
    • [fix][flaky-test] Fix PersistentTopicStreamingDispatcherTest #16801
    • [fix][flaky-test] Fix PulsarFunctionTlsTest.tearDown #16765
    • [fix][flaky-test] Fix failed test NonPersistentTopicE2ETest.testGCWillDeleteSchema #16381
    • [fix][flaky-test] Fix failed test PatternTopicsConsumerImplTest.testAutoSubscribePatternConsumer #16375
    • [fix][flaky-test] Fix flaky test testBacklogNoDelayedForPartitionedTopic #17180
    • [fix][flaky-test] Fix flaky test testConsumerBacklogEvictionTimeQuota… #16419
    • [fix][flaky-test] MessageTTLTest.testMessageExpiryAfterTopicUnload #16462
    • [fix][flaky-test] NonPersistentTopicE2ETest.testGC #16505
    • [fix][flaky-test] PersistentFailoverE2ETest.testSimpleConsumerEventsWithPartition #16493
    • [fix][flaky-test] Try to fix flaky test related to PersistentTopicTest.setup #16383
    • [fix][flaky-test] Fix testSplitBundleForMultiTimes #16562
    • [fix][flaky-test] PulsarFunctionLocalRunTest.testE2EPulsarFunctionLocalRunMultipleInstances #16872
    • [fix][flaky-test] AdminApi2Test.testDeleteNamespace #17157
    • [fix][flaky-test] BrokerInterceptorTest.testProducerCreation #17159
    • [fix][flaky-test] ConsumedLedgersTrimTest #17116
    • [fix][flaky-test] ManagedCursorMetricsTest.testCursorReadWriteMetrics #17045

    Others

    • [fix][build] Duplicate entry when merging services #17659
    • [cleanup][owasp] Suppress false positive netty-tcnative #17282
    • [fix][doc] Fix comments for exposeManagedLedgerMetricsInPrometheus field #17792
    • [fix][tool] Using int instead of long in python scripts #17215
    • [improve][package] Improve the package download process #16365
    Source code(tar.gz)
    Source code(zip)

  • v2.8.4(Oct 4, 2022)

    Broker

    • [improve][broker] Do not print the log that the topic does not exist #13535
    • [fix][broker] Fix NPE when nextValidLedger is null #13975
    • [fix][broker] Fix NPE when ReplicatedSubscriptionsController sends marker message with deduplication enabled #14017
    • [fix][broker] Fix avg-messagePerEntry metrics for consumer #14330
    • [improve][broker] Validate rack name when setting the bookie rack info #14336
    • [improve][broker] Optimize find nics process. #14340
    • [fix][broker] Fix NPE when subscription is already removed #14363
    • [fix][broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions #14367
    • [improve][broker] Optimize RawReader#create when using Compactor #14447
    • [fix][broker] Fix the latest message of the compacted topic cannot be read #14449
    • [fix][broker] Fix producerFuture not completed in ServerCnx#handleProducer #14467
    • [fix][broker] Fix NPW in ManagedLedgerImpl #14481
    • [improve][broker] Support shrink in ConcurrentLongHashMap #14497
    • [fix][broker] Fixed wrong behaviour caused by not cleaning up topic policy service state. #14503
    • [improve][broker] Optimize memory usage: support to shrink for pendingAcks map #14515
    • [fix][broker] Cancel offload tasks when managed ledger closed #14744
    • [improve][broker] Reduce unnecessary expansions for ConcurrentLong map and set #14562
    • [fix][broker] Fix metadata store deadlock when checking BacklogQuota #14634
    • [improve][broker] Support shrink for map or set #14663
    • [fix][broker] Fix lost message issue due to ledger rollover #14664
    • [fix][broker] Set splitNamespaceBundle with readonly=false #14680
    • [fix][broker] Fixed duplicated delayed messages when all consumers disconnect #14740
    • [fix][broker] Fix wrong state for non-durable cursor #14869
    • [fix][broker] Improve error logging for topic not found #14892
    • [fix][broker] Fix topic policy reader close bug #14897
    • [fix][broker] Return if reset in progress #14978
    • [fix][broker] Fix potential NPE in Replicator #15003
    • [fix][broker] Fix race condition in updating lastMarkDeleteEntry field #15031
    • [fix][broker] Fix potential to add duplicated consumer #16826
    • [fix][broker] Follow up on race condition fixes in ManagedCursorImpl #15031 #15067
    • [cleanup] [broker] Remove useless code to avoid confusion in OpReadEntry#checkReadCompletion #15104
    • [fix][broker] Cancel fencedTopicMonitoringTask when topic closed normally. #15202
    • [fix][broker] Fix MessageDeduplication#inactiveProducers may not be persistence correctly #15206
    • [fix][broker] Fix race condition between timeout and completion in OpAddEntry #15233
    • [fix][broker] Fix problem at RateLimiter#tryAcquire #15306
    • [improve][broker] Use shrink map for message redelivery. #15342
    • [improve][broker] Support shrink for ConcurrentSortedLongPairSet #15354
    • [improve][broker] Close connection if a ping or pong message cannot be sent #15382
    • [cleanup][broker] Fix typo in enum name and handle closing of the channel properly since writeAndFlush is asynchronous #15384
    • [fix][broker] Avoid heartbeat topic to offload. #15393
    • [fix][broker] Fix deadlock in broker after race condition in topic creation failure #15570
    • [fix][broker] fix calculation in getNumberOfEntriesInStorage #15627
    • [fix][broker] Fix NPE when set AutoTopicCreationOverride #15653
    • [fix][broker] Fix creating producer failure when set backlog quota #15663
    • [fix][broker] Fix NPE when put value to RangeCache #15707
    • [fix][broker] Fix can not enable system topic if AutoUpdateSchemaEnabled=false #15759
    • [fix][broker] Fix wrong unit of NIC speed on Linux #15770
    • [fix][broker] Fix NPE in MessageDeduplication #15820
    • [fix][broker] Fix NPE when ledger id not found in OpReadEntry #15837
    • [fix][broker] When skipping updating mark delete position, execute callback with executor to prevent deadlock #15971
    • [fix][broker] Fix NPE when drop backlog for time limit. #16235
    • [fix][broker]Fix getInternalStats occasional lack of LeaderInfo again #16238
    • [fix][broker] Fix RawReader out of order #16390
    • [fix][broker] Skip reading more entries for a pending read with no more entries #16400
    • [fix][broker] Fix No such ledger exception #16420
    • [fix][broker] Fix RawReader hasMessageAvailable returns true when no messages #16443
    • [fix][broker] Fix stats-internal with option -m cause active ledger recover then close #16662
    • [fix][broker] Fix passing incorrect authentication data #16840
    • [fix][broker] ManagedCursor: mark delete no callback when create meta-ledger fail #16841
    • [improve][broker] Avoid reconnection when a partitioned topic was created concurrently #16867
    • [improve][broker] Recycle OpReadEntry in some corner cases #16869
    • [fix][broker] Fix compaction subscription acknowledge Marker msg issue. #16918

    Security

    • [fix][sec] Use provider's canLookupAsync method for AuthorizationService #11777
    • [improve][sec] Optimize the logic of allowing namespace operation #13090
    • [improve][sec] Full-support set SSL provider, ciphers and protocols #13740
    • [fix][sec] Role with namespace produce authz can also get topics #13773
    • [improve][sec] Full-support SSL provider, ciphers, and protocols for broker service and proxy service #15034
    • [improve][sec] Add load multiple certificates support #14798
    • [fix][sec] Fix handling single role and non-jwt-token in MultiRolesTokenAuthorizationProvider #14857
    • [fix][sec] Use tlsCertRefreshCheckDurationSec instead of 0 for refresh value #15075
    • [improve][sec] Skip unnecessary DNS resolution when creating AuthenticationDataHttp instance #15221
    • [improve][sec] Improve skipping of DNS resolution when creating AuthenticationDataHttp instance #15228
    • [fix][sec] Fix MultiRolesTokenAuthorizationProvider authorize issue #15454
    • [fix][sec] Fix grant all permissions but can't list topic #15501
    • [improve][sec] Switch to rely on Netty for Hostname Verification #15824
    • [fix][sec] Fix create client with TLS config #16014
    • [fix][sec] Avoid AuthenticationDataSource mutation for subscription name #16065
    • [improve][sec] Improve get the basic authentication config #16526
    • [fix][sec] Fix multiple roles authorization #16645
    • [fix][sec] Add timeout of sync methods and avoid call sync method for AuthoriationService (#15694) #16831
    • [improve][sec] Support for get token from HTTP params #16871
    • [improve][sec] Adapt basic authentication configuration with prefix #16935

    Admin

    • [improve][admin] Add corresponding get command for namespace #12322
    • [fix][admin] Fix deleting namespace will fail when system topics exist #15040
    • [fix][admin] Fix validateGlobalNamespaceOwnership wrap exception issue #14612
    • [fix][admin] Fix wrong prompt exception when getting the non-persistent topic list without GET_BUDNLE permission #14638
    • [fix][admin] Fixed 404 error msg not returned correctly using http lookup #14677
    • [improve][admin] Provide an accurate error message when set autoTopicCreation #14684
    • [fix][admin] Fix cannot delete namespace with system topic #14730
    • [fix][admin] Fix NPE in PulsarAdminBuilder when the service is not set #14769
    • [fix][admin] Fix missing response type in swagger definitions #16022

    Function

    • [fix][fn] Fixing get functions for output topic and serde classname #14103
    • [improve][fn] Pass configured metricsPort to k8s runtime #14502
    • [fix][fn] fix some IOExceptions when create functions from package URL #14553
    • [fix][fn] handle NPE when getLeader returns null #15058
    • [improve][fn] Add KeyStore support in WebSocket, Function Worker HTTPS Servers #15084
    • [fix][function] Check executor null when close the FileSource #15247
    • [improve][fn] Provide default error handler for function log appender #15728
    • [fix][function] Fix error when user starts with the pulsar functions local runner #16565
    • [fix][function] Fix python instance not process zip file correctly #16697

    IO

    • [fix][io] throw exceptions when Kafka offset backing store failed to start #14491
    • [fix][io] Handle Kafka sinks that return immutable maps as configs #14780

    SQL

    • [fix][sql] Fix PulsarRecordCursor deserialize issue. #14615
    • [fix][sql] Fix messageQueue release message issue. #16155

    Offloader

    • [fix][offloader] Fix incorrect entryId in warn log when reading entry from tiered storage #14685
    • [improve] TieredStorage: add debug information #14907
    • [feat][offloader] Add pure S3 provider for the offloader #15710
    • [fix][offloader] Fix setManagedLedgerOffloadedReadPriority not work. #16436

    Proxy

    • [improve][proxy] Log warning when opening connection to broker fails #14710
    • [refactor][proxy] Refactor Proxy code and fix connection stalling by switching to auto read mode #14713
    • [improve][proxy] Configure Netty DNS resolver to match JDK DNS caching setting, share DNS resolver instance in Proxy #15219
    • [fix][proxy] Fix proxy connection leak when inbound connection closes while connecting is in progress #15366
    • [fix][proxy] Fix DNS server denial-of-service issue when DNS entry expires #15403
    • [improve][proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler #15415
    • [fix][proxy] Do not preserve host when forwarding admin requests. #16342
    • [fix][proxy] Fix client service URL #16834

    Monitor

    • [improve][monitor]: add metrics for pulsar web service thread pool #14742
    • [improve][monitor] add message ack rate metric for consumer #15674
    • [fix][monitor] Fix bug: fail to expose managed ledger client stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true #16343
    • [improve][monitor] Expose topic level storage write and read rate metrics #16855

    CLI

    • [fix][cli] Remove the trust certs check #14764

    Test & Other

    • [improve][test] Use Awaitility to replace Thread.sleep for pulsar-zookeeper-utils module. #11017
    • [fix][test] Fix flaky test in PersistentStreamingDispatcherBlockConsumerTest #12943
    • [fix][build] Fix NoClassDefFoundError: com/google/inject/AbstractModule in pulsar-io/batch-data-generator and Jcloud offloader #14150
    • [fix][test] Fix flaky AdminApiSchemaTest#testSchemaInfoApi #14508
    • [fix][test] Fix NamespacesTest execution order #14552
    • [fix][test] Fix AdminApiTest.testNamespaceSplitBundleConcurrent #14565
    • [fix][test] Fix flaky test MemoryLimitTest#testRejectMessages (#14220) #14628
    • [fix][ci] Upgrade zlib version to 1.2.12 #14964
    • [improve][build] Ignore case when obfuscating passwords in configuration scripts #15077
    • [fix][build] Fix LICENSE files for branch-2.8 #15261
    • [Improve][doc] Add config of IO and acceptor threads in proxy #15340
    • [improve][build] Configure DLog Bookie, Pulsar, and Admin clients via pass-through config #15818
    • [improve][build] Allow pulsar_tool_env.sh PULSAR_MEM to be Overridden #15868
    • [fix][test] TieredStorageConfigurationTests - clear system properties #15957
    • [fix][doc] Update/fix Swagger Annotation for param: authoritative #16222
    • [improve][test] Verify the authentication data in the authorization provider #16900

    Dependency Updates

    • [improve][build] Upgrade snakeyaml version to 1.30 #13722
    • [fix][build] Remove --illegal-access errors resulting from Google Guice (upgrade to 5.0.1 and JClouds to 2.4.0) #13810
    • [fix][build] Remove --illegal-access errors resulting from Google Guice - Pulsar IO, Offloaders and Pulsar SQL - Bump Guice to 5.1.0 #14300
    • [fix][build] Upgrade jackson and jackson-databind (2.13.2.1) to get rid of CVE-2020-36518 #14871
    • [improve][build] Remove log4j for CVE-2022-23307 #15109
    • [improve][build] Upgrade Netty to 4.1.76.Final, Netty Tcnative, grpc and protobuf #15212
    • [improve][build] Use grpc-bom to align grpc library versions #15234
    • [improve][build] Upgrade BookKeeper to 4.14.5 #15581
    • [improve][build] Upgrade Netty to 4.1.77.Final and netty-tcnative to 2.0.52.Final #15646
    • [improve][build] Upgrade JClouds to 2.5.0 [#15649](https://github.com/apache/pulsar/pull/15649
    • [fix][build] Tiered storage: Upgrade Hadoop to 3.3.3 to get rid of CVE-2022-26612 #15660
    • [fix][build] Upgrade to Jetty to 9.4.48.v20220622 to get rid of CVE-2022-2047 #16520
    • [improve][broker] Upgrade log4j2 version to 2.18.0 (#16884) #16914

    Java Client

    • [improve][java] Improve consumer listener logic #13273
    • [fix][java] Fix ConsumerBuilderImpl#subscribeAsync blocks calling thread #14614
    • [fix][java] Fix partitionsAutoUpdateFuture never completes #14625
    • [fix][java] Fix race condition in consumer redelivery #14687
    • [improve][java] Process maxRedeliverCount is 0 of DeadLetterPolicy #14706
    • [improve][java] Avoid timer task run before previous subscribe complete #14818
    • [fix][java] Fix potentially unfinished CompletableFuture in doReconsumeLater #14947
    • [fix][java] ConsumerBuilderImpl can not set null to deadLetterPolicy. #14980
    • [improve][java] Add test to ensure the message order in listener callbacks #15049
    • [fix][java] Fix performance regression with message listener #15162
    • [fix][java] Fix wrong behavior of deduplication for key based batching #15413
    • [improve][java] improve logic when ACK grouping tracker checks duplicated message id #15465
    • [fix][java] Remove consumer when close consumer command is received #15761
    • [fix][java] Fix conversion of TimestampMillisConversion has no effect when Jsr310Conversion is enabled #15863
    • [fix][java] Remove producer when close producer command is received #16028
    • [fix][java] Fix thread safety issue of LastCumulativeAck #16072
    • [improve][java] Send CloseConsumer on client timeout #16616
    • [fix][java] Fix ReconsumeLater will hang up if retryLetterProducer exception #16655
    • [fix][java] Fix load trust certificate #16789
    • [fix][java] Remove redundant check for chunked message TotalChunkMsgSize in ConsumerImpl #16797
    • [fix][java] Forget to update memory usage when message is invalid #16835
    • [fix][java] Fix PatternTopicsChangedListener blocked when topic removed #16842
    • [fix][java] Fix MaxQueueSize semaphore release leak in createOpSendMsg #16915

    C++ Client

    • [improve][cpp] Expose getLastMessageId in the Reader API #11723
    • [fix][cpp] Fix hasMessageAvailable returns wrong value for last message #13883
    • [fix][cpp] Fix thread safety issue for multi topic consumer #14380
    • [fix][cpp] Fix wrong unit of Access Token Response's expires_in field #14554
    • [fix][cpp] Handle exception in creating socket when fd limit is reached #14587
    • [cleanup][cpp] Add braces around initialization of subobject #14735
    • [fix][cpp] Fix producer is never destructed until client is closed #14797
    • [fix][cpp] Fix flaky tests about reference count #14854
    • [fix][cpp] Fix segmentation fault when creating socket failed #14834
    • [fix][cpp] Fix the race condition of connect timeout task #14823
    • [fix][cpp] Fix send callback might not be invoked in key-based batching #14898
    • [fix][cpp] Fix connection is not closed when broker closes the connection to proxy #15009
    • [fix][cpp] Fix single message metadata not set correctly #15072
    • [fix][cpp] Fix UnknownError might be returned for a partitioned producer #15161
    • [improve][cpp] Remove the flaky and meaningless tests #15271
    • [improve][cpp] Wait until event loop terminates when closing the Client #15316
    • [fix][cpp] Avoid race condition causing double callback on close #15508
    • [fix][cpp] Generate correct well-known OpenID configuration URL #15928
    • [fix][cpp] Rename function name: pulsar_producer_configuration_set_crypto_failure_action #16031
    • [cleanup][cpp] Clean up C++ client curl configuration #16064
    • [fix][cpp] Fix the close of Client might stuck or return a wrong result #16285
    • [improve][cpp] Fix flaky C++ ClientTest.testWrongListener #16510

    Python Client

    • [fix][python] Fix generated Python protobuf code not compatible with latest protobuf package #15846

    WebSocket Client

    • [fix][ws] Fix MultiTopicReader#getConsumer ClassCastException 15534
    Source code(tar.gz)
    Source code(zip)

  • v2.7.5(Sep 6, 2022)

    Broker

    • [fix][broker] Fix NPE when ledger id not found in OpReadEntry#16966
    • [fix][broker] Validate rack name #16850
    • [fix][broker] Fix invalid rack name cause bookie join rack failed #16845
    • [fix][broker] Avoid potentially blocking calls to metadata in PersistentTopic#getMessageTTL #16899
    • [fix][broker] Fix create client with TLS config #16326
    • [fix][broker] Fixed deadlock on metadata cache missing while doing checkReplication #16889
    • [fix][broker] Fix the wrong unit of NIC speed on Linux #15770
    • [fix][broker] Close connection if a ping or pong message cannot be sent #15382
    • [fix][broker] Fix MessageDeduplication#inactiveProducers may not be persistence correctly #15206
    • [fix][broker] Fix messageDedup delete inactive producer name #12493
    • [fix][broker] Fix the reader skips compacted data which original ledger been removed. #16407
    • [fix][broker] Fix getting the last message-id from an empty compact ledger. #16402
    • [fix][broker] Return message ID from compacted ledger while the compaction cursor reaches the end of the topic. #16395
    • [fix][broker] Fix skips compacted data for reader/consumer #16301
    • [fix][broker] Fix reader skipped remaining compacted data during the topic unloading #16300
    • [fix][broker] Fix passing incorrect authentication data #16201
    • [fix][broker] Fix incorrect returned last message ID while the lastConfirmedEntry with negative entry ID #16299
    • [fix][broker] Fix can not enable system topic if AutoUpdateSchemaEnabled=false. #15754
    • [fix][broker] Fix lost message issue due to ledger rollover. #14703
    • [fix][broker] Fix lost message issues 12221 #12223
    • [fix][broker] Fix TopicPoliciesCacheNotInitException issue. #14293
    • [fix][broker] Fix deadlock when use key_shared mode #11965
    • [fix][broker] Fix log content error when OverloadShedder is not used. #13540
    • [improve][broker] Skip unnecessary DNS resolution when creating AuthenticationDataHttp instance #15221
    • [improve][broker] Improve error logging for topic not found [#13950) #14893
    • [improve][broker] Trim configuration value string which contains blank prefix or suffix string #13984
    • [improve][broker] Configure DLog Bookie, Pulsar, and Admin clients via pass-through config #15818
    • [cleanup][broker] Improve skipping of DNS resolution when creating AuthenticationDataHttp instance #15228
    • [cleanup][broker] Remove useless code to avoid confusion in OpReadEntry#checkReadCompletion. #15104
    • [cleanup][broker] Clean up individually deleted messages before the mark-delete position #14261

    Security

    • [improve][sec] Improve get the basic authentication config #16947
    • [improve][sec] Support for get token from HTTP params #16892
    • [fix][sec] AuthorizationService should use provider's canLookupAsync method #11777
    • [fix][sec] Avoid AuthenticationDataSource mutation for subscription name #16065
    • [fix][sec] Return if namespace policies are read only #12514
    • [refactor][sec] Switch to rely on Netty for Hostname Verification #15824
    • [cleanup][sec] Ignore case when obfuscating passwords in configuration scripts #15077

    Storage

    • [fix][storage] ManagedCursor: mark delete no callback when create meta-ledger fail #16841
    • [fix][storage] Cancel offload tasks when managed ledger closed #16808
    • [fix][storage] Fix lost compaction data due to compaction properties missed during reset-cursor. #16404
    • [fix][storage] Do not move the non-durable cursor position when trimming ledgers while topic with compaction. #16403
    • [fix][storage] Fix issues in advanceNonDurableCursors #10667
    • [fix][storage] Follow up on race condition fixes in ManagedCursorImpl #15031 #15067
    • [fix][storage] Fix race condition in updating lastMarkDeleteEntry field #15031
    • [fix][storage] Fix NPE when removing cursor #12297
    • [improve][storage] If mark-delete operation fails, mark the cursor as "dirty" #14256

    Proxy

    • [fix][proxy] Fix client service url #16894
    • [fix][proxy] Prevent leak of unreleased lookupRequestSemaphore permits #13812
    • [fix][proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler #15415
    • [fix][proxy] Fix proxy connection leak when inbound connection closes while connecting is in progress #15366
    • [fix][proxy] Fix port exhaustion and connection issues in Pulsar Proxy #14078
    • [fix][proxy] Fix DNS server denial-of-service issue when DNS entry expires #15403
    • [fix][proxy] Configure Netty DNS resolver to match JDK DNS caching setting, share DNS resolver instance in Proxy #15219
    • [refactor][proxy] Refactor Proxy code and fix connection stalling by switching to auto read mode #14713
    • [improve][proxy] Fail proxy startup if brokerServiceURL is missing scheme #14682
    • [improve][proxy] Remove unnecessary Pulsar Client usage from Pulsar Proxy #13836

    Admin

    • [fix][admin] Fix validateGlobalNamespaceOwnership wrap exception issue. #14269
    • [cleanup][admin] Update/fix Swagger Annotation for param: authoritative #16222

    Function

    • [fix][fn] Fix python instance not process zip file correctly #16697

    Java Client

    • [fix][java] Fix semaphore release duplicated in ProducerImpl #16972
    • [fix][java] Fix MaxQueueSize semaphore release leak in createOpSendMsg #16915
    • [fix][java] Fix PatternTopicsChangedListener blocked when topic removed #16842
    • [fix][java] Fix ReconsumeLater will hang up if retryLetterProducer exception #16655
    • [fix][java] Fix DNS server denial-of-service issue when DNS entry expires #15403
    • [fix][java] Configure Netty DNS resolver to match JDK DNS caching setting, share DNS resolver instance in Proxy #15219
    • [refactor][java] Switch to rely on Netty for Hostname Verification #15824
    • [improve][java] Remove unnecessary Pulsar Client usage from Pulsar Proxy #13836

    C++ Client

    • [cleanup][cpp] Clean up C++ client curl configuration #16064
    • [cleanup][cpp] Add braces around initialization of subobject #14735

    Python Client

    • [fix][python] Fix generated Python protobuf code not compatible with the latest protobuf package #15846

    Test & Others

    • [improve][test] Verify the authentication data in the authorization provider #16945
    • [improve][test] Exclude inner classes in surefire #9875
    • [fix][test] Enable ignored tests #16435
    • [fix][test] Fix setting Bookie dbStorage_*CacheMaxSizeMb in pulsar-test-latest-version docker image #9623
    • [improve][doc] Add more configuration methods for basic authentication #16941
    • [fix][build] Use grpc-bom to align grpc library versions #15234
    • [improve][ci] Add set up Java #16430
    • Update notice year. #13653

    Dependency Updates

    • Upgrade protobuf to 3.16.1 to address CVE-2021-22569 #13695
    • Upgrade aircompressor to 0.20 #11790
    • Upgrade the BookKeeper version to 4.12.1 #16775
    • Upgrade Gson version 2.8.6 to 2.8.9 #13610
    • Upgrade Log4j2 to 2.17.1 #13552
    • Upgrade log4j2 version to 2.18.0 #16884
    Source code(tar.gz)
    Source code(zip)

  • v2.9.3(Jul 20, 2022)

    Important notice

    • [PIP-146] ManagedCursorInfo compression #14542
    • [PIP-153] Optimize metadataPositions in MLPendingAckStore #15137
    • [PIP-163] Add lowWaterMark check before appending entry to TB #15424

    Broker

    • [cleanup][broker] Cleanup already deleted namespace topics #12597
    • [cleanup][broker] Override close method to avoid caching exception #15529
    • [cleanup][broker] Remove useless code to avoid confusion in OpReadEntry#checkReadCompletion #15104
    • [fix][broker] Avoid heartbeat topic to offload #15008
    • [fix][broker] Cancel fencedTopicMonitoringTask when topic closed normally #15202
    • [fix][broker] Check for blank advertised listener name #14306
    • [fix][broker] Close publishLimiter when disable it #15520
    • [fix][broker] Fast return if ack cumulative illegal #15695
    • [fix][broker] Fix MessageDeduplication#inactiveProducers may not be persistence correctly #15206
    • [fix][broker] Fix MultiRolesTokenAuthorizationProvider authorize issue #15454
    • [fix][broker] Fix NPE in MessageDeduplication #15820
    • [fix][broker] Fix NPE when ledger id not found in OpReadEntry #15837
    • [fix][broker] Fix NPE when put value to RangeCache #15707
    • [fix][broker] Fix NPE when set AutoTopicCreationOverride #15653
    • [fix][broker] Fix NPE when subscription is already removed #14363
    • [fix][broker] Fix REST produce msg redirect issue #15551
    • [fix][broker] Fix call sync method in onPoliciesUpdate method #13885
    • [fix][broker] Fix call sync method in onPoliciesUpdate method #15227
    • [fix][broker] Fix cannot delete namespace with system topic #14730
    • [fix][broker] Fix creating producer failure when set backlog quota #15663
    • [fix][broker] Fix creating system namespace topic failure #14949
    • [fix][broker] Fix deadlock in broker after race condition in topic creation failure #15570
    • [fix][broker] Fix getPendingAckInternalStats redirect issue #14876
    • [fix][broker] Fix inconsistent prompt message when schema version is empty using AVRO #14626
    • [fix][broker] Fix incorrect entryId in warning logs when reading an entry from tiered storage #14685
    • [fix][broker] Fix metadata store deadlock when checking BacklogQuota #14634
    • [fix][broker] Fix no value present #14891
    • [fix][broker] Fix normal topic named ends with healthcheck becomes system topic issue #14671
    • [fix][broker] Fix parameter saslJaasBrokerSectionName in broker.conf #15110
    • [fix][broker] Fix potential to add duplicated consumer #15051
    • [fix][broker] Fix precision issue and initial value for Consumer#avgMessagesPerEntry #14666
    • [fix][broker] Fix problem at RateLimiter#tryAcquire #15306
    • [fix][broker] Fix producerFuture not completed in ServerCnx#handleProducer #14467
    • [fix][broker] Fix race condition between timeout and completion in OpAddEntry #15233
    • [fix][broker] Fix race condition in updating lastMarkDeleteEntry field #15031
    • [fix][broker] Fix rewind failed when redeliverUnacknowledgedMessages #15046
    • [fix][broker] Fix topic policy reader close bug #14897
    • [fix][broker] Fix typo in enum name and handle closing of the channel properly since writeAndFlush is asynchronous #15384
    • [fix][broker] Fix when nextValidLedger is null caused NPE #13975
    • [fix][broker] Fix wrong prompt exception when getting the non-persistent topic list without GET_BUDNLE permission #14638
    • [fix][broker] Fix wrong state for non-durable cursor #14869
    • [fix][broker] Fix wrong unit of NIC speed on Linux #15770
    • [fix][broker] Fix 404 error msg not being returned correctly using http lookup #14677
    • [fix][broker] Follow up on race condition fixes in ManagedCursorImpl #15067
    • [fix][broker] Remove the loadbalance/bundle-data node #13164
    • [fix][broker] Return if reset in progress #14978
    • [fix][broker] Use dispatchRateLimiterLock to update dispatchRateLimiter #15601
    • [fix][broker] When skipping updating mark delete position, execute callback with executor to prevent deadlock #15971
    • [fix][broker] Expose configurationMetadataStore and localMetadataStore #15661
    • [fix][broker] Filter the virtual NIC with relative path #14829
    • [fix][broker] Fix MetadataStoreException$NotFoundException while doing topic lookup #15633
    • [fix][broker] Fix calculation in getNumberOfEntriesInStorage #15627
    • [fix][broker] Fix error log miss stack trace when create tenant fail #14366
    • [fix][broker] Fix resource group does not report usage #15292
    • [fix][broker] Fix duplicated delayed messages when all consumers disconnect #14740
    • [fix][broker] Fix the pid occupied check when using pulsar-daemon start or stop process #14701
    • [fix][broker] Fix potential NPE in Replicator #15003
    • [improve][broker] Add log when updating namespace policies with error #14850
    • [improve][broker] Add publishRateLimitedTimes to topic metrics #15739
    • [improve][broker] Avoid using blocking calls for the async method checkTopicOwnership #15023
    • [improve][broker] Cancel offload tasks when managed ledger closed #14545
    • [improve][broker] Close connection if a ping or pong message cannot be sent #15382
    • [improve][broker] Configure DLog Bookie, Pulsar, and Admin clients via pass-through config #15818
    • [improve][broker] Full-support SSL provider, ciphers, and protocols for broker service and proxy service #14569
    • [improve][broker] Ignore the print the log that the topic does not exist #13535
    • [improve][broker] Optimize RawReader#create when using Compactor #14447
    • [improve][broker] Optimize find nics process #14340
    • [improve][broker] Optimize memory usage: support to shrink for pendingAcks map #14515
    • [improve][broker] Provide an accurate error message when set autoTopicCreation #14684
    • [improve][broker] Reduce unnecessary expansions for ConcurrentLong map and set #14562
    • [improve][broker] Set splitNamespaceBundle with readonly=false #14680
    • [improve][broker] Skip unnecessary DNS resolution when creating AuthenticationDataHttp instance #15221
    • [improve][broker] Support advertised listeners for HTTP and HTTPS services #14839
    • [improve][broker] Support shrink for ConcurrentSortedLongPairSet #15354
    • [improve][broker] Support shrink for map or set #14663
    • [improve][broker] Support shrink in ConcurrentLongHashMap #14497
    • [improve][broker] Switch to rely on Netty for Hostname Verification #15824
    • [improve][broker] Use shrink map for message redelivery #15342
    • [improve][broker] Use tlsCertRefreshCheckDurationSec instead of 0 for refresh value #15075
    • [improve][broker] Add metrics for pulsar web service thread pool #15741
    • [improve][broker] Allow pulsar_tool_env.sh PULSAR_MEM to be Overridden #15868
    • [improve][broker] Ignore case when obfuscating passwords in configuration scripts #15077

    Transaction

    • [fix][txn] Close the transaction buffer when deleting topics #14895
    • [fix][txn] Fix potentially unfinishable future. #15208
    • [fix][txn] Fix potentially unfinished CompletableFuture #14973
    • [fix][txn] Fix transaction PendingAck lowWaterMark #15530
    • [fix][txn] Fix transaction REST API redirect issue #15017
    • [fix][txn] Fix transaction admin redirect get 500 due to getCause #14965
    • [fix][txn] Fix transaction pendingAckStore asyncMarkDelete #14974
    • [fix][txn] Make transaction stats consistent at end txn #15472
    • [fix][txn] Properly close transaction-buffer-sub non durable cursor #14900
    • [fix][txn] Topic transaction buffer recover don't close reader when throw RuntimeException #15361
    • [fix][txn] TransactionMetadataService don't connect again if store exist #15114
    • [fix][txn] Avoid too many ServiceUnitNotReadyException for transaction buffer handler #14894
    • [fix][txn] Fix NPE of TransactionMetaStoreHandler #15840
    • [fix][txn] Fix cannot enable transaction when is allow auto update schema enabled=fasle #14809
    • [fix][txn] Fix pending ack is recovering throw CursorAlreadyClosedxception #14781
    • [fix][txn] Fix some exception handle in transaction buffer #14808
    • [fix][txn] Fix transaction buffer no snapshot close recover reader #14830
    • [fix][txn] Fix transaction buffer recover BrokerMetadataException close topic #14709
    • [fix][txn] Fix transaction buffer recover reader and writer fail #14801
    • [fix][txn] Fix transaction buffer recover throw cursor already close #14807
    • [fix][txn] Fix transaction log recover throw cursor already close #14810
    • [fix][txn] Fix transaction pending ack store managed ledger WriteFail state #14738
    • [fix][txn] Fix transaction producer stuck problem #15061
    • [fix][txn] Fix transaction component recover fillQueue #15418
    • [fix][txn] Fix transasction ack batch message #15875
    • [improve][txn] Avoid creating multiple future and exception handlers #15089
    • [improve][txn] Improve transaction perf logs #14816
    • [improve][txn] Optimize topic lookup when TC end tx #14991
    • [improve][txn] Optimize transaction lowWaterMark to clean useless data faster #15592
    • [improve][txn] Support configurable transactionBufferClientOperationTimeoutInMills #15011

    Security

    • [fix][auth] Add timeout of sync methods and avoid call sync method for AuthoriationService #15694
    • [fix][auth] Fix debug log authenticate role error #14784
    • [fix][auth] Fix grant all permissions but can't list topic #15501
    • [fix][auth] Fix handling single role and non-jwt-token in MultiRolesTokenAuthorizationProvider #14857
    • [fix][auth] Generate correct well-known OpenID configuration URL #15928
    • [fix][auth] Role with namespace produce authz can also get topics #15740
    • [improve][auth] Add KeyStore support in WebSocket, Function Worker HTTPS Servers #15084
    • [improve][auth] Allow to config web server's cipher and protocols #13354
    • [improve][auth] Full-support set SSL provider, ciphers, and protocols #13740
    • [improve][auth] Improve skipping of DNS resolution when creating AuthenticationDataHttp instance #15228
    • [improve][auth] Optimize the logic of allowing namespace operation #15731
    • [improve][auth] Remove sensitive msg from consumer/producer stats log #15483

    Admin

    • [fix][admin] Fix NPE in PulsarAdminBuilder when the service is not set #14769
    • [fix][admin] Fix pulsar-admin not prompting message when there is a 500 error #14856
    • [fix][admin] Fix reach max tenants error if the tenant already exists #15932
    • [fix][admin] Fix typo in validation message #16021
    • [fix][admin] Remove the trust certs check #14764

    Java Client

    • [fix][java] Fix ConsumerBuilderImpl can not set null to deadLetterPolicy #14980
    • [fix][java] Fix conversion of TimestampMillisConversion has no effect when Jsr310Conversion enabled #15863
    • [fix][java] Fix messages sent by producers without schema cannot be decoded #15622
    • [fix][java] Fix performance regression with message listener #15162
    • [fix][java] Fix potentially unfinished CompletableFuture in doReconsumeLater #14947
    • [fix][java] Fix race condition in consumer redelivery #14687
    • [fix][java] Fix the producer OOM if got an exception while adding messages to batch container #12170
    • [fix][java] PartitionsAutoUpdateFuture never complete #14625
    • [fix][java] Remove consumer when close consumer command is received #15761
    • [fix][java] Fix PartitionedProducerImpl flushAsync always fail when one partition send TimeOutException #14602
    • [improve][java] Add pending messages information while printing the producer stats #15440
    • [improve][java] AsyncHttpConnector doesn't use the system properties configured #15307
    • [improve][java] Avoid timer task run before previous subscribe complete #14818
    • [improve][java] Process maxRedeliverCount is 0 of DeadLeddterPolicy #14706
    • [improve][java] Improve logic when ACK grouping tracker checks duplicated message id #15465

    C++ Client

    • [fix][c++] Fix UnknownError might be returned for a partitioned producer #15161
    • [fix][c++] Fix connection is not closed when broker closes the connection to proxy #15009
    • [fix][c++] Fix producer is never destructed until client is closed #14797
    • [fix][c++] Fix segmentation fault when creating socket failed #14834
    • [fix][c++] Fix send callback might not be invoked in key-based batching #14898
    • [fix][c++] Fix single message metadata not set correctly #15072
    • [fix][c++] Fix the race condition of connect timeout task #14823
    • [fix][c++] Handle exception in creating socket when fd limit is reached #14587
    • [fix][c++] Wait until event loop terminates when closing the Client #15316
    • [fix][c++] Avoid race condition causing double callback on close #15508
    • [improve][c++] Add braces around initialization of subobject #14735

    Python Client

    • [fix][python] Fix generated Python protobuf code not compatible with the latest protobuf package #15846
    • [improve][python] Add build for Arm64 on Linux and Python 3.10 #15004
    • [improve][python] Add build for wheels on Alpine Linux #15016
    • [improve][python] Generate universal2 wheel files for MacOS #15054

    Websocket Client

    • [fix][websocket] Fix MultiTopicReader#getConsumer ClassCastException #15534

    Proxy

    • [fix][proxy] Fix proxy connection leak when inbound connection closes while connecting is in progress #15366
    • [fix][proxy] Log warning when opening connection to broker fails #14710
    • [fix][proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler #15415
    • [fix][proxy] Fix DNS server denial-of-service issue when DNS entry expires #15403
    • [improve][proxy] Configure Netty DNS resolver to match JDK DNS caching setting, share DNS resolver instance in Proxy #15219
    • [improve][proxy] Refactor Proxy code and fix connection stalling by switching to auto read mode #14713

    Pulsar SQL

    • [fix][sql] Fix the decimal type error convert in json schema #15687
    • [improve][sql] Add Java version trim agent for presto 332 #15236
    • [improve][sql] Pulsar SQL support for Decimal data type #15153

    Function

    • [fix][function] Pass configured metricsPort to k8s runtime #14502
    • [fix][function] Check executor null when close the FileSource #15247
    • [fix][function] Fix pulsar-managed runtimes failed start function with package URL from package management service #14814
    • [fix][function] Fix some IOExceptions when create functions from package URL #14553
    • [fix][function] Handle NPE when getLeader returns null #15058
    • [fix][function] Provide default error handler for function log appender #15728
    • [fix][function] Refine file io connector #15250

    Connector

    • [fix][connector] Pass client builder if no service URL is provided to Debezium connector #12145
    • [fix][connector] Throw exceptions when Kafka offset backing store failed to start #14491

    Tiered Storage

    • [fix][tiered-storage] Fix NoClassDefFoundError: com/google/inject/AbstractModule in pulsar-io/batch-data-generator and Jcloud offloader #14150
    • [improve][tiered-storage] Add debug information #14907
    • [improve][tiered-storage] Add pure S3 provider for the offloader #15710
    • [improve][tiered-storage] Reduce CPU usage when offloading the ledger #15063
    • [improve][tiered-storage] Upgrade JClouds to 2.5.0 #15649

    Dependencies

    • Add suppression for Kotlin stdlib CVE-2022-24329 - part 2 #14715
    • Add suppression for Kotlin stdlib CVE-2022-24329 #14629
    • Bump pyyaml from 5.3.1 to 5.4.1 to solve CVE-2020-14343 #15989
    • Fix false positive google-http-client-gson-1.41.0.jar #15651
    • Java version trim agent presto332 branch2.9 #15326
    • Remove --illegal-access errors resulting from Google Guice (upgrade to 5.0.1 and JClouds to 2.4.0) #13810
    • Remove --illegal-access errors resulting from Google Guice - Pulsar IO, Offloaders and Pulsar SQL - Bump Guice to 5.1.0 #14300
    • Upgrade BookKeeper to 4.14.5 (2.8, 2.9, 2.10 branches) #15581
    • Upgrade Hadoop to 3.3.3 to get rid of CVE-2022-26612 #15660
    • Upgrade Netty Reactive Streams to 2.0.6 #15990
    • Upgrade Netty to 4.1.76.Final, Netty Tcnative, grpc and protobuf #15212
    • Upgrade Netty to 4.1.77.Final and netty-tcnative to 2.0.52.Final #15646
    • Upgrade jackson and jackson-databind (2.13.2.1) to get rid of CVE-2020-36518 #14871
    • Use grpc-bom to align grpc library versions #15234
    • Remove log4j for CVE-2022-23307 #15109
    Source code(tar.gz)
    Source code(zip)

  • v2.10.1(Jul 4, 2022)

    Important notice

    • [fix][broker] Fix broker LoadBalance uneffective 15314
    • [fix][admin] Fix producer/consume permission can’t get schema 15956

    Broker

    • [fix][broker] Fix race condition in getManagedLedgerInternalStats when includeLedgerMetadata=true 15918
    • [improve][broker] Avoid contended synchronized block on topic load 15883
    • [fix][broker] Fix NPE in MessageDeduplication 15820
    • [improve][broker] Add timeout to closing CoordinationServiceImpl 15777
    • [improve][broker] Store Subscription properties 15757
    • [improve][broker] Support for updating the Subscription properties 15751
    • [improve][broker] Disable memory limit controller for broker client and replication clients 15723
    • [fix][broker] Fix NPE when put value to RangeCache. 15707
    • [fix][broker] Fast return if ack cumulative illegal 15695
    • [fix][broker] Fix creating producer failure when set backlog quota. 15663
    • [fix][broker] Expose configurationMetadataStore and localMetadataStore 15661
    • [fix][broker] Fix NPE when set AutoTopicCreationOverride 15653
    • [fix][broker] Fix MetadataStoreException$NotFoundException while doing topic lookup 15633
    • [fix][broker] Fix calculation in getNumberOfEntriesInStorage 15627
    • [fix][broker] Use dispatchRateLimiterLock to update dispatchRateLimiter 15601
    • [fix][broker] Sync topicPublishRateLimiter update 15599
    • [fix][broker] Fix deadlock in broker after race condition in topic creation failure 15570
    • [cleanup][broker] Override close method to avoid caching exception. 15529
    • [fix][broker] Close publishLimiter when disable it 15520
    • [fix][broker] Fix to avoid TopicStatsImpl NPE even if producerName is null 15502
    • [fix][broker] Fix key-shared delivery of messages with interleaved delays 15409
    • [fix][Broker] Fix bug in contructor of RocksdbMetadataStore 15405
    • [feature][broker] EntryFilter (PIP-105) - support per-Consumer filtering 15391
    • [fix][broker/client] Close connection if a ping or pong message cannot be sent 15382
    • [improve][broker] Support shrink for ConcurrentSortedLongPairSet 15354
    • [improve][broker] Support properties on NonDurable subscriptions (PIP-105) 15345
    • [improve][broker] Use shrink map for message redelivery. 15342
    • [fix][Broker] Fix error on recycled SubscriptionPropertiesList 15335
    • [fix][broker] Fix problem at RateLimiter#tryAcquire 15306
    • [fix][broker] Fix totalEntries calculation problem in AbstractBaseDispatcher#filterEntriesForConsumere 15298
    • [fix][broker] Fix resource group does not report usage 15292
    • [fix][Broker] Fix race condition between timeout and completion in OpAddEntry 15233
    • [fix][broker] Fix MessageDeduplication#inactiveProducers may not be persistence correctly 15206
    • [fix][broker] Cancel fencedTopicMonitoringTask when topic closed normally 15202
    • [fix][broker] Fix parameter saslJaasBrokerSectionName in broker.conf 15110
    • [cleanup][broker] Remove useless code to avoid confusion in OpReadEntry#checkReadCompletion 15104
    • [fix][broker] Ensure NamespaceEphemeralData has equals() operator 15092
    • [fix][broker] Fix potential to add duplicated consumer 15051
    • [fix][broker] Fix rewind failed when redeliverUnacknowledgedMessages 15046
    • [fix][broker]Fix race condition in updating lastMarkDeleteEntry field 15031
    • [improve][broker] Avoid using blocking calls for the async method checkTopicOwnership 15023
    • [fix][broker] Avoid heartbeat topic to offload. 15008
    • [fix][broker] Return if reset in progress 14978
    • [fix][broker] Fix topic policy reader close bug 14897
    • [fix][broker] Fix getPendingAckInternalStats redirect issue 14876
    • [fix][broker] Fix wrong state for non-durable cursor 14869
    • [improve][broker] Add log when update namespace policies with error 14850
    • [feature][broker] Support advertised listeners for HTTP and HTTPS services 14839
    • [fix][broker] Filter the virtual NIC with relative path 14829
    • [fix][broker] Fixed duplicated delayed messages when all consumers disconnect 14740
    • [fix][broker] Fix cannot delete namespace with system topic 14730
    • [fix][broker] Fixed 404 error msg not being returned correctly using http lookup 14677
    • [fix][broker] Fix normal topic named ends with healthcheck becomes system topic issue 14671
    • [improve][broker] Support shrink for map or set 14663
    • [improve][broker] Changing the topic creation flow and optimize heartbeat topic not trigger compaction 14643
    • [fix][broker] Fix wrong prompt exception when getting the non-persistent topic list without GET_BUDNLE permission 14638
    • [fix][broker] Fix inconsistent prompt message when schema version is empty using AVRO. 14626
    • [fix][broker] Fix update replication cluster but not update replicator 14570
    • [improve][broker] Reduce unnecessary expansions for ConcurrentLong map and set 14562
    • [improve][broker] Support ManagedCursorInfo compression 14542
    • [improve][broker] Optimize memory usage: support to shrink for pendingAcks map 14515
    • [improve][broker] Support shrink in ConcurrentLongHashMap 14497
    • [improve][broker] Optimize RawReader#create when using Compactor 14447
    • [fix][broker] Fix NPE when subscription is already removed 14363
    • [improve][broker] Optimize load manager find nics process 14340
    • [improve][broker] Make revokePermissionsOnTopic method async 14149
    • [fix][broker] Fix when nextValidLedger is null caused NPE 13975

    Clients

    • [fix][Java] Fixes NPE when TableView handles null value message 15951
    • [fix][Java] Fix conversion of TimestampMillisConversion has no effect when Jsr310Conversion enabled 15863
    • [fix][Java] Fix messages sent by producers without schema cannot be decoded 15622
    • [improve][Java] improve logic when ACK grouping tracker checks duplicated message id 15465
    • [improve][Java] Add pending messages information while printing the producer stats 15440
    • [fix][Java] Fix negative ack not redelivery 15312
    • [improve][admin/client] AsyncHttpConnector doesn't use the system properties configured 15307
    • [fix][Java] TableView should cache created readers 15178
    • [fix][Java] Fix performance regression with message listener 15162
    • [fix][Java] Fix internal receive used wrong timeout type 15014
    • [fix][Java] ConsumerBuilderImpl can not set null to deadLetterPolicy. 14980
    • [fix][Java] Returns immutable data set when use TableView. 14833
    • [improve][Java] Avoid timer task run before previous subscribe complete. 14818
    • [fix][C++] avoid race condition causing double callback on close 15508
    • [fix][C++] Wait until event loop terminates when closing the Client 15316
    • [fix][C++] Fix UnknownError might be returned for a partitioned producer 15161
    • [fix][C++] Fix single message metadata not set correctly 15072
    • [fix][C++] Fix connection is not closed when broker closes the connection to proxy 15009
    • [fix][C++] Fix send callback might not be invoked in key based batching 14898
    • [fix][C++] Fix segmentation fault when creating socket failed 14834
    • [fix][C++] Fix the race condition of connect timeout task 14823
    • [fix][C++] Fix producer is never destructed until client is closed 14797
    • [fix][Python] Fixes reserved keys are not removed when JsonSchema is being encoded 15947
    • [fix][Python] Fix generated Python protobuf code not compatible with latest protobuf package 15846
    • [improve][Python] Generate universal2 wheel files for macOS 15054
    • [improve][Python] Added build for wheels on Alpine Linux 15016
    • [improve][Python] Added build for Arm64 on Linux and Python 3.10 15004
    • [WebSocket] Fix ClassCastException when user create MultiTopicReader. #14316
    • [WebSocket] Add debug log for WebSocket. #12458

    Transaction

    • [fix][txn] Fix transasction ack batch message 15875
    • [fix][txn] Fix NPE of TransactionMetaStoreHandler 15840
    • [improve][txn] Optimize transaction lowWaterMark to clean useless data faster 15592
    • [fix][txn] Fix transaction PendingAck lowWaterMark 15530
    • [fix][txn] Make transaction stats consistent at end txn 15472
    • [improve][txn] Add lowWaterMark check before appending entry to TB 15424
    • [fix][txn] Fix transaction component recover fillQueue 15418
    • [fix][txn] Topic transaction buffer recover don't close reader when throw RuntimeException 15361
    • [fix][txn] Fix potentially unfinishable future 15208
    • [improve][txn] Optimize metadataPositions in MLPendingAckStore 15137
    • [fix][txn] TransactionMetadataService don't connect again if store exist 15114
    • [improve][txn] Avoid creating the multiple future and exception handler 15089
    • [fix][txn] Fix transaction REST API redirect issue. 15017
    • [improve][txn] support configurable transactionBufferClientOperationTimeoutInMills 15011
    • [improve][txn] Optimize topic lookup when TC end tx 14991
    • [fix][txn] Fix potential NPE in TransactionBufferDisable 14979
    • [fix][txn] Fix transaction pendingAckStore asyncMarkDelete 14974
    • [fix][txn] Fix potentially unfinished CompletableFuture 14973
    • [fix][txn] Fix transaction admin redirect get 500 due to getCause 14965
    • [fix][txn] Properly close transaction-buffer-sub non durable cursor 14900
    • [fix][txn] Close the transaction buffer when deleting topics 14895
    • [fix][txn] Avoid too many ServiceUnitNotReadyException for transaction buffer handler 14894
    • [fix][txn] Fix transaction buffer no snapshot close recover reader 14830
    • [fix][txn] Fix transaction log recover throw cursor already close 14810
    • [fix][txn] Fix cannot enable transaction when is allow auto update schema enabled=fasle 14809
    • [fix][txn] Fix some exception handle in transaction buffer 14808
    • [fix][txn] Fix transaction buffer recover throw cursor already close 14807
    • [fix][txn] Fix transaction buffer recover reader and writer fail 14801
    • [fix][txn] Fix pending ack is recovering throw CursorAlreadyClosedxception 14781
    • [fix][txn] Fix cursor readPosition is bigger than maxPosition in OpReadEntry 14667

    Pulsar IO and Pulsar Functions

    • [fix][connector] KCA sinks: fix offset mapping when sanitizeTopicName=true 15950
    • [improve][function] provide default error handler for function log appender 15728
    • [fix][connector] KCA Sink: org.apache.kafka.connect.errors.DataException: Invalid Java object for schema with type .. 15598
    • [improve][function] Refine file io connector 15250
    • [fix][function] Check executor null when close the FileSource 15247
    • [fix][function] Handle NPE when getLeader returns null 15058
    • [fix][function] Allow a Function<GenericObject,?> to access the original Schema of the Message and use it 14847
    • [fix][function] Fix pulsar-managed runtimes failed start function with package URL from package management service 14814
    • [improve][connector] Handle Kafka sinks that return immutable maps as configs 14780
    • [improve][connector] Support event-time-based index name in ES Sink 14383

    Tiered Storage

    • [feature][tiered-storage] Add pure S3 provider for the offloader [15710](https://github.com/apache/pulsar/pull/15710
    • [improve][tiered-storage] Upgrade JClouds to 2.5.0 15649
    • [improve][tiered-storage] Reduce CPU usage when offloading the ledger 15063
    • [fix][tiered-storage] Fix potential NPE in MockManagedLedger 15006
    • [improve][tiered-storage] Add debug information 14907

    Pulsar SQL

    • [fix][sql] Fix the decimal type error convert in json schema 15687
    • [fix][sql] Add Java version trim agent for presto 332 15236
    • [improve][sql] Pulsar SQL support for Decimal data type 15153

    Pulsar Proxy

    • [cleanup][proxy] Remove unnecessary blocking DNS lookup in LookupProxyHandler 15415
    • [fix][proxy] Fix DNS server denial-of-service issue when DNS entry expires 15403
    • [fix][proxy/client] Configure Netty DNS resolver to match JDK DNS caching setting, share DNS resolver instance in Proxy 15219
    • [refactor][proxy] Refactor Proxy code and fix connection stalling by switching to auto read mode 14713
    • [improve][proxy] Log warning when opening connection to broker fails 14710
    • [improve][proxy] Add support of PrometheusRawMetricsProvider for the Pulsar-Proxy 14681

    Observability

    • [improve][metrics] Add message ack rate metric for consumer 15674
    • [improve][metrics] Add metrics for pulsar web service thread pool 14742
    • [improve][metrics] Add non-persistent topic subscription metrics 13827

    CLI

    • [improve][admin] Support to get topic properties 15944
    • [improve][admin] Pulsar Admin: create subscripion with Properties (PIP-105) 15503
    • [improve][admin] Put validateTopicOwnershipAsync before validateTopicOperationAsync 15265
    • [fix][admin] Fix inconsistent parameter of TopicPolicies.getSubscriptionDispatchRate 15293
    • [fix]admin] Fix pulsar-admin not prompting message when there is a 500 error 14856
    • [fix][admin] Fix NPE in PulsarAdminBuilder when the service is not set 14769
    • [fix][admin] Remove the trust certs check 14764
    • [fix][admin] Provide an accurate error message when set autoTopicCreation 14684

    Security

    • [fix][authn] Generate correct well-known OpenID configuration URL 15928
    • [fix][authn] Switch to rely on Netty for Hostname Verification 15824
    • [fix][authz] Add timeout of sync methods and avoid call sync method for AuthoriationService 15694
    • [fix][authz] Fix grant all permissions but can't list topic. 15501
    • [fix][authz] Fix MultiRolesTokenAuthorizationProvider authorize issue. 15454
    • [fix][authn] Fix typo checkPermissionsAsync method typo 15273
    • [improve][authn] Improve skipping of DNS resolution when creating AuthenticationDataHttp instance 15228
    • [improve][authn] Skip unnecessary DNS resolution when creating AuthenticationDataHttp instance 15221
    • [fix][security] Remove log4j for CVE-2022-23307 15109
    • [improve][authn] Use tlsCertRefreshCheckDurationSec instead of 0 for refresh value 15075
    • [fix][security] Upgrade Spring Context in Pulsar IO batch-data-generator 14975
    • [fix][auth] Athenz: do not use uber-jar and bump to 1.10.50 to remove jackson-databind shaded dependency 14884
    • [fix][authz] Fix handling single role and non-jwt-token in MultiRolesTokenAuthorizationProvider 14857
    • [improve][authn] Full-support SSL provider, ciphers and protocols for broker service and proxy service 14569
    • [fix][authz] Role with namespace produce authz can also get topics 13773
    • [improve][authn] Full-support set SSL provider, ciphers and protocols for broker-web&websocket/proxy/function-worker 13740

    Others

    • [improve] Enable TCP/IP keepalive for all ZK client connections in all components starting with bin/pulsar 15908
    • [improve] Allow pulsar_tool_env.sh PULSAR_MEM to be Overridden 15868
    • [improve] Configure DLog Bookie, Pulsar, and Admin clients via pass-through config 15818
    • [fix][docker] Add write permissions to /pulsar subdirectories to enable running as non-root user 15769
    • [improve] Disable memory limit controller in internal Pulsar clients 15752
    • [improve] Disable Pulsar client memory limit by default 15748
    • [fix][owasp] Fix false positive google-http-client-gson-1.41.0.jar 15651
    • [fix][package-management] Fix the new path /data introduced regression 15367
    • [improve][common] Use Collection instead of List for FutureUtil. 15329
    • [fix][tools] Only apply maxPendingMessagesAcrossPartitions if it presents 15283
    • [fix][owasp] Suppress MariaDB false positives 15243
    • [fix][scripts] Ignore case when obfuscating passwords in configuration scripts 15077
    • [improve][tool] Improve transaction perf logs 14816
    • [fix][deploy] Fix the pid occupied check when using pulsar-daemon start or stop process 14701

    Library updates

    • Bump PyYAML from 5.3.1 to 5.4.1 to solve CVE-2020-14343 15989
    • Upgrade Netty to 4.1.76.Final, Netty Tcnative, grpc and protobuf 15212
    • Update spring library to fix CVE-2022-22965 15065
    • Upgrade MySQL client to 8.0.28 to get rid of CVE-2021-3711 14998
    • Upgrade jackson and jackson-databind (2.13.2.1) to get rid of CVE-2020-36518 14871
    • Upgrade Guice to 5.1.0 14300
    Source code(tar.gz)
    Source code(zip)

  • v2.10.0(Apr 18, 2022)

    Important notice

    • Remove -XX:-ResizePLAB JVM option which degrades performance on JDK11 #12940
    • Enable TCP keepAlive flag on the sockets #12982
    • Reduce the time it takes for namespace bundle unloading to time out #12995
    • Align configurations defaults between default file and Java object (broker.conf, proxy.conf, websocket.conf) #13272
    • [PIP 118] Do not restart brokers when ZooKeeper session expires as default #13341
    • [PIP 119] Enable consistent hashing by default on KeyShared subscriptions dispatcher #13352
    • [PIP 120] Enable client memory limit controller by default #13344
    • Make Docker images non-root, by default, and OpenShift compliant #13376
    • [PIP 122] Change loadBalancer default loadSheddingStrategy to ThresholdShedder #13733
    • Fix netcat returning early for probe #14088

    PIPs

    • [PIP 79] Add lazy-loading feature to PartitionedProducer #10279
    • [PIP 84] Pulsar client: Redeliver command add epoch #10478
    • [PIP 86] Pulsar Functions: Preload and release external resources #13205
    • [PIP 92] Topic policy across multiple clusters #12517
    • [PIP 104] Add new consumer type: TableView #12838
    • [PIP-105] Support pluggable entry filter in Dispatcher #12869 #12970 #12979
    • [PIP 106] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility #12536
    • [PIP 107] Introduce chunk message ID #12403
    • [PIP 110] Support Topic metadata - PART-1 create topic with properties #12818
    • [PIP 121] Pulsar cluster level auto failover on client side #13316
    • [PIP-124] Create init subscription before sending message to DLQ #13355
    • [PIP-130] Apply redelivery backoff policy for ack timeout #13707
    • [PIP 131] Resolve produce chunk messages failed when topic level maxMessageSize is set #13599
    • [PIP 135] Include MetadataStore backend for Etcd #13225

    Broker

    • [PIP 45] Pluggable metadata interface
      • Add BookKeeper metadata adapter based on MetadataStore #12770
      • Add Rocksdb metadata store #12776
      • Convert BookieRackAffinityMapping to use MetadataStore #12841
      • Allow to configure metadata store URL in broker.conf #13077
      • Remove old ZK caches implementations #13075
      • Allow to start bookie with Pulsar metadata store backend #13296
      • Remove ZooKeeperClientFactory #13303
      • Use reference counting in RocksDB metadata store #13309
      • Allow configuring metadata store URL in proxy.conf #13777
      • Allow configuring metadata store URL in functions_worker.yml #13782
      • Add configuration metadata store url setting in WebSocket #13786
      • Allow configuring configurationMetadataStore when initializing cluster metadata #13889
      • Using the consistent metadata store scheme name #13937
      • Allow configuring metadataStoreUrl in pulsar-perf managed-ledger #14145
      • Deprecate zookeeper settings #14147
      • Fix metadata cache inconsistency on doing refresh #14283
    • Support broker level dispatch rate limiter #11325
    • Support setting geo-replication clusters on topic level #12136
    • Add Multi-tiered storage key to broker.conf #12173
    • Support configuration to rate-limit dispatching on batch message #12294
    • Support splitting the largest bundle of the namespace #12361
    • Support creating a subscription by specifying the earliest or latest position #12872
    • Support roll-over ledgers for inactive topics #13073
    • Support graceful shutdown for Broker #14114
    • Transparent batching of ZK operations #13043
    • Add uniform load shedder strategy to distribute traffic uniformly across brokers #12902
    • Provide option to split bundle based on load #12378
    • Use AuthorizationService#grantPermissionAsync to grant topic permission #12515
    • Only store authentication data after authentication is complete #12077
    • Allow to have different instances LocalMemoryMetadataStore that share the same state #12390
    • Allow GetTopicsOfNamespace op with consume permission #12600
    • Add local filesystem backend for package manager #12708
    • Add stop replicator producer logic when start replicator cluster failed #12724
    • Apply PolicyHierarchyValue to inactiveTopicPolicies #12687
    • Fix can not get leader broker in follower brokers #11353
    • Fix broker gc log options #11285
    • Fix NPE of ZkBookieRackAffinityMapping #11947
    • Fix prefix setting in JWT authn and avoid multi calls for the getProperty #12132
    • Fix missed check exit code of stop before calling start #12368
    • Fix getting children of parent nodes in LocalMemoryMetadataStore #12491
    • Fix collection get bug in ResourceGroupService #12499
    • Fix deadlock in metadata-store callback thread #12753
    • Improve exceptions thrown when handling the schema resource #12155
    • Trim topic name #12453
    • Avoid unnecessary recalculation of maxSubscriptionsPerTopic in AbstractTopic #12658
    • Optimize isValidPath check in MetadataStore #12663
    • Close connection after receiving unexpected SendCommand #12780
    • Ensure cache is refreshed (and not just invalidated) after a store write #12788
    • Optimize topicMaxMessageSize with topic local cache. #12830
    • Optimize blocking backlogQuotaCheck to non-blocking in ServerCnx#handleProducer #12874
    • Only refresh metadata if path is already in cache after write. #12896
    • Optimize put and get methods in AbstractMetadataStore #12916
    • Fix zk-node leak of admin path #12972
    • Optimize TopicPolicy#deduplicationEnabled with HierarchyTopicPolicies #13000
    • Fix race condition in FaultInjectionMetadataStore#programmedFailure #13007
    • Fix usage of PULSAR_EXTRA_OPTS/BOOKIE_EXTRA_OPTS in startup scripts #13025
    • Consider topics in pulsar/system namespace as system topics #13050
    • Fix wrong result for looking up a non-exist topic by rest api #13055
    • Key_Shared dispatcher with no connected consumers should be recreated if allowOutOfOrderDelivery changes #13063
    • Make load-balancer config dynamic for the runtime tuning #13074
    • Optimize TopicPolicy#maxProducersPerTopic with HierarchyTopicPolicies #13082
    • Load balancer supports disabling max-session for bundle split #13108
    • Optimize TopicPolicies#subscriptionTypesEnabled with HierarchyTopicPolicies #13121
    • Log thread dump when Zookeeper session expires to help detect possible deadlocks #13124
    • Add removeMaxConsumersPerSubscription method for v1 namespace #13192
    • Fixe error value for 1MB in metrics #13213
    • Optimize TopicPolicies#messageTTLInSeconds with HierarchyTopicPolicies #13241
    • Add schema compatibility strategy on topic level #13297
    • Optimize TopicPolicies#maxConsumerPerTopic with HierarchyTopicPolicies #13361
    • Optimize TopicPolicies#replicationClusters with HierarchyTopicPolicies #13495
    • Fix compatibility issue with other metadata store implementations in resources #13394
    • Fix race condition in stopping replicator while it is starting #13412
    • Support backlog quota across multiple clusters #13445
    • Return null instead of RestException when getting bookieAffinityGroup #13462
    • Fix failed to get Partitioned metadata for health checking #13525
    • Optimize TopicPolicies#maxConsumersPerSubscription with HierarchyTopicPolicies #13548
    • Fix raw use of generic types in OwnershipCache #13592
    • Optimize TopicPolicies#maxUnackedMessagesOnConsumer with HierarchyTopicPolicies #13618
    • Optimize TopicPolicies#delayedDelivery Enabled and TickTimeMillis with HierarchyTopicPolicies #13649
    • Optimize TopicPolicies#compactionThreshold with HierarchyTopicPolicies #13710
    • Add deleteSubscriptionDispatchRate method for v1 namespace #13711
    • Fix NPE of checkReplication #13720
    • Optimize deduplicationSnapshotIntervalSeconds with HierarchyTopicPolicies #13769
    • Use shared executors for broker and geo-replication clients #13839
    • Fix call sync method in async rest api for
      • for internalDeletePartitionedTopic #13805
      • for internalCreateSubscription. #13873
      • for internalGetBacklogSizeByMessageId #13871
      • for internalTriggerCompaction #13853
      • for internalGetManagedLedgerInfo #13847
      • for internalGetSubscriptions #13846
      • for internalUnloadTopic #13845
      • for internalGetLastMessageId #13882
      • for internalExpireMessagesByTimestamp #13880
      • for internalResetCursorOnPosition #13879
      • for internalExpireMessagesByPosition #13878
      • for internalGetMessageById. #13876
      • for internalSetReplicatedSubscriptionStatus #13887
      • for internalGetPartitionedStats #13886
      • for onPoliciesUpdate method #13885
      • for internalGetPartitionedStatsInternal #13884
      • for internalTerminatePartitionedTopic #13890
      • for internalSkipAllMessages #13901
      • for preValidation #13962
      • for internalSetReplicationClusters and internalRemoveReplicationClusters #13961
    • Make PulsarAuthorizationProvider#grantPermissionAsync actually async #13897
    • Fix no value present when updating dispatch rate #13900
    • Optimize retentionPolicies with HierarchyTopicPolicies #13909
    • Improve 403 response message wih consume permission on namespace #13912
    • Fix arithmetic exception for uniform load shedder #13914
    • Add threshold for each unload round for uniform load shedder #13915
    • Fix call sync method in async rest api for internalDeleteSubscription #13884 #13917
    • Change BrokersBase api getActiveBrokers and getLeaderBroker to pure async. #13935
    • Do not create missing topic when loading namespace #13948
    • Only reply to client when code completes producerFuture #13949
    • Improve error logging for topic not found #13950
    • Make validateTenantOperation method async in PulsarWebResource #14008
    • Optimize maxUnackedMessagesOnSubscription with HierarchyTopicPolicies #14011
    • Set default value of applied to false in getSchemaCompatibilityStrategy #14012
    • Make validateTopicPolicyOperation method async in PulsarWebResource #14024
    • Fix producerFuture.completeExceptionally not called before sendErrorResponse #14025
    • Make triggerOffload method async #14027
    • Make offloadStatus method async #14029
    • Make validateNamespacePolicyOperation method async in PulsarWebResource #14033
    • Make internalSkipMessages method async #14045
    • Make PersistentTopicsBase#internalSetBacklogQuota async #14051
    • Adjust the validation for policy schemaCompatibilityStrategy #14061
    • Remove Persistent Topics v3 API - use custom media type instead #14117
    • Fix race condition in PulsarLedgerIdGenerator#generateShortLedgerId #14118
    • Optimize topic policy with HierarchyTopicPolicies about subscriptionDispatchRate #14151
    • Make BrokerBase#deleteDynamicConfiguration to pure async method #14163
    • Set default value of applied to false on topic policy #14181
    • Change broker producer fence log level #14196
    • Remove duplicated filter for UniformLoadShedder#findBundlesForUnloading #14198
    • Adjust topic exists check logic in http lookup process #14199
    • Fix NPE of internalExpireMessagesByTimestamp #14243
    • Fix rackaware placement policy does not take effect after delete rack configuration #14248
    • Fix print error log when server return redirect (http code 307) #14259
    • Optimize topic policy with HierarchyTopicPolicies about publishRate #14267
    • Fix ack-hole and backlog for persistent-replicator #14282
    • Fix NPE in internalSkipMessages #14297
    • Validate blank advertised listener name #14306
    • Let entries expire in the metadata caches #14154
    • Fix transaction system topic loop creation #12749
    • Fix topic transaction buffer handle null snapshot #12758
    • Optimize changeToCloseState method #14277

    Clients

    • [Java] Support creating a consumer in the paused state #11974
    • [Java] Support passing existing executor providers to the client #12037
    • [Java] Fix the producer OOM if got an exception while adding messages to batch container #12170
    • [Java] Allow to config client allocator out of memory policy #12200
    • [Java] Support negative ack redelivery backoff #12566
    • [Java] Fix confusing logs in UnAckedMessageTracker #13017
    • [Java] Fix parseProtobufSchema method will be called two times #13163
    • [Java] Add getNumPartitions method into PartitionedProducerImpl #13239
    • [Java] Allow config client dns bind addr and port #13390
    • [Java] Support adding custom properties for the reconsumeLater interface #13461
    • [Java] Allow Client Builder set Dnslookup params #13503
    • [Java] Avoid repeatedly set startMessageIdData to null for ConsumerImpl #13606
    • [Java] Let the 'properties' to be empty for ConsumerBuilder and ProducerBuilder #14074
    • [Java] Log producer batchSize and msgSize percentiles #14229
    • [C++] Add Wireshark cmake and fix build with latest Wireshark #13236
    • [C++] Wireshark Pulsar dissector naming replace yahoo with apache #13251
    • [C++] Support arm64 optimized CRC32c hardware-instructions #13246
    • [C++] Support more pulsar command names and versions in Wireshark dissector #13286
    • [C++] Optimize MessageBuilder and SharedBuffer to avoid unnecessary memory copy #13293
    • [C++] Fix in Apple Silicon macOS the clang-format cannot find #13333
    • [C++] Add clang-format check for Pulsar Wireshark dissector #13349
    • [C++] Adjust clang-format search names #13369
    • [C++] Fix Wireshark dissector decode send command metadata behavior #13471
    • [C++] PIP 37: Support large message size #13627
    • [C++] Fix the consumer configuration inconsistency with Java client #14070
    • [Python] Provide str operator for BytesSchema #12593
    • [Python] Support is_connected in Python Client #13662
    • [C] Add missing includes in reader_configuration.h #12966
    • [C] Add pulsar_client_subscribe_multi_topics and pulsar_client_subscribe_pattern #12965

    Pulsar IO and Pulsar Functions

    • [Functions] Prevent NPE while stopping a non started Pulsar LogAppender #12643
    • [Functions] Allow configuring different implementations for Pulsar functions state store #12646
    • [Functions] Clean os even when statusFuture complete exceptionally #12767
    • [Functions] Override inactive_topic_policies in Pulsar Functions namespace creation #13048
    • [Functions] Add possibility to pass additional JVM arguments to the function JVM (additionalJavaRuntimeArguments) #13282
    • [Functions] Fix getTlsTrustChainBytes not work when functions worker not run with broker #13875
    • [Functions] Fix distributed log metadata not correctly initialized #13891
    • [Functions] Remove sensitive information from log #14159
    • [IO Connector] Pass client builder if no service url provided to debezium connector #12145
    • [IO Connector] Fix: "Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed for user 'sa'.." in MS SQL integration test 12374
    • [IO Connector] Fix the marshal and unmarshal the sink config #12625
    • [IO Connector] Support renaming the file that has been processed #13373
    • [IO Connector] Support Fixed and ENUM datatypes for ElasticSearch Sink #13800
    • [IO Connector] Implement --retain-key-ordering (KEY_SHARED subscription) for Sinks #14083

    Observability

    • [Broker] Expose broker bundles metrics to prometheus #12366
    • [Broker] Add publishRateLimitedTimes to topic metrics #13538
    • [Broker] Fix bundle metrics would overwrite loadbalance metrics #13641
    • [Broker] Fix managed cursor acknowledgment state metric names #13844
    • [Java Client] Add pending-queue size metrics to producer stats #12674

    CLI

    • [Pulsar Admin] Support a mapping from brokers to partitioned topics that each broker serves #11763
    • [Pulsar Admin] Support the termination of a partitioned topic #11893
    • [Pulsar Admin] Update command descriptions from old 'property/cluster/namespace' format to current 'tenant/namespace' format #10485
    • [Pulsar Admin] Add remove-subscription-types-enabled command for namespace #12392
    • [Pulsar Admin] Fix output format of string by pulsar-admin command #11878
    • [Pulsar Admin] Add a metric to get the earliest time in the backlog #12523
    • [Pulsar Admin] Reduce severity of log "refreshing key manager" in KeyManagerProxy #12594
    • [Pulsar Admin] Support getting a list of topics under a namespace bundle #12632
    • [Pulsar Admin] Add get-replicated-subscription-status command for topic #12891
    • [Pulsar Admin] Add remove-subscription-types-enabled command for topic #12983
    • [Pulsar Admin] Validate the size options in cmd for topic and namespace #13002
    • [Pulsar Admin] Support subscription across multiple clusters #13482
    • [Pulsar Admin] Support deduplication across multiple clusters #13487
    • [Pulsar Admin] Support message TTL across multiple clusters #13484
    • [Pulsar Admin] Support persistence policies across multiple clusters #13483
    • [Pulsar Admin] Support publish rate across multiple clusters #13496
    • [Pulsar Admin] Support max consumers across multiple clusters #13521
    • [Pulsar Admin] Support max producer across multiple clusters #13519
    • [Pulsar Admin] Support compaction threshold across multiple clusters #13513
    • [Pulsar Admin] Support offload policies across multiple clusters #13534
    • [Pulsar Admin] Support max unacked messages per consumer across multiple clusters #13547
    • [Pulsar Admin] Support max message size support across multiple clusters #13579
    • [Pulsar Admin] Support deduplication snapshot interval across multiple clusters #13578
    • [Pulsar Admin] Support delayed delivery policy across multiple clusters #13550
    • [Pulsar Admin] Support max unacked messages on subscription across multiple clusters #13549
    • [Pulsar Admin] Support replicator dispatch rate across multiple clusters #13624
    • [Pulsar Admin] Support max subscriptions per topic across multiple clusters #13623
    • [Pulsar Admin] Move schema compatibility strategy cmd from topics to topicPolicies #14225
    • [Pulsar CLI] Add restart command to pulsar-daemon #12279
    • [Pulsar CLI] Print log for standalone when configuration is failed to load #12280
    • [Client Tool] Support Disabling Replication #13659
    • [Perf Tool] Add auth for transaction perf #14271

    Others

    • [Proxy] Fix Pulsar Proxy to re-use authentication instance #12245
    • [Proxy] Fix auto-cert refresh when proxy connects to broker #14130
    • [WebSocket] Fix the batch message ack #12530
    • [Package Management] check service status before run commands #12847
    • [Package Management] Add package management filesystem storage into the distribution lib #13202
    • [Package Management] Fix the filesystem storage failure #13218
    • [Schema] Fix pulsar use json or avro primitive schema #12886
    • [Tiered Storage] Fix FileSystemManagedLedgerOffloader can not cleanup outdated ledger #12309
    • [Tiered Storage] Fix NoClassDefFoundError: com/google/inject/AbstractModule in pulsar-io/batch-data-generator and Jcloud offloader #14150
    • Add lua wireshark for pulsar #13564
    • Fix missing new line at the end of proxy.conf #14359

    Library updates

    • Upgrade function's Go client version to v0.7.0 #12839
    • Upgrade dependencies (guava and what brought in older guava) to get rid of the guava-related CVE-2018-10237 and CVE-2020-8908 #13716
    • Use dependencyManagement to enforce snakeyaml version to 1.30 #13722
    • Upgrade dependencies to get rid of pulsar-io/jdbc related CVE-2020-13692 #13753
    • Remove --illegal-access errors resulting from Google Guice (upgrade to 5.0.1 and JClouds to 2.4.0) #13810
    • Upgrade jakarta.el to 3.0.4 to get rid of CVE-2021-28170 #13943
    • Upgrade Netty to 4.1.73.Final #13981
    • Remove net.jodah.failsafe dependency (fix JDK17 build) #14124
    • Upgrade netty version to 4.1.74.Final #14257
    Source code(tar.gz)
    Source code(zip)

  • v2.9.2(Apr 8, 2022)

    ƒ Security

    • Get rid of CVEs in Solr connector #13822
    • Get rid of CVEs in InfluxDB connector #13821
    • Get rid of CVEs in batch-data-generator #13820
    • Get rid of CVEs brought in with aerospike #13819
    • Suppress false positive Avro CVE-2021-43045 #13764
    • Upgrade protobuf to 3.16.1 to address CVE-2021-22569 #13695
    • Upgrade Jackson to 2.12.6 #13694
    • Upgrade Log4j to 2.17.1 to address CVE-2021-44832 #13552
    • Upgrade to Log4J 2.17.0 to mitigate CVE-2021-45105 #13392
    • Cipher params not work in KeyStoreSSLContext #13322
    • Use JDK default security provider when Conscrypt isn't available #12938
    • Return if namespace policies are read only #12514
    • Upgrade Postgre driver to 42.2.25 to get rid of CVE-2022-21724 #14119
    • Bump slf4j from 1.7.25 to 1.7.32 #13595
    • Upgrade Netty to 4.1.72 - CVE-2021-43797 #13328

    Broker

    • Fix lost message issue due to ledger rollover. #14664
    • Fix thread leak in MLTransactionMetadataStore #14524
    • Fix npe in ManagedLedgerImpl #14481
    • Fix Future.join() causing deadlock. #14469
    • Fix Broker HealthCheck Endpoint Exposes Race Conditions #14367
    • Validate rack name #14336
    • Fix avg-messagePerEntry metrics for consumer #14330
    • Fix the wrong parameter in the log. #14309
    • Fix batch ack count is a negative issue. #14288
    • Bug fix: IllegalArgumentException: Invalid period 0.0 to calculate rate #14280
    • Clean up individually deleted messages before the mark-delete position #14261
    • If mark-delete operation fails, mark the cursor as "dirty" #14256
    • Fix detecting number of NICs in EC2 #14252
    • Remove log unacked msg. #14246
    • Change broker producer fence log level #14196
    • Avoid creating any topics in NamespaceService#checkTopicExists during topic lookup. #14134
    • Fix NPE of cumulative ack mode and incorrect unack message count #14021
    • Fix NPE - ReplicatedSubscriptionsController send marker message when enable deduplicated. #14017
    • KeyShared stickyHashRange subscription: prevent stuck subscription in case of consumer restart #14014
    • Add null check to workaround NPE in unit tests with Mockito/PowerMock #14006
    • Fix handling of consumers with equal names on on key shared selector with consistent hashing #13991
    • Trim configuration value string which contains blank prefix or suffix #13984
    • WaitingCursors potential heap memory leak #13939
    • Fix read schema compatibility strategy priority #13938
    • Fix SystemTopicBasedTopicPoliciesService NPE issue. #13840
    • NPE when get isAllowAutoUploadSchema #13831
    • Fix topic produced through REST not support Authorization #13771
    • Fix call sync method in async rest API for internalGetSubscriptionsForNonPartitionedTopic #13745
    • Fix the deadlock while using zookeeper thread to create ledger #13744
    • Fix inefficient forEach loop #13742
    • Fix non persistent topic subscription error. #13685
    • Fix invalid rack name cause bookie join rack failed #13683
    • Release old bundle from ownership cache when operator split bundle #13678
    • Avoid call sync method in async rest API for force delete subscription #13668
    • Avoid call sync method in async rest API for delete subscription #13666
    • Fix getInternalStats occasional lack of LeaderInfo #13656
    • Fix internal topic effect by InactiveTopicPolicy. #13611
    • Fix bug :Infinity value for CPU or Bandwidth usage #13609
    • Change ContextClassLoader to NarClassLoader in BrokerInterceptor #13589
    • Fix NPE when unloading namespace bundle #13571
    • Allow both limit and limitsize be null #13557
    • The log prints NamespaceService#isServiceUnitActive exception stack information. #13553
    • Update log content #13540
    • HealthCheck v2 failed #13525
    • Change ContextClassLoader to NarClassLoader in AdditionalServlet #13501
    • Optimize the debug log that affects performance, and unify the style #13498
    • Fix multi roles authz cannot handle empty roles case #13477
    • Fix the NPE in system topics policies service #13469
    • Fix race conditions in closing producers and consumers #13428
    • Fix deadlock in metadata-store callback thread for branch 2.9 #13426
    • Remove check resource when delete failure domain #13421
    • Fix create the dynamic configuration resource if not exist #13420
    • Fix batch message ack does not decrease the unacked-msg count. #13383
    • Fix dead loop in BacklogQuotaManager.dropBacklogForTimeLimit #13249
    • Modify return result of NamespacesBase#internalGetPublishRate #13237
    • Optimize ManagedLedger Ledger Ownership Check #13222
    • Fix shedding heartbeat ns #13208
    • Fix when deleting topic with NotFoundException, do not return to client. #13203
    • Clean up active consumer on already closed connection #13196
    • Update cursor last active timestamp when reseting cursor #13166
    • Use current resourceUsage value as historyUsage when leader change in ThresholdShedder #13136
    • Don't attempt to delete pending ack store unless transactions are enabled #13041
    • Fix race condition in ResourceLockImpl#revalidate #13006
    • Fix NPE in PersistentTopic.checkSubscriptionTypesEnable #12961
    • Fix wrong isEmpty method of ConcurrentOpenLongPairRangeSet #12953
    • Correct param of delete method for v1 topic #12936
    • Change create topic return error to Status.BAD_REQUEST #12919
    • Fix topic policy listener deleted by mistake. #12904
    • Remove readerCaches and close reader when exception occurs in SystemTopicBasedTopicPoliciesService #12873
    • Fix deleting tenants with active namespaces with 500. #12848
    • Bug Fix: topic policy is not properly init if namespace is loaded first. #12833
    • Add broker config isAllowAutoUpdateSchema #12786
    • Fix TopicPoliciesCacheNotInitException issue. #12773
    • The problem of two exception handling #12744
    • Handle lookup redirect for V1-topics with different cluster #12743
    • Fix race condition in PersistentTopic#addReplicationCluster #12729
    • Remove catching NamingException. #12725
    • Even if always compatible is set, Consumers cannot be created #12721
    • Fix znode leakage caused by deleting tenant #12711
    • Remove unused listeners if it have no listeners. #12654
    • Optimize exception information for schemas #12647
    • Support retry when creating reader of Topic Policies #12622
    • Fix String should use equals but not ==. #12619
    • Fix 12614, waitingForPingResponse needs to be modified with volatile for concurrent sence #12615
    • Cleanup ProxyPublishConsumeTest #12607
    • Fix additional servlets nar might extract to null directory #12585
    • Fix log typo in NamespaceService#checkHeartbeatNamespace #12582
    • Clean up the metadata of the non-persistent partitioned topics. #12550
    • Add git branch information for PulsarVersion #12541
    • Websocket should pass the encryption context to the consumers #12539
    • The count of topics on the bundle is less than 2, skip split #12527
    • Remove redundant null check for getInternalListener #12474
    • Optimize the code: remove extra spaces #12470
    • Fix wrong property name in NamespaceIsolationDataImpl#secondary #12433
    • Fix the null point caused by deleting the system topic policy #12367
    • Future completed twice in the method of impl.MLPendingAckStore#closeAsync #12362
    • Fix the race of delete subscription and delete topic #12240
    • Fix delete authentication policies when delete topic. #12215
    • Fix update ledger list to znode version mismatch failed, ledger not delete #12015
    • Fix issues 11964, deadlock bug when use key_shared mode #11965
    • Fix validateGlobalNamespaceOwnership wrap exception issue. #14269
    • Fix reader skipped remaining compacted data during the topic unloading. #13629
    • Return message ID from compacted ledger while the compaction cursor reach the end of the topic #13533
    • Fix getting the last message-id from an empty compact ledger #13476
    • Fix lost compaction data due to compaction properties missed during reset-cursor #12698
    • Do not move the non-durable cursor position when trimming ledgers while topic with compaction #12602
    • Fix can't read the latest message of the compacted topic #14449
    • Fix skips compacted data for reader/consumer #12464
    • Fix wrong behaviour caused by not cleaning up topic policy service state. #14503
    • Avoid thread deadlock problem when creating topic policy reader #13837
    • Bug Fix: topic policy is not properly init if namespace is loaded first. #13422
    • Fix parse BigDecimal #14019
    • Fix: bug when allAll bucket #13467
    • Support GET_BACKLOG_SIZE topic op after enable auth #12850
    • Reduce the readFailureBackoff time #12444
    • Add readWorkerThreadsThrottlingEnabled to conf/bookkeeper.conf #12666

    Transaction

    • Fix cursor readPosition is bigger than maxPosition in OpReadEntry #14667
    • Fix MLTransactionMetadataStore.update async fail #14532
    • Fix end transaction at state of timeout #14370
    • Optimize testEndTBRecoveringWhenManagerLedgerDisReadable #14303
    • Delete changeMaxReadPositionAndAddAbortTimes when checkIfNoSnapshot #14276
    • Adopt single thread pool in TC #14238
    • Fix send normal message can't change MaxReadPosition #14192
    • Optimize transaction timeout #14172
    • Fix subscription ack transaction marker. #14170
    • Fix OpBase.callback is not called in TransactionMetaStoreHandler#failPendingRequest #14133
    • Fix deadlock on txn semaphore permit exhaustion #14131
    • Fix unack message count for transaction Ack while disabled batch index ack #14071
    • Fix individual ack with transaction decrease unAckMessageCounnt #14020
    • Resolve the performance bottleneck of TransactionBufferHandle #13988
    • RetryException should not be return #13828
    • Fix recover of TransactionBuffer #13739
    • Duplicate TYPE of Prometheus metrics #13704
    • Remove txn add partition info log #13670
    • Fix no response to client when handleSubscribe because PendingAckHandleImpl init fail #13655
    • PendingAckHandleImpl handle isInCacheRequest #13481
    • Stop TB recovering with exception #13425
    • Optimize transaction FieldUpdater to static final #13396
    • Delete the redundant code #13327
    • Allow transaction to be committed or aborted in the state of aborting or committing. #13323
    • Remove request if can not send #13308
    • Fix generate transactionId some comments #13234
    • Fix transaction sequenceId generate error. #13209
    • No TransactionCoordinatorNotFound, but automatic reconnect #13135
    • Add a check for uninitialized PendingAck #13088
    • Handle exception double #12881
    • Stop TP replaying with Exception #12700
    • Stop TB recovering with exception #12636
    • Txn client check timeout #12521
    • Fix maxReadPosition with normal publish #12386
    • Fix bugs, Exception thrower by TB::appendBufferToTxn must be ManagedLedgerException. #12376
    • Merge transactionBuffer exception into a class #12358
    • Add transaction perf #11933
    • Adopt Single_thread to handle TcClient connecting #13969
    • Stop TC replaying with exception #12705
    • Fix performance #13253

    Admin

    • Fix Broker HealthCheck Endpoint Exposes Race Conditions. #14658
    • Fix delete namespace issue. #14657
    • Make sure policies.is_allow_auto_update_schema not null #14409
    • Fix delete namespace issue. #14215
    • Fix NPE when reset-cursor at a non-existent topic (RestException without cause) #13573
    • Add transaction admin #13447
    • Print topic internal info as formatted json #12709
    • Add --all option to get all version schema of topic #12535
    • Modify exception of set-properties for namespace #12436
    • Get schema validation enforce add applied. #12349
    • Add corresponding get command for namespace #12322
    • Pulsar admin exposes secret for source and sink #13059
    • Perfect judgment conditions of pulsar-admin #12315

    Proxy

    • Fix port exhaustion and connection issues in Pulsar Proxy #14078
    • Allow config of IO and acceptor threads in proxy #14054
    • Prevent leak of unreleased lookupRequestSemaphore permits #13812
    • Fix issue when Proxy fails to start and logs about an uncaught exception #13171
    • Add error log when new jetty client exception #12840
    • Remove unnecessary Pulsar Client usage from Pulsar Proxy #13836

    Clients

    • [Java] Fix ConsumerBuilderImpl#subscribeAsync blocks calling thread. #14433
    • [Java] Fix adding message to list potential issue #14377
    • [Java] Fix send to deadLetterTopic not working when reach maxRedeliverCount #14317
    • [Java] Fix time unit mismatch in errMsg when producer send fails. #14299
    • [Java] Fix PersistentAcknowledgmentsGroupingTracker set BitSet issue. #14260
    • [Java] Check getTlsTrustStorePath NPE when user forget to set it. #14253
    • [Java] Fix consume failure when BatchReceivePolicy#maxNumBytes < message size #14139
    • [Java] Use scheduleWithFixedDelay instead of scheduleAtFixedRate for java producer batch timer #14125
    • [Java] Add a default timeout for OAuth2 Metadata Resolver #14056
    • [Java] Fix send chunking message failed when ordering key is set. #13699
    • [Java] Fix send chunking message failed when encryption enabled #13689
    • [Java] Fix Producer semaphore permit release issue #13682
    • [Java] Fix the wrong multi-topic has message available behavior #13634
    • [Java] Use PulsarByteBufAllocator to allocate buffer for chunks #13536
    • [Java] Fix resources leak when create producer failed #13505
    • [Java] Fix semaphore and memory leak when chunks failed to enqueue #13454
    • [Java] Fix invalid setting of enabled ciphers to fix warning from BoringSSL #13435
    • [Java] Fix multi topic reader has message available behavior #13332
    • [Java] Improve consumer listener logic #13273
    • [Java] Fix memory leak when message payload processor is configured #13233
    • [Java] Change the time units from ns to ms #13057
    • [Java] Use sendAsync instead of send when produce message to retry topic. #12946
    • [Java] Avoid IllegalStateException in ClientCnx debug logs #12899
    • [Java] Fix pending queue-size stats for batch messages #12704
    • [Java] Fix invalid firstSentAt in log message when timeout first time #12588
    • [Java] Add conf backoff values #12520
    • [Java] Update producer stats when producer close #12500
    • [Java] Fix a typo in UnAckedMessageTracker #12467
    • [Java] Fix the retry topic's REAL_TOPIC & ORIGIN_MESSAGE_ID property #12451
    • [Java] Change the producer fence error log to debug level #12447
    • [Java] Add log error tracking for semaphore count leak #12410
    • [Java] Fix race condition of OpSendMsgQueue when publishing messages #14231
    • [C++] Fix wrong unit of Access Token Response's expires_in field #14554
    • [C++] Fix thread safety issue for multi topic consumer #14380
    • [C++] Fix pulsar client cpp build fail in gcc-4.8.5 #14053
    • [C++] Fix hasMessageAvailable returns wrong value for last message #13883
    • [C++] Fix in macOS CMake might find error boost-python libs path #13193
    • [C++] Fix libcurl miss auth header when broker return 307 #13112
    • [C++] Define and expose PULSAR_VERSION macro #12769
    • [C++] Fix request timeout for GetLastMessageId doesn't work #12586
    • [C++] Support setting priority for consumers #12526
    • [C++] Fix connection read error logging #12492
    • [C++] Use weak ref to ClientConnection for timeout task #12409
    • [C++] Fix Version.h not found when CMake binary directory is customized #13324
    • [C++] Fix GCC compilation failure caused by warning macro #14402
    • [C++] Fix frequent segmentation fault of Python tests by refactoring ExecutorService #12427
    • [Python] Apply clang-format check for python wrapper #13418
    • [Python] Python schema support custom Avro configurations for Enum type #12642
    • [Python] JsonSchema encoding is not idempotent #12490
    • [Python] Cannot use any values that evaluates to False #12489
    • [Python] Python client support using custom Avro schema definition #12516
    • [WebSocket] Fix ClassCastException when user create MultiTopicReader. #14316
    • [WebSocket] Add debug log for WebSocket. #12458

    CleanUp

    • Remove unnecessary boxing unboxing #12790
    • Optimize if statement #12741
    • Remove unnecessary string operation #12719
    • Remove redundant placeholders #12717
    • Remove unused code in PersistentTopic.java #12715
    • Remove unnecessary powermock annotation #12713
    • Remove useless method "PersistentTopic#getPersistentTopic" #12655
    • Remove unused isTxnMessage #12472
    • Optimize SecurityUtility code flow #12431
    • Remove redundant code #12424

    Doc

    • Fix doc for the wrong default value of maxPendingChunkedMessage #14144
    • Update notice year. #13653
    • Fix java doc for MultipleListenerValidator #12389

    Pulsar SQL

    • Fix PulsarRecordCursor deserialize issue. #14379
    • Support protobuf/timestamp #13287
    • Handle message null schema version in PulsarRecordCursor #12809
    • Support query chunked messages feature in Pulsar SQL #12720
    • Pulsar SQL support query big entry data #12448
    • Fix time field use error #12249

    Dependency

    • Bump netty version to 4.1.74.Final #14257
    • Upgrade commons-cli to 1.5.0 #14094
    • Upgrade Gson version 2.8.6 to 2.8.9 #13610
    • Fix issue of mixed SLF4J library versions in presto libs #13603
    • Remove the unused junit4 dependency in managed ledger #13514
    • Upgrade to BookKeeper 4.14.3 #12760
    • Some dependencies in integration tests scope should be test #12696
    • Upgrade debezium to 1.7.1 #12644
    • Upgrade BK to 4.14.4 and Grpc to 1.42.1 #13714

    Test

    • BacklogQuotaManagerTest#testProducerExceptionAndThenUnblockSizeQuota #14213
    • BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuotaWithEmptyLedger #13724
    • Fix flakiness issue when spying ServerCnx #13608
    • Fix flaky test for BrokerServiceLookupTest.testModularLoadManagerSplitBundle #13159
    • Fix flaky caused by message receive timeout in testBlockBrokerDispatching #12954
    • Fix flaky test in PersistentStreamingDispatcherBlockConsumerTest #12943
    • Fix flaky test in testReplicatorProducerName #12898
    • JavaInstanceTest should be AssertEquals #12836
    • ElasticSearchClientTests tests time out #12694
    • Close Zk database on unit tests #12649
    • Optimize the tests extends from TransactionTestBase #12560
    • Broker resource group test optimize fail msg #12438
    • Fix some tests not enabled in integration tests #12417
    • Fix windows test path probleam #12398
    • Fix ProxyServiceStarterTest flaky tests #12344
    • AdminApiTest.testNamespaceSplitBundleConcurrent #14565
    • Add test to ensure correct zk children cache invalidation #14178
    • Upgrade Mockito to latest stable 3.x version, 3.12.4 #13622
    • Add @Test annotation to test methods #12640

    Build

    • Add suppression for pulsar-package-bookkeeper-storage #13451
    • Suppress false positive check for netty-tcnative-classes #13364

    Tool

    • Fix NPE in cmdTopics #13450
    • Add missed import #13037
    • Support listenerThreads configuration. #12892
    • Improve exception info for invaild time-related option #12828
    • Fix doc for pulsar-admin bookies cmd #12542

    Function

    • Miss runtime set in GoInstanceConfig #13031
    • Remove corrupt visible testing annotation in function workers #12621
    • Pulsar admin exposes secrets for function #12950

    CLI

    • Support auto-recovery service in pulsar cli #12985
    • Enable CLI to publish non-batched messages #12641

    CI

    • Replace deprecated "adopt" OpenJDK distribution with "temurin" in GitHub Actions config #12945

    Connector

    • [ES-Sink] Use topic name as the index name if indexName is not configured #13064
    • [ES-Sink] Correct @FieldDoc defaultValue for some fields #12697
    • Don't create AvroData for each KafkaSourceRecord #12859

    Tiered Storage

    • Add retry to tolerate the offload index file read failure #12452
    • Fix the read performance issue in the offload readAsync #12443
    Source code(tar.gz)
    Source code(zip)

  • v2.8.3(Apr 8, 2022)

    Important Notices

    • Fix detecting number of NICs in EC2 #14252. In the event that Pulsar cannot determine the NIC speed from the host, please set loadBalancerOverrideBrokerNicSpeedGbps.
    • Bump BookKeeper 4.14.3 12906
    • Add broker config isAllowAutoUpdateSchema 12786

    Security

    • Upgrade Postgres driver to 42.2.25 to get rid of CVE-2022-21724 14119
    • Get rid of CVEs in Solr connector 13822
    • Get rid of CVEs in InfluxDB connector 13821
    • Get rid of CVEs in batch-data-generator 13820
    • Get rid of CVEs brought in with aerospike 13819
    • [owasp] suppress false positive Avro CVE-2021-43045 13764
    • Upgrade protobuf to 3.16.1 to address CVE-2021-22569 13695
    • Upgrade Jackson to 2.12.6 13694
    • Upgrade Log4j to 2.17.1 to address CVE-2021-44832 13552
    • Cipher params not work in KeyStoreSSLContext 13322
    • [Broker] Remove tenant permission verification when list partitioned-topic 13138
    • Use JDK default security provider when Conscrypt isn't available 12938
    • [Authorization] Return if namespace policies are read only 12514

    Pulsar Admin

    • Make sure policies.is_allow_auto_update_schema not null 14409
    • pulsar admin exposes secret for source and sink 13059
    • Fix deleting tenants with active namespaces with 500. 13020
    • [function] pulsar admin exposes secrets for function 12950

    Bookkeeper

    • Upgrade BK to 4.14.4 and Grpc to 1.42.1 13714
    • Bump BookKeeper 4.14.3 12906

    Broker

    • Fix the wrong parameter in the log. 14309
    • Fix batch ack count is negative issue. 14288
    • bug fix: IllegalArgumentException: Invalid period 0.0 to calculate rate 14280
    • Clean up individually deleted messages before the mark-delete position 14261
    • If mark-delete operation fails, mark the cursor as "dirty" 14256
    • Fixed detecting number of NICs in EC2 14252
    • Remove log unacked msg. 14246
    • Change broker producer fence log level 14196
    • Fix NPE of cumulative ack mode and incorrect unack message count 14021
    • KeyShared stickyHashRange subscription: prevent stuck subscription in case of consumer restart 14014
    • Trim configuration value string which contains blank prefix or suffix 13984
    • waitingCursors potential heap memory leak 13939
    • Fix read schema compatibility strategy priority 13938
    • NPE when get isAllowAutoUploadSchema 13831
    • Fix call sync method in async rest API for internalGetSubscriptionsForNonPartitionedTopic 13745
    • Fix the deadlock while using zookeeper thread to create ledger 13744
    • Fix inefficient forEach loop 13742
    • [Issue 13640] Fix non persistent topic subscription error. 13685
    • Fix invalid rack name cause bookie join rack failed 13683
    • Avoid call sync method in async rest API for force delete subscription 13668
    • Avoid call sync method in async rest API for delete subscription 13666
    • Fix getInternalStats occasional lack of LeaderInfo 13656
    • Fix reader skipped remaining compacted data during the topic unloading. 13629
    • [Issue 13479] Fixed internal topic effect by InactiveTopicPolicy. 13611
    • Fix bug :Infinity value for CPU or Bandwidth usage 13609
    • Change ContextClassLoader to NarClassLoader in BrokerInterceptor 13589
    • Fix NPE when unloading namespace bundle 13571
    • update log content 13540
    • Return message ID from compacted ledger while the compaction cursor reach the end of the topic 13533
    • Change ContextClassLoader to NarClassLoader in AdditionalServlet 13501
    • fix(Auth): Fix multi roles authz cannot handle empty roles case 13477
    • Fix getting the last message-id from an empty compact ledger 13476
    • Fixes the NPE in system topics policies service 13469
    • Fix race conditions in closing producers and consumers 13428
    • Fix batch message ack does not decrease the unacked-msg count. 13383
    • [Issue 13194][pulsar-broker] Fix dead loop in BacklogQuotaManager.dropBacklogForTimeLimit 13249
    • Modify return result of NamespacesBase#internalGetPublishRate 13237
    • Optimize ManagedLedger Ledger Ownership Check 13222
    • Close old compacted ledger when open new. 13210
    • fix shedding heartbeat ns 13208
    • Fix when deleting topic with NotFoundException, do not return to client. 13203
    • Update cursor last active timestamp when reseting cursor 13166
    • Remove tenant permission verification when list partitioned-topic 13138
    • Use current resourceUsage value as historyUsage when leader change in ThresholdShedder 13136
    • Don't attempt to delete pending ack store unless transactions are enabled 13041
    • Fix NPE in PersistentTopic.checkSubscriptionTypesEnable 12961
    • Fix wrong isEmpty method of ConcurrentOpenLongPairRangeSet 12953
    • Correct param of delete method for v1 topic 12936
    • Clean up the metadata of the non-persistent partitioned topics. 12910
    • Fix topic policy listener deleted by mistake. 12904
    • Fix deleting tenants with active namespaces with 500. 12848
    • [Issue 12757] add broker config isAllowAutoUpdateSchema 12786
    • Remove unused listeners if it have no listeners. 12654
    • Clean up the metadata of the non-persistent partitioned topics. 12550
    • [managedledger] NPE on OpAddEntry while ManagedLedger is closing 12364
    • fix issues 11964, deadlock bug when use key_shared mode 11965

    Build

    • [C++] Fix GCC compilation failure caused by warning macro 14402
    • [C++] Fix Version.h not found when CMake binary directory is customized 13324
    • [Issue 9888] add python3.9 on manylinux2014 build support 10954

    C++ Client

    • Fix GCC compilation failure caused by warning macro 14402
    • Fix pulsar client cpp build fail in gcc-4.8.5 14053
    • Fix hasMessageAvailable returns wrong value for last message 13883
    • Fix Version.h not found when CMake binary directory is customized 13324
    • Fix in macOS CMake might find error boost-python libs path 13193

    CI

    • Upgrade Windows runner os to windows-2022 and generator 14368
    • Replace deprecated "adopt" OpenJDK distribution with "temurin" in GitHub Actions config 12945

    Pulsar CLI

    • Feat(cli): support autorecovery service in pulsar cli 12985
    • [pulsar-perf] Write histogram files for consume command 12569

    Java Client

    • Fix adding message to list potential issue 14377
    • Fix send to deadLetterTopic not working when reach maxRedeliverCount 14317
    • Fix time unit mismatch in errMsg when producer send fails. 14299
    • Fix PersistentAcknowledgmentsGroupingTracker set BitSet issue. 14260
    • Check getTlsTrustStorePath NPE when user forget to set it. 14253
    • [Issue 12262] Fix consume failure when BatchReceivePolicy#maxNumBytes < message size 14139
    • Use scheduleWithFixedDelay instead of scheduleAtFixedRate for java producer batch timer 14125
    • Add a default timeout for OAuth2 Metadata Resolver 14056
    • Fix send chunking message failed when ordering key is set. 13699
    • Fixed Producer semaphore permit release issue 13682
    • Fix the wrong multi-topic has message available behavior 13634
    • Use PulsarByteBufAllocator to allocate buffer for chunks 13536
    • Fix resources leak when create producer failed 13505
    • Fix semaphore and memory leak when chunks failed to enqueue 13454
    • Fix invalid setting of enabled ciphers to fix warning from BoringSSL 13435
    • [Producer] Change the time units from ns to ms 13057
    • Fix consume message order issue when use listener. 13023
    • Use sendAsync instead of send when produce message to retry topic. 12946
    • [Java Client] Avoid IllegalStateException in ClientCnx debug logs 12899
    • [pulsar-client] Add conf backoff values 12520
    • Add log error tracking for semaphore count leak 12410

    Compaction

    • Fix reader skipped remaining compacted data during the topic unloading. 13629
    • Return message ID from compacted ledger while the compaction cursor reach the end of the topic 13533
    • [Broker] Fix getting the last message-id from an empty compact ledger 13476

    Functions

    • Fix(functions): missing runtime set in GoInstanceConfig 13031
    • [function] pulsar admin exposes secrets for function 12950

    Pulsar IO

    • Pulsar admin exposes secret for source and sink 13059
    • Pass client builder if no service url provided to debezium connector 12145
    • [Cherry-pick] Pass client builder to debezium database history 12112
    • Pass client builder if no service url provided to debezium connector 14040

    Key-Shared Subscription

    • [Issue 12885] Fix unordered consuming case in Key_Shared subscription. 12890

    Pulsar Metadata

    • AbstractMetadataStore: invalidate childrenCache correctly when node created 14177

    Metrics

    • Fix: bug when allAll bucket 13467

    Pulsar proxy

    • Fix port exhaustion and connection issues in Pulsar Proxy 14078
    • Allow config of IO and acceptor threads in proxy 14054
    • Remove unnecessary Pulsar Client usage from Pulsar Proxy 13836

    Python Client

    • Apply clang-format check for python wrapper 13418

    Schema

    • Fix parse BigDecimal 14019
    • Fix pulsar use json or avro primitive schema. 12886

    SQL

    • Support protobuf/timestamp 13287
    • Fix time field use error 12249

    Test

    • Improved 9 flaky tests.

    Tools

    • Fix NPE in cmdTopics 13450
    • [pulsar-perf] Support listenerThreads configuration. 12892
    • [docs] Fix doc for pulsar-admin bookies cmd 12542

    Topic policy

    • [Broker] Avoid thread deadlock problem when creating topic policy reader 13837
    • Fixed internal topic effect by InactiveTopicPolicy. 13816

    Transaction

    • Fix topicTransactionBuffer handle null snapshot (#12758) 14510
    • Fix transaction system topic create in loop 12889
    • Fix transaction system topic create in loop. 12749

    Websocket

    • Fix ClassCastException when user create MultiTopicReader. 14316

    Dependency

    • Upgrade Gson version 2.8.6 to 2.8.9 13610
    • Upgrade commons-cli to 1.5.0 14094
    • Bump netty version to 4.1.74.Final 14257
    Source code(tar.gz)
    Source code(zip)

  • v2.8.2(Jan 12, 2022)

    Security

    • Upgrade to Log4J 2.17.0 to mitigate CVE-2021-45105 #13392
    • Upgrade Netty to 4.1.72 - CVE-2021-43797 #13328
    • Bump log4j to 2.15.0 #13226
    • Revert new AuthorizationProvider method #13133
    • Support CLEAR_BACKLOG namespace op after enable auth #12963
    • Upgrade netty to 4.1.68.Final #12218
    • Support disabling non-TLS service ports #11681
    • Upgrade Jetty to 9.4.43.v20210629 #11660

    Broker

    • Fix and improve topic ownership assignment #13069
    • Fix LeaderElectionService.getCurrentLeader and add support for empheralOwner in MockZooKeeper #13066
    • Do not reuse the Failed OpAddEntry object which leads to the bundle unloading timeout. #12993
    • Remove readerCaches and close reader when exception occurs in SystemTopicBasedTopicPoliciesService #12873
    • Fix TopicPoliciesCacheNotInitException issue. #12773
    • Support UNSUBSCRIBE namespace op after enabling auth #12742
    • Fix race condition in PersistentTopic#addReplicationCluster #12729
    • Even if always compatible is set, consumers cannot be created #12721
    • Fix the incorrect total size when BrokerEntryMetadata is enabled #12714
    • Fix lost compaction data due to compaction properties missed during reset-cursor #12698
    • Fix TestRunMain test #12675
    • Support GET_METADATA topic op after enabling auth #12656
    • Fix false positive ownership check in OwnershipCache#checkOwnership #12650
    • Optimize exception information for schemas #12647
    • Add @Test annotation to test methods #12640
    • Support retry when creating reader of Topic Policies #12622
    • Fix String should use equals but not ==. #12619
    • Fix 12614, waitingForPingResponse needs to be modified with volatile for concurrent sence #12615
    • Cleanup ProxyPublishConsumeTest #12607
    • Avoid passing OpAddEntry across a thread boundary in asyncAddEntry #12606
    • Do not move the non-durable cursor position when trimming ledgers while topic with compaction #12602
    • Allow GetTopicsOfNamespace op with consume permission #12600
    • Allow configuring schema compatibility policy for system topics #12598
    • Cleanup already deleted namespace topics. #12597
    • Fix additional servlets NAR might extract to null directory #12585
    • Fix log typo in NamespaceService#checkHeartbeatNamespace #12582
    • Add OpAddEntry to pendingAddEntries after the state check #12570
    • Cancel scheduled tasks when deleting ManagedLedgerImpl #12565
    • Add git branch information for PulsarVersion #12541
    • Websocket should pass the encryption context to consumers #12539
    • The count of topics on the bundle is less than 2，skip split #12527
    • Fix the reader skips compacted data which original ledger been removed #12522
    • Fix messageDedup delete inactive producer name #12493
    • Optimize the code: remove extra spaces #12470
    • Future completed twice in the method of impl.MLPendingAckStore#closeAsync #12362
    • Fix the race of delete subscription and delete topic #12240
    • Disable stats recorder for built-in PulsarClient #12217
    • Fix delete authentication policies when deleting topics. #12215
    • Optimize the memory usage of Cache Eviction #12045
    • Avoid adding duplicated BrokerEntryMetadata #12018
    • Fix update ledger list to znode version mismatch failed, ledger not delete #12015
    • Fix messages in TopicPolicies will never be cleaned up #11928
    • Fix returned wrong hash ranges for the consumer with the same consumer name #12212
    • Add Key_Shared metadata to topic stats #11839
    • Fix build from submodules (broker, transaction coordinator) #11795
    • Add method to clear up transaction buffer snapshot #11934
    • Increase the test stability of transactionTest #11541
    • Add maven.restlet.org repository #13248
    • Fix and improve topic ownership assignment (#13069) #13117
    • Evaluate the current protocol version once #13045
    • Revert "Set default root log level to debug" and make PULSAR_LOG_ROOT_LEVEL default to PULSAR_LOG_LEVEL #12941
    • Catch exceptions in scheduled tasks to prevent unintended cancellation #12853
    • Fix producer getting incorrectly removed from topic's producers map #12846
    • Synchronize updates to the inactiveProducers map in MessageD… #12820
    • Close connection after receiving unexpected SendCommand #12780
    • Fix namespace policy override ignored when creating subscription #12699
    • Update lombok to 1.18.22 #12466
    • Fix skips compacted data for reader/consumer #12464
    • Remove data race in MultiTopicsConsumerImpl to ensure correct message order #12456
    • Fix the retry topic's REAL_TOPIC & ORIGIN_MESSAGE_ID property #12451
    • Change the producer fence error log to debug level #12447
    • Reduce the readFailureBackoff time #12444
    • Fix wrong property name in NamespaceIsolationDataImpl#secondary #12433
    • Optimize SecurityUtility code flow #12431
    • Fix compactor skips data from last compacted Ledger #12429
    • Remove redundant code #12424
    • Fix some tests not enabled in integration tests #12417
    • Use weak ref to ClientConnection for timeout task #12409
    • Fix cherry-pick issue #12397
    • Fix the null point caused by deleting the system topic policy #12367
    • Update delete inactive topic configuration documentation #12350
    • Add log to print exception stack. #12345
    • Avoid potentially blocking calls to metadata on critical threads #12339
    • Fix NPE when removing cursor #12297
    • Print log when configuration is failed to load #12280
    • Fix incorrect returned last message ID while the lastConfirmedEntry with negative entry ID #12277
    • Fix TTL expiry does not take effect #12266
    • The loadbalancer should avoid offload the heartbeat namespace #12252
    • Fix typo of the returned last message ID when the last message ID is from compacted ledger #12237
    • Add support for splitting topics and partition labels in Prometheus #12225
    • Fix lost message issues 12221 #12223
    • Allow to config pulsar client allocator out of memory policy #12200
    • Remove redundant parameters #12188
    • Fix incorrect logger numbers in tests #12168
    • Return the last position of the compacted data while the original data has been deleted #12161
    • Improve exceptions thrown when handling the schema resource #12155
    • Fix prefix setting in JWT authn and avoid multi calls for the getProperty #12132
    • Fix used after recycle issue in OpAddEntry #12103
    • Bugfix: Fix rackaware placement policy init error #12097
    • Fix wrong key-hash selector used for new consumers after all the previous consumers disconnected #12035
    • Fix cherry-pick issue on branch-2.8 #11982
    • Remove unused variable and unnecessary box in NamespaceBundleFactory #11975
    • Print position info when can't find next valid position. #11969
    • Fix NPE ZkBookieRackAffinityMapping #11947
    • Avoid to infinitely split bundle #11937
    • Improved logic for pausing replicated subscription snapshots when no traffic #11922
    • Fix ZKSessionTest.testReacquireLocksAfterSessionLost #11886
    • Schema compatibility strategy in broker level. #11856
    • Use TestRetrySupport for BaseMetadataStoreTests to cleanup state between retries #11771
    • Remove replace_maven-wagon-http-version.sh script #11718
    • Check null or empty instead of catch NPE #11655
    • Avoid duplicate deletion of schema #11640
    • Fix subscribeRateLimiter cannot be disabled #11614
    • Fix race condition in concurrent schema deletion #11606
    • Use get instead of join to avoid getting stuck #11597
    • Avoid to cal getMaximumRolloverTimeMs everytime #11513
    • Fix improper class/method/field modifiers #10837
    • Support max-connection and max-connection-per-IP #10754
    • Allow Integration Tests Jar to be deployed to Maven central #12292

    C++ Client

    • Fix libcurl miss auth header when broker return 307 #13112
    • Define and expose PULSAR_VERSION macro #12769
    • Fix request timeout for GetLastMessageId doesn't work #12586
    • Fix connection read error logging #12492
    • Delay ClientCredentialFlow::initialize to the first authenticate call #12372
    • Use URL encoded content type for OAuth 2.0 authentication #12341
    • Handle OAuth 2.0 exceptional cases gracefully #12335
    • Support configuring optional scope field for OAuth2 authentication #12305
    • Fix the issue of attempting to connect to multiple IP addresses #11958
    • Handle error when shutting down client after forks #11954
    • Move all C symbols into C++ pulsar namespace #11919
    • Make some cleanup methods thread safe #11762
    • Turn on more compiler warnings and enforce warnings as errors #11668
    • Fix use-after-free and constructor bugs in UnAckedMessageTrackerEnabled #11630
    • Allow partitioned producers to start lazily #11570

    Functions

    • K8s runtime: force deletion to avoid hung function worker during connector restart #12504
    • Fix k8s pulsar functions containers not exposing metrics port for scraping #12065
    • Enable protobuf-native schema support for function #11868
    • Pass SubscriptionPosition from FunctionDetails to FunctionConfig / SinkConfig #11831
    • Reorganize the context hierarchy for functions #10631
    • Remove the deprecated API usage in HDFS #12080
    • Stop OffsetStore when stopping the connector #12457
    • Support set subscription position #11990
    • Sync to the latest function proto #11853
    • Fix classloader leaks #12973
    • Add missing dependency #12246
    • ConcurrentHashMap should be used for caching producers #11820
    • Support KEY_BASED batch builder for Java based functions and sources #11706

    Pulsar Admin

    • Print topic internal info as formatted JSON #12709
    • Fix last exit code storage #12581
    • Fix the issue of failing to update partitions of topics #11683
    • Perfect judgment conditions of pulsar-admin #12315
    • Fix log level config for pulsar-admin, pulsar-client and pulsar-perf #12915
    • Modify exception of set-properties for namespace #12436
    • Get schema validation enforce add applied #12349

    Tiered Storage

    • Add retry to tolerate the offload index file read failure #12452
    • Fix the read performance issue in the offload readAsync #12443
    • Fix FileSystemManagedLedgerOffloader can not clean up outdated ledger #12309
    • Fix the potential race condition in the BlobStore readhandler #12123

    Pulsar SQL

    • Handle message null schema version in PulsarRecordCursor #12809
    • Pulsar SQL support query big entry data #12448

    Websocket

    • Fix the batch message ack for WebSocket proxy. #12530
    • Add debug log for WebSocket. #12458
    • Remove System.out.println from ConsumerHandler #11459

    Java Client

    • Remove the uncorrected VisableTesting annotation in pulsar-client #11784
    • Fix packages tool parameter desc #11809
    • Add an optional params scope for pulsar oauth2 client #11931
    • Fix producer data race to get cnx #13176
    • Send CloseProducer on timeout #13161
    • Let producer reconnect for state RegisteringSchema #12781
    • Use epoch to version producer's cnx to prevent early de… #12779
    • Pulsar Client: restore SchemaInfo.builder() API #12673
    • Remove invalid call to Thread.currentThread().interrupt(); #12652
    • Add additional error handling in auto partition update task MultiTopicsConsumerImpl #12620
    • Fix invalid firstSentAt in log message when timeout first time #12588
    • Update producer stats when producer close #12500
    • 'StartMessageId' and 'RollbackDuration' not working in MultiTopicsReader for non-partitioned topics #12308
    • Use failPendingMessages to ensure proper cleanup #12259
    • Auto-recovery after exception like out of direct memory #12170
    • Fix endless receiveAsync loop in MultiTopicsConsumer #12044
    • Make Audience Field Optional in OAuth2 Client Credentials #11988
    • Forget to update memory usage on producer close #11906
    • Fix ConcurrentModificationException in sendAsync #11884
    • Hide option -s and substitute -ss(0) for it #11828
    • Fix seek at batchIndex level receive duplicated messages #11826
    • Reduce redundant FLOW requests for non-durable multi-topics consumer #11802
    • Add close method in the class of NegativeAcksTracker #12469
    • Fix a typo in UnAckedMessageTracker #12467
    • Fix message being ignored when the non-persistent topic reader reconnect. #12348
    • Fix deadLetterPolicy is not working with key shared subscription under partitioned topic #12148
    • Forget to call SendCallback on producer close #11939

    CLI

    • Enable CLI to publish non-batched messages #12641
    • Make it possible to disable poolMessages #12108
    • Add total messages when periodic printing throughput #12084
    • Make it possible to disable poolMessages #12090

    Python Client

    • Python schema support custom Avro configurations for Enum type #12642
    • Fix frequent segmentation fault of Python tests by refactoring ExecutorService #12427
    • Support setting namespace for python schema #12175
    • Force Python CI to use earlier version of Protobuf which supports Python2 #12058
    • py::call_method error without mutating internal state #11840
    • Python client support using custom Avro schema definition #12516
    • Remove python3 from vcpkg.json #12092
    • Provide str operator for BytesSchema #12593
    • Do not sort schema fields by default #12232
    • Make python3 the default python in java-test-image #12130
    • Expose Client.shutdown() method #11955

    Unit Test

    • Broker resource group test optimize fail msg #12438
    • Fix windows test path problem #12398
    • Make AuthenticationTokenTest to run on windows #12329
    • Use correct line separator instead of \n #12143

    Bookkeeper

    • Add readWorkerThreadsThrottlingEnabled to conf/bookkeeper.conf #12666
    • UseV2WireProtocol for bookkeeper autorecovery #12311

    Proxy

    • Reduce the severity of log "refreshing key manager" in KeyManagerProxy #12594
    • Set default HTTP proxy request timeout #11971
    • Set default httpProxyTimeout to 5 minutes #12299
    • Fix Pulsar Proxy to re-use authentication instance #12245
    • Fix NPE in ProxyConnection with no auth data #12111
    • Fix ProxyConnection to check for existence of auth_data field #12057
    Source code(tar.gz)
    Source code(zip)

  • v2.7.4(Dec 27, 2021)

    Security

    • Upgrade log4j to 2.17.0 - CVE-2021-45105 #13392
    • Upgrade commons-compress to 1.21 - CVE-2021-35515, CVE-2021-35516, CVE-2021-35517 and CVE-2021-36090 #11345

    Broker

    • Fix DispatchRateLimiter does not take effect #8611
    • Fix set-publish-rate when using preciseTopicPublishRateLimiterEnable=true #10384
    • Fix retention policy in topic policy not work #11021
    • Fix GetTopicsOfNamespace with binary lookup service not check auth #11172
    • Fix retention of keys in compaction #11287
    • Fix using partitioned topic name to get Policy #11294
    • Fix NPE when unloading persistent partitioned topic #11310
    • Remove RateLimiter constructors and replace with builder usage #11599
    • Add backoff for setting for getting topic policies #11574
    • Fix decode compression managedLedgerInfo data #11569
    • Fix some topic policy operation without backoff #11560
    • Fix data lost when using earliest position to subscribe to a topic #11547
    • Fix the schema deletion when delete topic with delete schema #11501
    • Do not create system topic for heartbeat namespace #11499
    • Compress managed ledger info #11490
    • Add backoff for setting for getting topic policies #11487
    • Solving precise rate limiting does not take effect #11446
    • Fix replicated subscriptions direct memory leak #11396
    • Close the replicator and replication client when deleting cluster #11390
    • Invalidate the read handle after all cursors are consumed #11389
    • Fix NPE when unloading persistent partitioned topic #11386
    • Fix retention of keys in compaction #11384
    • Disable replicate system topic across clusters #11376
    • Parallel Precise Publish Rate Limiting Fix #11372
    • Pulsar Admin List Subscription lists only subscriptions created for Partition-0 when partition specific subscriptions are created #11355
    • Fix inconsistent behavior for Namespace bundles cache #11346
    • Fix update ledger list to znode version mismatch failed, ledger not delete #12015
    • Print position info when can't find next valid position #11969
    • Forbid to read other topic's data in managedLedger layer #11913
    • Fix using partitioned topic name to get topic policies #11897
    • Refine topic level backlog quota policies warning log #11863
    • Producer getting producer busy is removing existing producer from list #11804
    • Handle NPE and memory leak when full key range isn't covered with active consumers #11749
    • Call .release() when discarding entry to prevent direct memory leak #11748
    • Fix the topic in fenced state and can not recover #11737
    • Support disabling non-TLS service ports #11724
    • Fix the bug, can not update topic when the update topicName is contained by an existed topic as a part #11686
    • Fix Pulsar didn't respond error messages when throw InterceptException #11650
    • Avoid redundant calls for getting the offload policies from the offloader #11629
    • Optimize ManagedLedger Ledger Ownership Check #13222
    • Do not reuse the Failed OpAddEntry object which lead bundle unloading timeout #12993
    • Fix producer getting incorrectly removed from topic's producers map #12846
    • Allow to configure schema compatibility policy for system topics #12598
    • Cleanup already deleted namespace topics #12583
    • Add OpAddEntry to pendingAddEntries after the state check #12570
    • Cancel scheduled tasks when deleting ManagedLedgerImpl #12565
    • NPE on OpAddEntry while ManagedLedger is closing #12364
    • Avoid potentially blocking calls to metadata on critical threads #12340
    • Fix used after recycle issue in OpAddEntry #12103
    • Optimize the memory usage of Cache Eviction #12045
    • Fix wrong key-hash selector used for new consumers after all the previous consumers disconnected #12035
    • Only auth_errors should log at error level #9325

    Dependency upgrade

    • Remove Boost::System runtime dependency #9498
    • Upgrade Jetty to 9.4.43.v20210629 #11660
    • Add maven.restlet.org repository #13248
    • Upgrade ZooKeeper version to 3.5.9 #12981
    • Remove pulsar-dashboard from the publish process of images #12534
    • Roll back to using Java 8 for docker images #12357
    • Upgrade TestContainers to 1.15.1 #9120

    Client

    • [Java] Refactor seek to reuse common logic #9670
    • [Java] Process partitioned-topic messages on different listener-threads #10017
    • [Java] Fix Consumer listener does not respect receiver queue size #11455
    • [Java] Add a optional params scope for pulsar oauth2 client #11931
    • [Java] Fix seek at batchIndex level receive duplicated messages #11826
    • [Java] Fix race condition on multi-topic consumer #11764
    • [Java] Remove consumer reference from PulsarClient on subscription failure #11758
    • [Java] Fix block forever bug in Consumer.batchReceive #11691
    • [Java] Send CloseProducer on timeout #13161
    • [Java] Remove invalid call to Thread.currentThread().interrupt() #12652
    • [Python] Fix fields that are ignoring the required key argument #11508
    • [C++] Remove usages of boost::regex #9533
    • [C++] Use sendRequestWithId to add timeout to hasMessageAvailable #11600
    • [Pulsar perf] Fix log level config for pulsar-admin, pulsar-client and pulsar-perf #12915

    Functions and Pulsar IO

    • Fix source stats exposing empty exceptions list #11478
    • Fix function API can not use authdata to check superuser #11418
    • Support setting KEY_BASED batch builder for Pulsar Sinks #11710
    • Detect .nar files and prevent spammy logs on functions boot #12665
    • Remove the deprecated api usage in hdfs #12080

    Tiered Storage

    • Fix the potential race condition in the BlobStore readhandler #12123

    Build

    • Force CMake to find Python2 #9690
    • Use ubuntu 20.04 as docker image base #12017
    • Fix docker image install python3.7-dev problem #11942
    Source code(tar.gz)
    Source code(zip)

  • v2.8.1(Sep 18, 2021)

    Broker

    • Fix the issue of all web threads get stuck when deleting a namespace #11596
    • Improve error logs in BacklogQuotaManager #11469
    • Solve the issue of precise rate limiting does not take effect #11446
    • Fix replicated subscriptions direct memory leak #11396
    • Change ContextClassLoader to NarClassLoader in ProtocolHandler #11276
    • Fix the issue of ledger rollover scheduled tasks were executed before reaching the ledger maximum rollover time #11116
    • Fix publish_time not set error when broker entry metadata enable without AppendBrokerTimestampMetadataInterceptor #11014
    • Fix parseMessageMetadata error cause by not skip broker entry metadata #10968
    • Fix issue where Key_Shared consumers could get stuck #10920
    • Fix throwable exception not thrown #10863
    • Handle multiple topic creation for the same topic-name in broker #10847
    • Add REST API to enable or disable replicated subscriptions #10790
    • Fix issue that message ordering could be broken when redelivering messages on Key_Shared subscription #10762
    • Fix set-publish-rate when using preciseTopicPublishRateLimiterEnable=true #10384
    • Upgrade BookKeeper version to resolve the BouncyCastle issue #11759
    • Fix getPreviousPosition NPE #11621
    • Remove duplicated configuration #11283
    • Source tarball: apply executable file permissions to shell scripts #11858
    • Fix java_test_functions build failed #11829
    • Fix generate javadoc for kafka-connect-adaptor failed #11807
    • Fix unnecessary user interactions when building pulsar-standalone image #11623
    • Do not expose meaningless stats for publisher #11454
    • Add metrics storageLogicalSize for the TopicStats and NamespaceStats #11430
    • Compress managed ledger info #11490
    • Print message metadata when getting message by id #11092
    • Query parameter "negativeAckRedeliveryDelay" should be effective even if DLQ is disabled #11495
    • Fix websocket TLS bug #11243
    • Fix the Pulsar Proxy flaky test (Collector already registered that provides name: jvm_memory_direct_bytes_used) #11900
    • Fix flaky test testReacquireLocksAfterSessionLost #11815
    • Fix flaky test testUpdateDynamicLocalConfiguration #11115
    • Fix flaky test testBrokerRanking #11114
    • Fix flaky test in AdminApiOffloadTest #11028
    • Fix the flaky test in the ManagedLedgerTest #11016
    • Make Metadata ZKSessionTest less Flaky #10955
    • Make MetadataCacheTest reliable. #10877
    • Fix pulsar_standalone docker image build failed #11862
    • Producer getting producer busy is removing existing producer from list #11804
    • Revert PR 11594 to avoid copy data to direct buffer #11792
    • Upgrade aircompressor to 0.20 #11790
    • Fix wrappedBuffer always using the same block of memory #11782
    • Fix com.squareup.okhttp-okhttp-2.7.4.jar unaccounted for in LICENSE bug #11769
    • Handle NPE and memory leak when full key range isn't covered with active consumers #11749
    • Call .release() when discarding entry to prevent direct memory leak #11748
    • Avoid duplicated disconnecting producers after fail to add entry#11741
    • Expose compaction metrics to Prometheus #11739
    • Fix the topic in a fenced state and can not recover#11737
    • Remove subscription when closing Reader on non-persistent topics #11731
    • Fix branch-2.8 cherry-pick issue. #11694
    • KeyShared dispatcher on non-persistent topics was not respecting consumer flow-control #11692
    • Fix the bug, can not update topic when the update topicName is contained by an existed topic as a part #11686
    • If a topic has compaction policies configured, we must ensure the subscription is always pre-created #11672
    • Fix testSetReplicatedSubscriptionStatus run failed #11661
    • Fix Pulsar didn't respond error messages when throw InterceptException #11650
    • Fix license mismatch #11645
    • Remove unnecessary jar name in LICENSE files #11644
    • Fix java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer when enabling topic metadata compression #11594
    • Fix decode compression managedLedgerInfo data #11569
    • Fix data lost when using earliest position to subscribe to a topic #11547
    • Add test for auto-created partitioned system topic #11545
    • Replace orElse with orElseGet to avoid calling too many times. #11542
    • Fix the schema deletion when delete topic with delete schema #11501
    • Add metrics for writing or reading size of cursor #11500
    • Do not create system topic for heartbeat namespace #11499
    • Add additional servlet support to broker #11498
    • Add metrics [AddEntryWithReplicasBytesRate] for namespace #11472
    • Deep copy the tenants to avoid concurrent sort exception #11463
    • Reduce the probability of cache inconsistencies #11423
    • Reduce integration test memory usage in CI #11414
    • Swap getTopicReference(topic) with serviceUnit.includes to reduce calling getTopicReference #11405
    • Invalidate the read handle after all cursors are consumed#11389
    • Parallel Precise Publish Rate Limiting Fix #11372
    • Fix concurrency issues in NarUnpacker #11343
    • Close the replicator and replication client when delete cluster#11342
    • Add multi roles support for authorization #11341
    • Fix NPE when unloading persistent partitioned topic #11310
    • Fixed retention of keys in compaction #11287
    • Fix missing replicator metrics #11264
    • Simplify managedLedger retention trim logic #11255
    • Fix retention size policy delete too much ledgers #11242
    • Fix init WebSocketService with ClusterData #11234
    • Make the compaction phase one loop timeout configurable #11206
    • Fixed using CommandSubscribe.getConsumerName() without checking #11199
    • Fix some typos of the PersistentTopics #11187
    • Fix failing auth test. #11186
    • Fix the deadlock when using hasMessageAvailableAsync and readNextAsync #11183
    • Fix compaction entry read exception #11175
    • Set -Dio.netty.tryReflectionSetAccessible=true for pulsar processes #11138
    • Fix broker dispatch byte rate limiter. #11135
    • Change test group to broker for ReplicatorTest and fix the test #11134
    • Fix subscription permission not working in reset cursor #11132
    • Fix Flaky-test: [TopicFromMessageTest].[testMultiTopicConsumerBatchShortName #11125
    • Fix the timestamp description for resetCursor #11121
    • Fix MsgDropRate missing from NonPersistentTopics stats output. #11119
    • Fix GetListInBundle return all Topics in bundle #11110
    • Added missing configuration entries #11095
    • Fix inputs to return a list of topic #11094
    • Add authoritative flag for topic policy to avoid redirect loop #11051
    • Made the PulsarClusterMetadataTeardown deletes idempotent #11042
    • Fix flaky test testEnableAndDisableTopicDelayedDelivery #11009
    • Do not expose meaningless stats for consumers#11005
    • Fix NoClassDefFoundError - io.airlift.compress.lz4.UnsafeUtil #10983
    • Fix direct memory leak in getLastMessageId #10977
    • Fix the backlog issue with --precise-backlog=true #10966
    • Fix create partitioned topic in replicated namespace #10963
    • Fix incorrect port of advertisedListener #10961
    • Fix NonRecoverableLedgerException when get last message ID by Reader #10957
    • Fix compaction not working for system topic #10941
    • Fix peek message failure when broker entry metadata is enabled #10924
    • Fix the unit tests for the websocket and run tests under websocket group #10921
    • When the Replicator is enabled, no managedLedger is created when updating the number of partitions #10910
    • Correct code example in transaction doc #10901
    • When topic does not exist, optimize the prompt message #10845

    Topic Policy

    • Refine topic level backlog quota policies warning log #11863
    • Avoid redundant calls for getting the offload policies from the offloader #11629
    • Fix some topic policy operation without backoff #11560
    • Add backoff for setting for getting topic policies#11487
    • Disable replicating system topic across clusters#11376
    • When deleting a topic, delete the topic policy together#11316
    • Fix using partitioned topic name to get Policy #11294
    • Fix replay topic policy message not work #11136
    • Fix race condition of the SystemTopicBasedTopicPoliciesService #11097
    • Fix retention policy in topic policy not work #11021
    • Fix potential data lost on the system topic when topic compaction have not triggered yet #11003
    • Make getTopicPoliciesAsyncWithRetry as a default method #11518

    Proxy

    • Fixed Proxy leaking outbound connections #11848

    Functions

    • Support protobuf schema for Pulsar function #11709
    • Fix cast exception occurs if function/source/sink type is ByteBuffer #11611
    • Fix source stats exposing empty exceptions list #11478
    • Set exposePulsarAdmin to true if enabled #11417
    • Add instanceId and fqn into log message properties #11399
    • Fix tls_validate_hostname is not supported in python functions runtime #11087
    • Use the subscription name defined in function details #11076
    • Fix build failure because of spotbugs #10792
    • Use keyword argument to create pulsar_client #11080

    Java Client

    • Fixed accessing MessageImpl after it was enqueued on user queue #11824
    • Remove consumer reference from PulsarClient on subscription failure #11758
    • Clean up MultiTopicsConsumerImpl reference on consumer creation failure #11754
    • Fix null MessageId may be passed to its compareTo() method #11607
    • Fix Consumer listener does not respect receiver queue size #11455
    • Avoid infinite waiting for consumer close #11347
    • Fix non-persistent topic get partitioned metadata error on discovery #10806
    • Add AvroSchema UUID support fix #10428
    • Handle receiveAsync() failures in MultiTopicsConsumer #11843
    • Fix bin/pulsar-client produce not supporting v2 topic name through websocket #11069
    • Fixed race condition on multi-topic consumer #11764
    • Forget to update memory usage on message send timeout #11761
    • Fixed block forever bug in Consumer.batchReceive #11691
    • Fix add listenerName for geo-replicator #10779

    C++ Client

    • Use same regex code at ZTSClient #11323
    • Use sendRequestWithId to add timeout to hasMessageAvailable #11600
    • Fix bugs that were not exposed by broken C++ CI before #11557
    • Simple implementation of getting number of references from C++ client #11535
    • Fix brew error in site docs to compile C++ client. #11512
    • Support Windows Debug mode build #11302
    • Fix missed header for some compilers #11152
    • Fix boost download link in Dockerfile #11129
    • Fix Setting KeySharedMeta in SubscribeCommand #11088
    • Fix Windows 32 bits compile and runtime failures #11082
    • Add connection timeout configuration #11029
    • Fix Windows build issues about static library #10956
    • Fix incorrect connect timeout implementation #11889
    • Fix CPP build not failing when tests fail #11575
    • Avoid throwing exceptions when setting socket option #11329

    Python Client

    • Fix redefined Record or Enum in Python schema #11595
    • Fix Python doc generate #11585
    • Fix Python schema array map with record #11530
    • Fixed import when AvroSchema is not being used #11034
    • Fix deadlock caused by ExecutorService::close #11882
    • Fixed crash when using Python logger #10981
    • Ensure producer is keeping the client object alive #11887
    • Fix fields that are ignoring the required key argument #11508
    • Fix handle complex schema #11400
    • Check if the record is not None #11559

    Security

    • Upgrade commons-compress to 1.21 #11345
    • Fix GetTopicsOfNamespace with binary lookup service not check auth #11172
    • Use ubuntu:20.04 base image for Pulsar docker images #11026
    • Upgrade vertx to 3.9.8 to address CVE-2019-17640 #10889
    • Exclude and remove freebuilder dependency #10869
    • Upgrade bouncycastle version to 1.69 #10867
    • Upgrade K8s client-java to 12.0.1 #10866
    • Upgrade caffeine to 2.9.1 #10865
    • Upgrade commons-codec to 1.15 #10864
    • Load credentials from secrets for Kinesis connectors #10822
    • Forbid to read other topic's data in managedLedger layer #11912
    • Bump Netty version to 4.1.66.Final #11344

    Transaction

    • Pending ack set managed ledger config true #11494
    • Add getTxnID method in Transaction.java #11438
    • Fix direct memory leak related to commit and abort markers #11407
    • Fix transaction buffer client handle endTxn op when topic or sub have been deleted#11304
    • Fix the transaction markers that are not deleted as expected#11126
    • Fix delete sub then delete pending ack#11023
    • Prevent NPE in case of closeAsync() without a successful execution of startAsync() #10948
    • Fixed possible deadlock in the initialization of MLTransactionLog #11194
    • Fix broker init transaction related topic. #11022

    Pulsar Admin

    • Fix pulsar admin method:getMessageById#11852
    • Allow create functions with package URL #11666
    • Add compacted topic metrics for TopicStats in CLI #11564
    • Fix time based backlog quota. #11509
    • Add offload ledger info for admin topics stats #11465
    • Add complete metadata for admin.topics().examineMessages #11443
    • Remove duplicate check for replicationClusterSet #11429
    • Pulsar Admin List Subscription lists only subscriptions created for Partition-0 when partition specific subscriptions are created #11355
    • Expose broker entry metadata and deliverAtTime to peekMessages/getMessages #11279
    • Allow null to be set as namespace level subscription TTL #11253
    • Enable peeking encrypted batch messages #11244
    • Fix async response filter #11052
    • Add error log for schema admin operation #11427

    Tiered Storage

    • OffloadPoliciesImplBuilder missing method and not implements OffloadPolicies.Builder #11453
    • Remove unused listener to reduce creating executor pool #11215

    Pulsar IO

    • Make KafkaSourceRecord ack() non-blocking to avoid deadlock #11435
    • Allow Sinks to use native AVRO and JSON #11322
    • Refine the key in redis sink when key is null #11192
    • Change the nar package name for pulsar-io-kafka-connect-adaptor #10976
    Source code(tar.gz)
    Source code(zip)

  • v2.7.3(Aug 11, 2021)

    Broker

    • Fix Delayed Messages. #11374
    • Fix missing replicator metrics. #11264
    • Fix branch-2.7 test. #11254
    • Fix broker dispatch byte rate limiter #11249
    • Fix websocket TLS bug #11243
    • Fix init WebSocketService with ClusterData #11234
    • Fix ledger roll over scheduler task #11226
    • Remove unused listener to reduce creating executor pool #11215
    • Make the compaction phase one loop timeout configurable #11206
    • Fix failing auth test. #11186
    • Fix the dead lock when using hasMessageAvailableAsync and readNextAsync #11183
    • Fix compaction entry read exception #11175
    • On multi-topic consumer, we shouldn't keep checking the partitioned metadata #11168
    • Fix replay topic policy message not work #11136
    • Fix broker dispatch byte rate limiter. #11135
    • Change test group to broker for ReplicatorTest and fix the test #11134
    • Fix ledger rollover scheduled task #11116
    • Fix race condition of the SystemTopicBasedTopicPoliciesService #11097
    • Print message metadata when getting message by id #11092
    • Fix flaky test testEnableAndDisableTopicDelayedDelivery #11009
    • Fix potential data lost on the system topic when topic compaction has not triggered yet #11003
    • Fix direct memory leak in getLastMessageId #10977
    • Fix the backlog issue with --precise-backlog=true #10966
    • Fix NonRecoverableLedgerException when get last message ID by Reader #10957
    • Fix compaction not working for system topic #10941
    • Fix issue where Key_Shared consumers could get stuck #10920
    • When the Replicator is enabled, no managedLedger is created when updating the number of partitions #10910
    • Handle multiple topic creation for the same topic-name in broker #10847
    • Release OpAddEntry.data when entry is copied and discarded #10773
    • Fix issue that message ordering could be broken when redelivering messages on Key_Shared subscription #10762
    • Fix solution for preventing race conditions between timeout and completion #10740
    • Cancel scheduled tasks as the first step in closing #10739
    • MINOR: Add error message to setMaxPendingMessagesAcrossPartitions #10709
    • Make PrometheusMetricsTest. testAuthMetrics pass on CI #10699
    • Fix issue where StackOverflowError occurs when trying to redeliver a large number of already acked messages #10696
    • Revert "Creating a topic does not wait for creating cursor of replicators" #10674
    • Use single instance of parser #10664
    • Ensure all the ReadHandle gets properly closed on cache invalidation #10659
    • Fix ConcurrentOpenLongPairRangeSet remove all ranges #10656
    • TopicPoliciesTest.testMaxSubscriptionsFailFast fails #10640
    • Add metrics for non-contiguous deleted messages range #10638
    • Fixed missed ZK caching when fetching a list of namespaces for a tenant #10594
    • Made OpAddEntry.toString() more robust to nulls to prevent NPEs #10548
    • Fix partitioned system topic check bug #10529
    • Make failPendingMessages called from within the ProducerImpl object mutex #10528
    • Fix deadlock on Monitoring thread blocked by LeaderService.isLeader() #10512
    • Fix: Topic loading fails without any error when replicator init fails #10432
    • Fix hasMessageAvailable return true but can't read message #10414
    • Added more unit tests to the JavaInstanceTest class #10369
    • Fix authorization error if partition number of partitioned topic is updated. #10333
    • Fix the inconsistency of AdvertisedAddress #10312
    • Fix missing LoggerFactoryPtr type. #10164
    • Ensure read-lock is not continuously held on a section while iterating over concurrent maps #9787
    • Zookeeper connections monitor data #9778
    • Change getWorkerService method to throw UnsupportedOperationException #9738
    • Fix flaky unit test #9262
    • Supply debug log for OpAddEntry #9239

    Dependency upgrade

    • Upgrade Jetty to 9.4.42.v20210604 #10907

    Proxy

    • Enable AutoTopicCreationType partitioned through proxy #8048

    Pulsar Admin

    • Fix create partitioned topic in replicated namespace #11140
    • Add authoritative flag for topic policy to avoid redirect loop #11131
    • Fix non-persistent topic get partitioned metadata error on discovery #10806
    • Fix kinesis sink backoff class not found #10744

    Docker

    • K8s Function Name Length Check Allows Invalid StatefulSet #10531

    Client

    • [Java] Cleaned some code in GenericJsonRecord #10527
    • [C++] Avoid sending flow requests with zero permits #10506

    Functions and Pulsar IO

    • Fix kinesis sink connector does not ack messages #10769
    • Remove reference to ProducerSpec from Pulsar Functions GO #10635
    • Process async results in the same Java runnable thread #10618
    Source code(tar.gz)
    Source code(zip)

  • v2.8.0(Jun 15, 2021)

    PIPs

    Transactions

    • Support pending ack state persistence #8881
    • Stable position and low watermark for the transaction buffer #9195
    • Transaction timeout support #9229
    • Transaction coordinator retry to complete the transaction #9236
    • Fix race condition when appending transaction log #9238
    • Transaction buffer snapshot #9490
    • Add metrics for transaction coordinator #9706
    • Clean the useless transaction individual acknowledgements based on low watermark #9722
    • Fix memory leak when deleting transaction marker #9751
    • Check the transaction state at the client side #9776
    • Clean aborted transactions for the transaction buffer #9974
    • Fix transaction coordinator retry to end transaction #10131
    • Fix NPE when opening a new transaction #10139
    • Fix transaction log failed to recover #10146
    • Fix transaction coordinator recover timeout #10162
    • Handling committing and aborting state when recovering transaction coordinator #10179
    • Fix NPE in case of enableTransactionCoordinator=false #10182
    • Fix transaction buffer client timeout #10206
    • Fix recover max local id issue for the transaction coordinator #10224
    • Support using transactions on standalone #10238
    • Fix transaction buffer lookup issue #10257
    • Fix transaction timeout issue at the client side #10323
    • Fix transaction client reconnect issue after transaction coordinator unloaded #10327
    • Fix transaction timeout not canceled after expired #10366
    • Transaction log low watermark optimization #10422
    • Fix the issue of transaction buffer client channel is inactive #10407
    • Add Admin API for getting transaction coordinator stats #10639
    • Add Admin API for getting transaction in transaction buffer stats #10642
    • Add Admin API for getting transaction in pending ack stats #10648
    • Add Admin API for getting transaction buffer stats and pending ack stats #10650
    • Add Admin API for getting transaction coordinator internal stats #10653
    • Setup transaction metadata with metadata store #10677
    • Fix issue with acknowledge messages from multiple subscriptions of a topic #10689
    • Admin API for getting transaction metadata #10690
    • Admin API for getting slow transactions #10701
    • Fix transaction log handle managed ledger WriteFail state #10711
    • Admin API for getting pending ack internal stats #10725
    • Fix transaction ack delete marker position when don't have transaction ack #10741
    • Fix transaction pending ack generate managedLedgerStats fail exception #10797
    • Use Zookeeper Prometheus metric provider to export Zookeeper metrics #10803
    • Always allow system topic for transaction buffer snapshot auto create #10876

    Security

    • Optional auth method name header in HTTP authentication #6799
    • Topics level authorization support #7523
    • Authorization for setting topic/subscription auto-creation and subscription expire time on namespace #7900
    • Allow serializable stream-provider field into AuthenticationTls #10020
    • Add configuration for running OWASP Dependency Check for all modules #10288
    • Update default TLS protocols to TLSv1.3 and TLSv1.2 for the broker and proxy #10598

    Broker

    • Share EventLoopGroup between broker and BookKeeper client #2603
    • Dispatch batch messages according to consumer permits 7266
    • Improve the max pending publish buffer mechanism 7406
    • Allow disabling HTTP TRACE/TRACK verbs #7907
    • Fix DispatchRateLimiter does not take effect #8611
    • Validate namespace isolation policy regex before updating #8804
    • Perform automatically cert refresh for pulsar-admin #8831
    • Fix updating rack info dynamically #8844
    • Fix deadlock when checking message expiration #8877
    • Support limit max topics per namespace #8942
    • Make ledger rollover check task internally #8946
    • Clean up topic that failed to unload from the cache #8968
    • Support get broker response for the message acknowledgement #8996
    • Support message index for a topic #9039
    • Supporting limit max topics per namespace by namespace policy #9042
    • Streaming dipsatcher support #9056
    • Make Netty acceptor threadPool size configurable #9061
    • Fix deadlock when unloading namespace bundles #9116
    • Fixed checking for maxTopicsPerNamespace #9121
    • Change the service URL to a not required param when creating a cluster #9127
    • Support setting replicator dispatch rate policy at the topic level #9175
    • Fix max topic for namespace does not work #9193
    • Fix memory leak of the managed ledger interceptor #9194
    • Fix max consumers per topic cannot be disabled at the namespace level #9214
    • Support schema REST API for V1 topics #9218
    • Fix peek message metadata while enabling broker entry metadata #9255
    • Support fetching metadata from entry data in publish callback #9257
    • Wait for the async broker port listener close operations to complete at shutdown #9308
    • Support script based mapping network topology #9363
    • Make managed ledger storage configurable #9397
    • Support setting enabled subscription types #9401
    • Fixed NPE and cache invalidation in leader election #9460
    • Fix exception when get an optional field for Protobuf message #9468
    • Ignore replicated subscription configurations from the client when disabled by broker #9523
    • Support expose producer metrics through Prometheus endpoint #9541
    • Fix NPE that occurs in PersistentStickyKeyDispatcherMultipleConsumers when debug log enabled #9587
    • Make LocalPolicies immutable to avoid concurrent modify inconsistent #9598
    • Fix writing/encoding issue of GenericJsonRecord #9608
    • Expose the native record for struct schema #9614
    • Add metrics for producer throttling #9649
    • Fix MaxUnackedMessagesPerConsumer cannot be changed dynamically and cannot be disabled #9657
    • Added support for force deleting tenant #9677
    • Fix managed ledger not found exception when force delete namespace #9691
    • Reduce CPU consumption of metrics creation #9735
    • Ensure read-lock is not continuously held on a section while iterating over concurrent maps #9787
    • Add pending read subscription metrics to stats-internal #9788
    • Allow broker to discover and unblock stuck subscription #9789
    • Disabled the tenants/namespaces force deletion by default #9819
    • Add metrics for the connections of the broker #9876
    • Make readModifyUpdate in MetadataCacheImpl thread-safe #9900
    • Optimize NamespaceBundle.toString() which is on the hot path #9976
    • Support set compaction threshold in broker.conf #9989
    • Support set properties for a namespace #10015
    • Fix cannot cleanup expired data after managed-ledger restart #10087
    • [Default configuration] Enable sticky read by default #10090
    • Add time based backlog quota #10093
    • Fix IllegalStateException in PersistentReplicator #10098
    • Support set max consumers per subscription for non-persistent dispatcher #10121
    • Limit the number of producers/consumers that can connect per topic for each IP address #10188
    • Close namespace clients when PulsarService is closed #10196
    • Shutdown Broker gracefully, but forcefully after brokerShutdownTimeoutMs #10199
    • Update the authentication data when an authentication refresh happens #10203
    • Add preciseTopicPublishRateLimiterEnable to broker.conf #10216
    • Fix the typo in the chunkedMessageRate stats and metrics #10223
    • Fix entry cache size to be a negative value #10225
    • Fix replicated subscriptions related LightProto issues #10247
    • Pause the replicated subscriptions snapshots when there is no real traffic #10292
    • Fix the inconsistency of advertised address #10312
    • Support listenerName for HttpLookupService #10319
    • Support truncate topic #10326
    • Fix authorization error if partition number of partitioned topic is updated 10333
    • Fix managed ledger name that transaction log used #10334
    • Catch topic policy not hit exception in handleSubscribe #10341
    • Fix ConcurrentModificationException when attempting to update local broker data #10347
    • Support seek to separate messageId/time for multiple topic reader #10348
    • Resource locks should automatically revalidate after a metadata session is re-established #10351
    • Add authentication data for the remote cluster #10357
    • Support array type claims in JWT #10375
    • Optimize locks in AuthenticationAthenz #10381
    • Prevent carrying state of PositionImplRecyclable when recycled #10404
    • Dispatch messages to consumer with permits #10417
    • Fix NPE in unblock stuck subscription task when dispatcher has not created #10430
    • Fix topic loading fails without any error when replicator init fails #10432
    • Set timeout to unblock web-threads on update partition API #10447
    • Fix CPU 100% when deleting namespace #10454
    • Remove residual info after forcibly deleting the namespace #10465
    • Fix potential memory leak of TopicPolicies #10466
    • Fix publish callback's entry data is null during ledger rollover #10467
    • Fix readModifyUpdate should return the new value in the future for metadata cache #10474
    • Fix partitioned system topic check issue #10529
    • Removed AspectJ based metrics for ZooKeeper #10533
    • Allow creating MetadataCache with custom serde #10543
    • Fix ack receipt version check issue #10551
    • Expose average message size metrics for a topic #10553
    • Fixed missed ZK caching when fetching list of namespaces for a tenant #10594
    • Setup pulsar cluster with MetadataStore #10600
    • Setup initial namespaces with MetadataStore #10612
    • Convert bundle split into an HTTP async operation #10619
    • Add metrics for non-contiguous deleted messages range #10638
    • Fix consumer related topic stats only available while consumer or reader are connected #10644
    • Allow configuring the number of BK client worker threads #10649
    • Fix ConcurrentOpenLongPairRangeSet remove all ranges #10656
    • Ensure all the ReadHandle gets properly closed on cache invalidation #10659
    • Avoid the context switch when managedLedgerNewEntriesCheckDelayInMillis=0 #10660
    • Allow configuring busy-wait in broker and client #10661
    • Use a single JWT parser instance #10664
    • Fix issues in advanceNonDurableCursors #10667
    • LockManager use a concurrent hash map to handle locks notifications #10680
    • Avoid making copies of internal maps when iterating #10691
    • Fix issue where StackOverflowError occurs when trying to redeliver a large number of already acked messages #10696
    • Fix NPE when filtering read entries #10704
    • Fix deadlock when enabling managed ledger interceptor #10706
    • Fix inconsistent behavior in LongPairRangeSet #10713
    • Fix repeated iterator generation of LongPairRangeSet #10722
    • Cancel scheduled tasks as the first step in closing for the managed ledger #10739
    • Prevent race conditions between timeout and completion of managed ledger #10740
    • Add advertised listener name for geo-replicator #10751
    • Add a read lock when traversing batchDeletedIndexes #10763
    • Replace use of Prometheus client CollectorRegistry.getSampleValue #10764
    • Release OpAddEntry.data when entry is copied and discarded for the managed ledger #10773
    • Avoid warning logs on topic policies not present #10785
    • Remove pulsar-zookeeper module and call zookeeper quorumPeerMain to start zookeeper #10823
    • Fix consumer stuck issue due to reuse entry wrapper #10824
    • Fix possible data race in getFirstAvailableConsumerPermits #10831

    Functions

    • Update default function log location in function_worker.yaml #8470
    • Optimize batch source discovery and task ack #8498
    • Provide an interface for functions worker service #8560
    • Improve naming of threads used in batch source #8608
    • Fix the reader used to read the metadata topic during worker service initialization not been closed #8637
    • Add timeout to hasMessageAvailable to leader election process #8687
    • Support key based batch builder for Go Functions #8761
    • Fix panic when discarding message for Go Functions #8776
    • Move initialize dlog namespace metadata to bin/pulsar #8781
    • Support URL fetching for Go/Python Functions #8808
    • Add cmd flag for retaining key ordering #8886
    • Supports to use package command #8973
    • Support create and update sink with package name #8987
    • Support create and update source with package name #8988
    • Allow stats operations not to be blocked in functions #9005
    • Adding timeout to open table call for function state #9006
    • Fix get function info error of REST API #9115
    • Improve the error message when creating sinks #9131
    • Expose Pulsar Admin through Function context #9246
    • Enable Function Workers to use exclusive producer to write to internal topics #9275
    • Add additional logging when setting up state table for function instance #9304
    • Allow memory limit to be set for the pulsar client used in the ThreadRuntime in Pulsar Function #9320
    • Make admin operations on Statestore non blocking #9348
    • Fix maxPendingAsyncRequests not affect Kubernetes Runtime #9349
    • Add download directory support to function Kubernetes Runtime #9377
    • Optimize built-in source/sink startup by eliminating redundant NAR unpacking and checksum calculation #9413 #9500
    • Enhance Kubernetes manifest customizer with default options #9445
    • Fix possible deadlock on broker-function service startup #9499
    • Close InputStreams properly #9568
    • Add maximum allowed amount of resources setting for functions #9584
    • Support writing general records to Pulsar sink #9590
    • Move metrics port configuration to InstanceConfig #9610
    • Add resource granularity settings for functions #9736
    • Prevent invalid broker or proxy configuration for authorization #9746
    • Log stacktraces of threads that failed to terminate on shutdown within timeout in ExecutorProvider #9840
    • Support get partition index for a Record #9947
    • Deleting a Pulsar Function with a name that includes a colon character crashes the pulsar broker #9946
    • Exposing Prometheus metrics for Pulsar function local run mode #10156
    • Expose local run config metrics-port-start to CLI #10185
    • Fix Pulsar Function localrun with multiple instances and metrics server is enabled #10208
    • Improve localrun performance by using JVM options #10273
    • Fix function API can not use authdata to check superuser #10364
    • Fix potential bug getting stats and remove unnecessary error log #10500
    • Fix deadlock on Monitoring thread blocked by LeaderService.isLeader() #10502
    • Allow user to set custom configs to plugin worker service #10504
    • K8s Function Name Length Check Allows Invalid StatefulSet #10531
    • Get function cluster from broker config when start function worker with broker #10552
    • Process async results in the same Java runnable thread #10618
    • Support using AutoProduceBytesSchema as the function output schema #10716
    • ReflectionUtils use Class.forName in order to properly discover classes in Functions Runtime while using DefaultImplementation #10827
    • Fix the out of index issue when dispatch messages based on the avgBatchSizePerMsg #10828

    IO Connectors

    • [Kafka Source Connector] Fix invalid topic name generation #9035
    • [Kafka Source Connector] Allow managing Avro encoded messages #9448
    • Configure Pulsar IO connector YAML file generator for all connectors #9629
    • [Kafka Source Connector] KeyValue schema support for KafkaBytesSource #10002
    • Allow using GenericObject for developing a Sink connector #10034
    • Sink unwrap internal AutoConsumeSchema and allow to handle topics with KeyValue schema #10211
    • Using ObjectMapper instead of Gson to parse Source/SInk configuration #10441
    • Expose subscription type in the SinkContext #10446
    • Add the ability to seek/pause/resume for a topic to the SinkContext #10498
    • Use Message.getReaderSchema() in Pulsar IO Sinks when possible #10557
    • [Kinesis Sink Connector] Fix backoff class not found #10744
    • [Kinesis Sink Connector] Fix does not acknowledge messages #10769
    • [Kafka Sink Adaptor] Support non-primitive schemas #10410

    Pulsar SQL

    • Fix the misleading setting in presto configuration #8549
    • Fix injection factory cast error #9472
    • Add max split entry queue size bytes limitation #9628
    • Fix Pulsar SQL query bytes schema data error #9631
    • Fix Pulsar SQL query compression data #9663
    • Support native Protobuf decoder #9841
    • Support query uppercase topic name #9980
    • Only return persistent topic when list tables #10368
    • Fix Presto startup on JDK11 #10559
    • Fix pulsar sql issues when run select count(*) for the table with primary schema #10840

    Tiered Storage

    • Fix AWS credentials usages #8950
    • Support ALI OSS tiered storage #8985

    Proxy

    • Fix the proxy does not support partitioned topic auto-creation type #7903
    • Enable Conscrypt for Jetty in the Broker and in the Proxy #10541

    Java Client

    • Fix warn log on the producer side when duplicated messages have been dropped #8729
    • Allow setting subscription name for Reader #8801
    • Fix authParams showing in log with secret string(*****) #8910
    • Avoid enabling DLQ on Key_Shared subscription #9163
    • Add support for the JSON format token #9313
    • Expose reached end of topic API for Reader/Consumer #9381
    • Fix returned a completed future when acknowledging a batch message before complete the individual acknowledgments #9383
    • Fix NPE when strip checksum for the producer #9400
    • Fix inconsistent equals and hashCode for MessageIds #9440
    • Allow disabling producer max queue size #9650
    • Include pulsar-client-admin-api in the shaded version of pulsar-client-admin #9689
    • Fix NPE in the ClientCnx #9761
    • Fix DLQ can't work with AUTO_CONSUME schema #9935
    • Fix NPE caused by null value of SchemaInfo's properties #9985
    • Support multi-topic reader #9995
    • Process messages from different partitions on different listener threads #10017
    • Ensure close resource to avoid memory leak #10028
    • Support set start message ID for each topic/partition on Reader #10033
    • Add a JSON RecordBuilder to the GenericJsonSchema #10052
    • Allow GenericRecord to wrap any Java Object #10057
    • Fix NPE while configuring consumer builder #10063
    • Support get native schema #10076
    • Support KeyValue on Message.getValue() when using AutoConsumeSchema #10107
    • Change the default retry topic name and dead letter topic name #10129
    • Fix KeyValue with SEPARATED encoding for the GenericObject #10186
    • Ensure download schema before decoding the payload for the AUTO_CONSUME schema #10248
    • Fix typo of the maxPendingChunkedMessage method #10223
    • Trait NONE schema as BYTE schema for AUTO_CONSUME schema #10277
    • Fix pause consume issue with MultiTopicsConsumerImpl #10305
    • Make message consumption thread safe and lock-free #10352
    • Reset state before recycling OpSendMsg instance #10405
    • Fix hasMessageAvailable return true but can't read message #10414
    • Fix NPE in GenericJsonRecord #10482
    • Fix behaviour of Schema.AUTO_CONSUME() with KeyValueSchema and multi versions #10492
    • Avoid sending flow requests with zero permits #10507
    • Make failPendingMessages called from within the ProducerImpl object mutex #10528
    • Add schemaType field in SchemaHash #10573
    • Fix NPE when ACK grouping tracker checks duplicated message id #10586
    • Support consume multiple schema types messages by AutoConsumeSchema #10604
    • Fixed issues in pulsar-client shading configuration #10614
    • MessageCrypto interface should not expose Netty ByteBuf class in the API #10616
    • Added org.apache.bookkeeper:cpu-affinity to shaded profile #10681
    • Skip the periodic re-check of the partitions count on non-partitioned topic #10708
    • Unlock the write lock of the UnAckedMessageTracker before call redeliverUnacknowledgedMessages #10768
    • Fix AutoConsumeSchema decode data without schema version #10811

    C++ Client

    • Fix dangling reference bug in getRandomName #8596
    • Optimize batch message buffer allocation #8749
    • Make pool connections configurable in perf tools #8913
    • Support setting listener name #9119
    • Fix batch message handling of the UnAckedMessageTracker #9170
    • Fix ServerError is not converted to string in log #9277
    • Remove Boost::System runtime dependency #9498
    • Removed usages of boost::regex #9533
    • Account for different variables names on different CMake versions #9559
    • Allow to disable static or dynamic lib at build time #9570
    • Avoid multiple compilations of same source files #9675
    • Support configure debug level logs simply #10031
    • Add /opt/homebrew/ as a possible path for OpenSSL on Mac #10141
    • Fix race condition in MemoryLimitController #10142
    • Fix releasing semaphore and memory quota after send timeout #10144
    • Allow configuring memory limit from C API #10145
    • Fix use-after-free undefined behavior due to object lifetime problem #10220
    • Support enable replicate subscription #10243
    • Fix C++ client cannot be built with Boost <=1.53 #10307
    • Support check connect state #10349
    • Avoid sending flow requests with zero permits #10506
    • Add single file logger factory #10712
    • Reduce redeliverMessages when message listener is enabled #10726

    Python Client

    • Replace Exceptions with PulsarExceptions #7600
    • Added logger wrapper support to make logging configurable #7713
    • Initial python 3.9 client wheel build support #9389
    • Support enable replicate subscription #10243

    WebSocket

    • Negative acknowledge support #8249
    • Support deliverAt and deliverAfter attribute #8945
    • Fix returned status code does not depend on the exception type #9031
    • Allow to consume and pass message to client without decryption #10026
    • Support pong command #10035

    Pulsar Admin

    • Support get applied message TTL policy for a topic #9225
    • Support get applied inactive topic policy for a topic #9230
    • Support get applied delayed delivery policy for a topic #9245
    • Support get applied max unacked message per subscription policy for a topic #9290
    • Support get applied max producer for a topic #9293
    • Support get applied max consumer for a topic #9296
    • Support get applied deduplication status policy for a topic #9339
    • Support get applied retention policy for a topic #9362
    • Support get applied offloader policy for a topic #9505
    • Support get applied max unacked messages per consumer for a topic #9694
    • Support get applied dispatch rate policy for a topic #9824
    • Support get applied subscription dispatch rate policy for a topic #9827
    • Support get applied backlog quota policy for a topic #9828
    • Support get applied persistence policy for a topic #9831
    • Support get applied cluster subscribe rate for a topic #9832
    • Support get applied replicator dispatch rate for a topic #9833
    • Support get applied compaction threshold #10038
    • Lazily init PulsarAdmin in PulsarAdminTool #9312
    • Fix create partition of existing topic does not throw RestException #9342
    • Support get service URL of the leader broker #9799
    • Support get persistent topics or non-persistent topics for pulsar admin client #9877
    • Fix can not disable and remove max consumer per subscription #10070
    • Add support for setting time based limit on backlog quota #10401
    • Make client-admin-api to use interfaces with builders instead of POJOs #10818

    Tools

    • [Pulsar Perf] Support load WebSocket service URL from config file #9000)
    • [BookKeeper Shell] Make bookkeeper shell more user friendly #9281
    • [Client Tools] Supports end-to-end encryption #9615
    • [Pulsar Perf] Support specify topics and subscriptions #9716
    • [Client Tools] Allow printing GenericRecord contents #9785
    • [Pulsar Perf] Fix compatibility issues with previous version #9838
    • [Pulsar Perf] Add ability to create partitioned topics #9859
    • [Client Tools] handle data with different schemas when consuming messages #10301
    • [Client Tools] Fix NoClassDefFoundError when running pulsar cli tools in dev directory #10807

    Dependencies

    • Upgrade Kubernetes client and remove sundr-codegen #8576
    • Upgrade ZooKeeper to 3.6.2 and Curator to 5.1.0 #8549
    • [CVE-2020-26238] Upgrade cron-utils to 9.1.3 and sprint-context to 5.3.1 #8822
    • Upgrade Swagger to 1.6.2 and Maven Swagger Plugin to 3.1.7 #8845
    • Upgrade to Apache Avro 1.10.2 #9898
    • Add jersey-client as dependency of pulsar-client-auth-sasl #10055
    • Upgrade Athenz version to 1.10.9 and remove yahoo.bintray.com repository #10079
    • [CVE-2020-15250] Upgrade junit version to 4.13.1 #10147
    • [CVE-2020-8908,CVE-2018-10237] Upgrade jclouds to 2.3.0 #10149
    • Remove jackson-mapper-asl dependency to resolve multiple CVEs #10262
    • [CVE-2021-21409] Upgrade Netty to 4.1.63.Final #10266
    • [CVE-2018-12541] Upgrade vertx to 3.9.7 #10261
    • Upgrade BookKeeper to 4.14.1 #10686
    • [CVE-2021-28169] Upgrade Jetty to 9.4.42.v20210604 #10907
    Source code(tar.gz)
    Source code(zip)

  • v2.6.4(Jun 2, 2021)

    Official release notes: http://pulsar.apache.org/release-notes/#2.6.4

    Download area: http://pulsar.apache.org/en/download/

    Broker

    • Disallow parsing of token with none signature in authenticateToken #9172
    • Fix marking individual deletes as dirty #9732
    • Issue 9082: Broker expires messages one at a time after topic unload #9083
    • [logging] Upgrade Log4j2 version to 2.14.0, replace legacy log4j dependency with log4j-1.2-api #8880
    • Upgrade Bouncy castle to newest version #8047
    • Fixed logic for forceful topic deletion #7356
    • Perform periodic flush of ManagedCursor mark-delete posistions #8634
    • Fix the batch index ack persistent issue. #9504
    • Fix the partition number not equals expected error #9446
    • fix the closed ledger did not delete after expired #9136
    • Fix testBrokerSelectionForAntiAffinityGroup by increasing OverloadedThreshold #9393

    Tiered storage

    • [tiered-storage] Allow AWS credentials to be refreshed #9387

    Java client

    • Compression must be applied during deferred schema preparation and enableBatching is enabled #9396

    C++ client

    • [C++] Remove namespace check for MultiTopicsConsumerImpl #9520
    Source code(tar.gz)
    Source code(zip)

  • v2.7.2(May 25, 2021)

    Official release notes: http://pulsar.apache.org/release-notes/#2.7.2

    Download area: http://pulsar.apache.org/en/download/

    Broker

    • Fix the useless retry when the maximum number of subscriptions is reached #9991
    • wrong timeunit in updating lastLedgerCreationInitiationTimestamp #10049
    • Avoid spammy logs in case of BK problems #10088
    • Fix NonDurableCursorImpl initialPosition by startCursorPosition greater than lastConfirmedEntry problem. #10095
    • fix 8115 Some partitions get stuck after adding additional consumers to the KEY_SHARED subscriptions #10096
    • Add underReplicate state in the topic internal stats #10013
    • Continue graceful shutdown even if web service closing fails #9835
    • Issue 9804: Allow to enable or disable the cursor metrics #9814
    • Allow to configure BookKeeper all BK client features using bookkeeper_ prefix #9232
    • Fix NPEs and thread safety issue in PersistentReplicator #9763
    • Non Persistent Topics: Auto-create partitions even when the auto-creation is disabled #9786
    • Issue 9602: Add schema type validation #9797
    • Fix message not dispatch for key_shared sub type in non-persistent subscription #9826
    • zkBookieRackAffinityMapping bug to support for bookkeeper dnsResolver #9894
    • Messaging Fix delay message block #10078
    • Make PersistentDispatcherMultipleConsumers.readMoreEntries synchronized #10435
    • Fix issue in reusing EntryBatchIndexesAcks instances #10400
    • Fix schema not added when subscribing an empty topic without schema #9853
    • Support advertisedListeners for standalone #10297
    • Fix schema ledger deletion when deleting topic with delete schema. #10383
    • Fix primitive schema upload for ALWAYS_COMPATIBLE strategy. #10386
    • Fix schema type check issue when use always compatible strategy #10367
    • Fix CPU 100% when deleting namespace #10337
    • add return statement to exit asyncMarkDelete early on failure #10272
    • Adding more permits debug statements to better diagnose permit issues #10217

    Bookie

    • Fallback to PULSAR_GC if BOOKIE_GC is not defined #9621
    • Fallback to PULSAR_EXTRA_OPTS if BOOKIE_EXTRA_OPTS isn't defined #10397

    Dependency upgrade

    • Upgrade Bouncy Castle to 1.68 #9199
    • Upgrade athenz version and remove yahoo.bintray.com repository #10471
    • Upgrade Netty version to 4.1.60.final #10073
    • Upgrade commons-io to address CVE-2021-29425 #10287
    • Upgrade Jetty libraries to 9.4.39.v20210325 #10177

    Proxy

    • Issue 10221: Fix authorization error while using proxy and Prefix subscription authentication mode #10226

    Pulsar Admin

    • Add get version command for pulsar rest api, pulsar-admin, pulsar-client #9975

    Pulsar SQL

    • Using pulsar SQL query messages will appear NoSuchLedger… #9910

    Docker

    • Allow DockerImage to be built from source tarball #9846
    • Fix docker standalone image error #10359
    • Suppress printing of "skip Processing" lines in startup scripts #10275
    • Issue 10058:apply-config-from-env.py to commented default values #10060

    Client

    • [Java] Fix: seemingly equal ClientConfigurationData's objects end up not being equal #10091
    • [Java] Fix AutoConsumeSchema KeyValue encoding #10089
    • [Java] Fix error OutOfMemoryError while using KeyValue<GenericRecord, GenericRecord> #9981
    • [Java] Fix concurrency issue in incrementing epoch (#10278) #10436
    • [Java] Allow pulsar client receive external timer #9802
    • [Java] Handle NPE while receiving ack for closed producer #8979
    • [Java] Fix batch size not set when deserializing from byte array #9855
    • [Java] Fix ensure single-topic consumer can be closed #9849
    • [Java] Issue 9585: delete disconnected consumers to allow auto-discovery #9660
    • [Python] Support Python Avro schema set default value. #10265
    • [Python] Fix nested Map or Array in schema doesn't work #9548
    • [C++,Python] [PIP-60] Add TLS SNI support for cpp and python clients #8957
    • [C++] Fix C++ client cannot be built on Windows #10363
    • [C++] Fix paused zero queue consumer still pre-fetches messages #10036
    • [C++] Fix segfault when get topic name from received message id #10006
    • [C++] SinglePartition message router is always picking the same partition #9702
    • [C++] Reduce log level for ack-grouping tracker #10094
    • [WebSocket Client] WebSocket url token param value optimization #10187
    • [WebSocket Client] Make the browser client support the token authentication #9886

    Functions and Pulsar IO

    • Allow customizable function logging #10389
    • Pass through record properties from Pulsar Sources #9943
    • ISSUE 10153: Pulsar Functions Go fix time unit ns -> ms #10160
    • Kinesis Connector: Fix kinesis sink can not retry to send messages #10420
    • Kinesis Connector: Fix null error messages in onFailure exception in KinesisSink. #10416

    Tiered Storage

    • Prevent Class Loader Leak; Restore Offloader Directory Override #9878
    • Add logs for cleanup offloaded data operation #9852
    Source code(tar.gz)
    Source code(zip)

  • v2.7.1(Mar 22, 2021)

    Broker

    • Fix topic ownership is not checked when getting topic policy #9781
    • Fix the issue of consumers cannot be created for older subscriptions if the limit of maxSubscriptionsPerTopic is reached #9758
    • Fix marking individual deletes as dirty #9732
    • Fix broker-address header added when response has already been committed #9744
    • Fix ByteBuffer allocate error in the AirliftUtils #9667
    • Use Atomic Field Updater to increment volatile messagesConsumedCounter #9656
    • Schema comparison logic change #9612
    • Add metrics for the cursor ack state #9618
    • Fix race condition in BrokerService topic cache #9565
    • Avoid introducing bookkeeper-common into the pulsar-common #9551
    • Async read entries with max size bytes #9532
    • Fix the metric data of msgDelayed for partitioned topics is not aggregated #9529
    • Fix the batch index ack persistent issue #9504
    • Fix logic in ManagedLedgerWriter when config threadNum >= ledgerNum #9479
    • Do not use a static map of listeners in TopicPoliciesService #9486
    • Makes subscription start from MessageId.latest as default #9444
    • Fix setting backlogQuota will always succeed #9382
    • Skip clear delayed messages while dispatch does not init #9378
    • Expose offloaded storage size to the topic stats #9335
    • Expose more info with unknown exception #9323
    • Add alerts for expired/expiring soon tokens #9321
    • Fix fake complete issue in offloading #9306
    • Fix system topic can not auto created #9272
    • Fix BookkeeperSchemaStorage NPE #9264
    • Fix race condition on producer/consumer maps in ServerCnx #9256
    • Fix interceptor disabled in ResponseHandlerFilter.java #9252
    • Fix the interceptor that not handle boundary for multipart/form-data #9247
    • Add authentication metrics #9244
    • Handle web application exception to redirect request #9228
    • Skip the interceptor for MediaType.MULTIPART_FORM_DATA #9217
    • Keep topic-level policies commands consistent with that for namespace level #9215
    • Fix handle topic loading failure due to broken schema ledger #9212
    • Fix issue with topic compaction when compaction ledger is empty #9206
    • Fix incoming message size issue that introduced in #9113 #9182
    • Disallow parsing of token with none signature in authenticateToken #9172
    • Fix locking for ConsumerImpl when creating deadLetterProducer #9166
    • Fix maxProducersPerTopic cannot be disabled at the namespace level #9157
    • Fix wrong default value #9149
    • Fix the closed ledger did not delete after expired #9136
    • Additional error checks in TwoPhasesCompactor #9133
    • Fix master broker while subscribing to non-persistent partitioned topics without topic auto-creation #9107
    • Support chained authentication with same auth method name #9094
    • Broker expires messages one at a time after topic unload #9083
    • Add refresh authentication command in broker #9064
    • Add updateRates method for kop to collect publish rate #9094
    • Fix DelayedDelivery at the broker level has a default value #9030
    • Getting the stats of a non-persistent topic that has been cleaned causes it to re-appear #9029
    • Add raw Prometheus metrics provider #9021
    • Improve error handling when broker doesn't trust client certificates #8998
    • Remove duplicated broker Prometheus metrics type 8995
    • Peeking at compressed messages throws an exception (Readonly buffers not supported by Airlift) #8990
    • Make namespaces isolation policy updates take effect on time #8976
    • Fix NPE in PersistentStickyKeyDispatcherMultipleConsumers #8969
    • Fix the recovery not respect to the isolation group settings #8961
    • Add properties default value for SchemaInfoBuilder #8952
    • Consumer support update stats with specified stats #8951
    • Support configure max subscriptions per topic on the topic level policy #8948
    • Fix subscription dispatch rate does not work after the topic unload without dispatch rate limit #8947
    • Avro custom schema not working in consumer #8939
    • Expose non-contiguous deleted messages ranges stats #8936
    • Intercept beforeSendMessage calls #8932
    • Monitor if a cursor moves its mark-delete position #8930
    • Capture stats with precise backlog #8928
    • Support configure max subscriptions per topic on the namespace level policy #8924
    • Export Prometheus metric for messageTTL #8871
    • Add pulsar-perf new feature: one subscription has more than one consumer #8837
    • Execute removing non-persistent subscription of a topic from a different thread to avoid deadlock when removing inactive subscriptions #8820
    • Fix get partition metadata problem for a non-existed topic #8818
    • Fix the problem that batchMessageId is converted to messageIdImpl #8779
    • Clear delayed messages when clear backlog #8691
    • Fixes first automatic compaction issue #8209

    Proxy

    • Fix Proxy Config bindAddress does not working for servicePort #9068
    • Return correct authz and auth errors from proxy to client #9055
    • Fix the metadata setup compatibility issue #8959
    • Support HAProxy proxy protocol for broker and proxy #8686

    Pulsar Perf

    • Dump JVM information #9769
    • pulsar-perf uses DefaultCryptoKeyReader for E2E encryption #9668
    • Add --batch-index-ack for the pulsar-perf #9521

    Transaction

    • Fix deleteTransactionMarker memory leak #9752
    • Fix transaction messages order error and deduplication error #9024
    • Fix transaction log replay not handle right #8723

    Pulsar Admin

    • Validate offload param #9737
    • Inform user when expiring message request is not executed. #9561
    • Fix get-message-by-id throwing NPE when message is null #9537
    • Expire message by position #9519
    • Add subscription backlog size info for topicstats #9302
    • Expose schema ledger in topic stats-internal #9284
    • Fix potential HTTP get hangs in the Pulsar Admin #9203
    • Fix admin-api-brokers list failed #9191
    • Fix force delete namespace did not delete all topics of the namespace #8806
    • Change method getWebServiceUrl into async #8746
    • Fix cannot get lastMessageId for an empty topic due to message retention #8725

    Pulsar SQL

    • Duplicate key __pfn_input_topic__ in presto server #9686
    • Pulsar sql key-value schema separated model support #9685
    • Fix OffloadPolicies json serialization error in Pulsar SQL #9300

    Client

    • [Java] Add original info when publishing message to dead letter topic #9655
    • [Java] Fix hasMessageAvailable() with empty topic #9798
    • [Java] Add BouncyCastleProvider as security provider to prevent NPE #9601
    • [Java] Async the DLQ process #9552
    • [Java] Fix the partition number not equals expected error #9446
    • [Java] Cleanup consumer on multitopic subscribe failure #9419
    • [Java] Compression must be applied during deferred schema preparation and enableBatching is enabled #9396
    • [Java] Add default implementation of CryptoKeyReader #9379
    • [Java] Able to handling messages with multiple listener threads in order for the Key_Shared subscription #9329
    • [Java] Fix NPE when MultiTopicsConsumerImpl receives null value messages #9113
    • [Java] Fix Unavailable Hash Range Condition #9041
    • [Java] Add more information in send timeout exception #8931
    • [Java] GenericJsonReader converts the null value to string "null" #8883
    • [Java] Always remove message data size #8566
    • [Python] Support python end to end encryption #9588
    • [C++] Add 'encrypted' option in commands.newproducer() #9542
    • [C++] Remove namespace check for MultiTopicsConsumerImpl #9520
    • [C++] Fix broken replication msg to specific cluster #9372
    • [C++] Fix compilation issue caused by non-virtual destructor #9106
    • [C++] Expose cpp end to end encryption interface #9074
    • [C++] Fix Consumer send redeliverMessages repeatedly #9072
    • [C++] Add consumer's configs for reader #8905
    • [C++] Add reader internal subscription name setter #8823
    • [C++] Fix race condition in BlockingQueue #8765
    • [C++] Fix cpp client do AcknowledgeCumulative not clean up previous message #8606
    • [C++] Implement batch aware producer router #8395
    • [Websocket] Fix the initial sequence id error #8724

    Function

    • Add downloadDirectory support to function k8s runtime #9619
    • Kubernetes runtime functions create rfc1123 compliant labels #9556
    • Fix can't create functions with m-TLS #9553
    • Fix reading metrics will always get stuck in some cases #9538
    • Call the corresponding restart according to the componentype #9519
    • Fix narExtractionDirectory not set #9319
    • Fix java function logging appender not added to java function logger #9299
    • Fix don't attempt to clean up packages when Source/Sink is builtin #9289
    • Fix function worker get superuser role #9259
    • Fix broker and functions-worker authentication compatibility #9190
    • Splitting the authentication logic of function worker and client #8824
    • [Go] Fix metrics server handler error #9394
    • [Go] Add metrics server to go function #9318
    • [Go] Fix publishfunc example is broken #9124

    Pulsar IO

    • Add option for auto.offset.reset to kafka source #9482
    • Fix debezium-connector error log #9063
    • Fix NSQ source META-INF file name and sourceConfigClass #8941
    • Make Source topic Schema information available to downstream Sinks #8854

    Tiered Storage

    • Allow AWS credentials to be refreshed #9387
    • Offload manager initialization once #8739
    • Configurable data source for offloaded messages #8717
    Source code(tar.gz)
    Source code(zip)

  • v2.7.0(Dec 3, 2020)

    The following lists fixes and enhancements in the 2.7.0 release.

    Transactions

    • Implement the Transaction Buffer Client #6544
    • Support produce messages with transaction and commit transaction #7552
    • Support consume transaction messages #7781 #7833
    • Message acknowledgment with transaction #7856 #8007
    • Support transaction abort on partition #7953
    • Support transaction abort on subscription #7979
    • Handle pending ack at the client side #8037
    • Pending ack state implementation #8426
    • Support get reponse for message acknowledge #8161
    • Refactor the transaction buffer implementation #8291 #8347
    • Transaction marker deletion #8318
    • Support produce messages with transaction in batch #8415
    • Register transaction metadata before send or ack messages #8493
    • Expose transaction interface #8505
    • Guarantee transaction metadata handlers connected #8563
    • Add the batch size in transaction ack command #8659
    • Implement the Transaction Log #8658

    Topic policy

    • Support setting message TTL on topic level #7738
    • Support setting retention on topic level #7747
    • Support setting delayed delivery policy on topic level #7784
    • Support setting max unacked message per subscription on topic level #7802
    • Support setting persistence policie on topic level #7817
    • Support setting max unacked messages per consumer on topic level #7818
    • Support setting deduplication policy on topic level #7821
    • Support setting message dispatch rate on topic level #7863)
    • Support setting compaction threshold on topic level #7881
    • Support setting offload policy on topic level #7883
    • Support setting max producers for a topic #7914
    • Support setting max consumers for a topic #7968
    • Support setting publish rate limitation for a topic #7948
    • Support setting inactive topic policy on topic level #7986
    • Support setting subscribe rate for a topic #7991
    • Support setting max consumers per subscription on topic level #8003
    • Support setting subscription dispatch rate on topic level #8087
    • Support setting deduplication snapshot interval on topic level #8552

    Broker

    • Upgrade BookKeeper version to 4.12.0 #8447
    • Capture the add entry latency of managed-ledger #4419
    • Keep max-concurrent http web-request configurable #7250
    • Perform the unload in background after bundle split #7387
    • Cleanup already deleted namespace topics when remove cluster #7473
    • Support partitioned topics in the Reader #7518
    • Support partitioned topic lookup #7605
    • Make OrderedExecutor threads number configurable #7765
    • Add config to lazily recover cursors when recovering a managed ledger #7858
    • Make BookKeeper throttle configurable #7901
    • Report compacted topic ledger info when calling get internal stats #7988
    • Add broker config to enforce producer to publish encrypted message #8055
    • Expose ensemble placement policy in bookkeeper.conf #8210
    • Support limit topic publish rate at the broker level #8235
    • Support limit the max tenants of the Pulsar cluster #8261
    • Support limit the max namespaces per tenant #8267
    • Support limit max subscriptions per topic #8289
    • Added metrics for topic lookups operations #8272
    • Added REST handler for broker ready probe #8303
    • Configure namespace anti-affinity in local policies #8349
    • Handle hash collision in KeyShared subscription mode #8396
    • Configure maxMsgReplDelayInSeconds for each repl-cluster #8409
    • Support taking de-duplication snapshots based on time #8474
    • Support namespace-level duplication snapshot #8506
    • Expose consumer names after the mark delete position for the Key_Shared subscription #8545
    • Close topics that remain fenced forcefully #8561

    Functions

    • Separate out FunctionMetadata related helper functions #7146
    • Attach names for all producers/readers in worker service #7165
    • Add support to read compacted topic #7193
    • Re-work Function MetaDataManager to make all metadata writes only by the leader #7255
    • Fix leader/scheduler assignment processing lag problem #7237
    • Set source spec's negativeacktimeout as well as timeout #7337
    • Add an endpoint to check whether function worker service is initialized #7350
    • Functions metadata compaction #7377
    • Implement rebalance mechanism #7388
    • Improve security setting #7424
    • Allow function rebalance to be run periodically #7449
    • Log scheduler stats for Pulsar Functions #7474
    • Add BatchPushSource interface #7493
    • Rejigger contract between LeaderService and rest of components #7520
    • Allow null consume in BatchPushSource #7573
    • Add readiness api for the worker leader #7601
    • Reduce in the leader init time in Pulsar Functions #7611
    • Export Function worker internal stats via Prometheus #7641
    • Allow ability to specify retain key ordering in functions #7647
    • Added ability to specify runtime for localrunner #7681
    • Add additional metrics for Pulsar Function Worker #7685
    • Use available cores for io thread processing #7689
    • Added ability to specify producer config for functions and sources #7721
    • Allow the option to make producers thread local #7764
    • Add ability for BatchPushSource to notify errors asynchronously #7865
    • Allow ability to specify sub position in functions #7891
    • Add hostname to consumer/producer properties in Pulsar Functions #7897
    • Allow specifying state storage url for Source/Sink localrun #7930
    • Enable function worker JVM metrics to be reported via Prometheus #8097
    • Add ability to specify EnvironmentBasedSecretsProvider in LocalRunner #8098
    • Added ability to specify secrets class in localrunner builder #8127
    • Add access to the current message from the function context #8290
    • Enable e2e encryption for Pulsar Function #8432
    • Support key_based batch builder for functions and sources #8523
    • Refactor Context and State API to allow plugging different state store implementations #8537

    IO connectors

    • [HDFS] Add config to create sub directory from current time #7771
    • [NSQ] Add NSQ Source #8372

    Schema

    • Add java8 date and time type to primitive schemas #7874
    • Native protobuf schema support #7874
    • Refactor multi-version schema reader #8464

    Tiered storage

    • Support Azure BlobStore offload #8436

    Clients

    • [Java] Support acknowledging a list of messages #7688
    • [Java] Remove UUID generation on sending message #7705
    • [Java] Perform producer compression from IO threads #7733
    • [C++] Allow to configure KeyShared with out of order delivery #7842
    • [Java] Await thread pool termination when closing Pulsar client #7962
    • [Java] Support non-durable subscription for pulsar-client cli #8100
    • [Java] Cancel producer sendtimeout task after creation failure #8497
    • [cgo] Remove CGO client from repo #8514

    Admin

    • [Pulsar Admin] support config request timeout #7698
    • [Pulsar Admin] Ensure deleting a partitioned-topic on a non existing namespace returns 404 #7777
    • [Pulsar Admin] Added support to force deleting namespace #7993
    • [Pulsar Admin] Allow to get ledger metadata along with topic stats-internal #8180
    • [Pulsar Admin] Support remove namespace level offload policy #8446
    • [Pulsar Admin] Suport get list of bundles under a namespace #8450
    • [Pulsar Admin] Add ability to examine specific message by position relative to earliest or latest message #8494
    • [Pulsar Admin] Add key-shared consumer range to internal topic stats #8567

    Fixes

    • [Java Client] Fix connection leak #6524
    • [Broker] Prevent redirection of lookup requests from looping #7200
    • [Broker] Ensure that admin operations are gated by super user check #7226
    • [Broker] Fix race condition when delete topic forcelly #7356
    • [Tiered Storage] Fix NPE when offload data to GCS #7400
    • [Function]Fix race condition in which exitFuture in FunctionAssignmentTailer never gets completed even though the tailer thread has exited #7351
    • [Function] Various fixes and optimizations for processing assignments in function worker #7338
    • [Function] Fix deadlock between create function and leader initialization #7508
    • [Pulsar Admin] Fix exceptions being ignored in PulsarAdmin #7510
    • [Broker] Fix the nondurable consumer can not specify the initial position #7702
    • [Broker] Fixed race condition on deleting topic with active readers #7715
    • [Broker] Avoid ConcurrentModificationException of LocalBrokerData #7729
    • [C++ Client] Fix race condition caused by consumer seek and close #7819
    • [Pulsar Proxy] Fix memory leak with debug log-level #7963
    • [Broker] Double check from zookeeper if availableBrokers is empty for discovery service #7975
    • [Broker] Fix broker-ml bucket stats show high metrics rate #8218
    • [Broker] Fix incorrect configuration for zk-cache expire time #8302
    • [Function] Fix returned status code for get function state when state does not exist #8437
    • [Broker] Fix the residual of inactive partitioned-topic cleaning #8442
    • [Pulsar Proxy] Fix request.getContentLength() to return 0 if it is less than 0 #8448
    • [Broker] Fix race condition when calling acknowledgementWasProcessed() #8499
    • [Java Client] Fix handling errors for client requests #8518
    • [C++ Client] Fix potential crash caused by AckGroupTracker's timer #8519
    Source code(tar.gz)
    Source code(zip)

  • v2.6.2(Nov 9, 2020)

    The following lists fixes and enhancements in the 2.6.2 release.

    Broker

    [Broker] Catch throwable when start pulsar 7221 [Broker] Protobuf-shaded package can not update version 7228 [Broker] Check for null arguments in Namespaces Rest API 7247 [Broker] Handle SubscriptionBusyException in resetCursor api 7335 [Broker] Converted the namespace bundle unload into async operation 7364 [Broker] Update Jersey to 2.31 7515 [Broker] Stop to dispatch when skip message temporally since Key_Shared consumer stuck on delivery 7553 [Broker] Fix bug where producer for geo-replication is not closed when topic is unloaded 7735 [Broker] Make resetting cursor in REST API asynchronous 7744 [Broker] Reestablish namespace bundle ownership from false negative releasing and false positive acquiring 7773 [Broker] make pulsar executor pool size configurable 7782 [Broker] Redirect Get message by id request when broker not serve for the topic 7786 [Broker] Make zk cache executor thread pool size configurable 7794 [Broker] Implement toString() method for TopicMessageIdImpl class 7807 [Broker] Fix pending batchIndexAcks bitSet batchSize in PersistentAcknowledgmentsGroupingTracker 7828 [Broker] Fix deadlock when adding consumer 7841 [Broker] Split message ranges by ledger ID and store them in individualDeletedMessages 7861 [Broker] Fix pulsar metrics providing wrong information 7905 [Broker] Don't fail the health check request when trying to delete the previous subscription 7906 [Broker] Add configuration to set number of channels per bookie 7910 [Broker] Fix publish buffer of one ServerCnx calculated multi-times when check broker's publish buffer 7926 [Broker] Add some logging to improve Authentication debugging and Fix typos in code "occured" -> "occurred" 7934 [Broker] Fix NPE when acknowledge messages at the broker side 7937 [Broker] Fix the wrong issuer url concatenate 7980 [Broker] Upgrade the snakeyaml version to 1.26 7994 [Broker] Exclude vertx from bookkeeper-http package 7997 [Broker] Check null point before setting auto read 7999 [Broker] Fix IndexOutOfBoundsException in the KeyShared subscription when dispatching messages to consumers 8024 [Broker] Upgrade jetty-util version to 9.4.31 8035 [Broker] Add replicated check to checkInactiveSubscriptions 8066 [Broker] Add get-last-message-id admin for v1 api 8081 [Broker] Fix client lookup hangs when broker restarts 8101 [Broker] Should not cache the owner that does not belong to current server 8111 [Broker] Support to specify multi ipv6 hosts in brokerServiceUrl 8120 [Broker] Intercept messages to consumers and add intercept exception 8129 [Broker] Add ChannelFutures utility class to pulsar-common 8137 [Broker] Support Disable Replicated Subscriptions 8144 [Broker] Fix error code returned to client when service unit is not ready 8147 [Broker] Skip intercepting multipart requests 8156 [Broker] Enable intercept filters only when interceptors are configured 8157 [Broker] Clean inactive non-persistent subscriptions 8166 [Broker] Add a new state for namespace-level TTL 8178 [Broker] Fix peek messages failed with subscriptionName not exist 8182 [Broker] Fix pulsar service close exception 8197 [Broker] Use ThreadPoolExecutor instead of EventLoop 8208 [Broker] Close ZK connections at end of metadata setup 8228 [Broker] Delete associated ledgers before deleting cluster metadata 8244 [Broker] Fix stuck lookup operations when the broker is starting up 8273 [Broker] Fix Broker enters an infinite loop in ManagedLedgerImpl.asyncReadEntries 8284 [Broker] Fix message TTL on Key_Shared subscription and Fix ordering issue when replay messages 8292 [Broker] Fix race condition in updating readPosition in ManagedCursorImpl 8299 [Broker] Refresh ZooKeeper-data cache in background to avoid deadlock and blocking IO on ZK thread 8304 [Broker] Upgrade hdfs2 version to 2.8.5 8319 [Broker] Upgrade solr version to 8.6.3 8328 [Broker] Fix deadlock that occurred during topic ownership check 8406

    Proxy

    [Proxy] Add advertisedAddress config field to ProxyConfiguration 7542 [Proxy] Fix deadlock in pulsar proxy 7690 [Proxy] Handle NPE while updating proxy stats 7766 [Proxy] Fix the null exception when starting the proxy service 8019 [Proxy] Add proxy plugin interface to support user defined additional servlet 8067

    Pulsar SQL

    [Pulsar SQL] Upgrade Presto version to 332 7194 [Pulsar SQL] Replace com.ning.asynchttpclient with org.asynchttpclient 8099

    Java Client

    [Java Client] Support input-stream for trustStore cert 7442 [Java Client] Avoid subscribing the same topic again 7823 [java Client] Add autoPartitionsUpdateInterval for producer and consumer 7840 [Java Client] Avoid resolving address for sni-host + thread-safe connection creation 8062 [Java Client] Websocket interface decode URL encoding 8072 [Java Client] Always use SNI for TLS enabled Pulsar Java broker client 8117 [Java Client] Improve timeout handling in ClientCnx to cover all remaining request types (GetLastMessageId, GetTopics, GetSchema, GetOrCreateSchema) 8149 [Java Client] Fix ConsumerImpl memory leaks 8160 [Java Client] Fix issue where paused consumer receives new message when reconnecting 8165 [Java Client] Improve refactored client connection code 8177 [Java Client] Add log level configuration in pulsar-client 8195 [Java Client] Remove unnecessary locks 8207 [Java Client] Fix AutoUpdatePartitionsInterval setting problem 8227 [Java Client] Add read position when joining in the consumer stats 8274 [Java Client] Support reset cursor to a batch index of the batching message 8285 [Java Client] Support exclude the message when reset cursor by message ID 8306 [Java Client] Increasing timeout for pulsar client io threads to shutdown 8316 [Java Client] Support cancelling message & batch futures returned from Reader & Consumer 8326 [Java Client] Disable batch receive timer for Readers 8381 [Java Client] Fix pause does not work for new created consumer 8387

    CPP Client

    [CPP Client] Wait for all seek operations completed 7216 [CPP Client] Ensure parallel invocations of MultiTopicsConsumerImpl::subscribeAsync with the same topic name do not produce an error. 7691 [CPP Client] Throw std::exception types 7798 [CPP Client] Make clear() thread-safe 7862 [CPP Client] Support key based batching 7996 [CPP Client] The token endpoint should get from the well-known configuration 8006 [CPP Client] Add Snappy library to Docker images for building C++ packages 8086 [CPP Client] Add epoch for C++ client HandleBase to handle create producer timeout 8191 [CPP Client] Fix some pending requests may never complete when broker's down 8232 [CPP Client] Client is allocating buffer bigger than needed 8283 [CPP Client] Client back-pressure is done on batches rather than number of messages 8331 [CPP Client] Fix message id error when subscribing a single partition 8341

    Python Client

    [Python Client] Add python schema field default value 8122 [Python Client] Expose schema version (of writerSchema) in Message 8173

    Pulsar Functions

    [Pulsar Functions] During Function update, cleanup should only happen for temp files that were generated 7201 [Pulsar Functions] Have metadata tailer use its own thread for processing 7211 [Pulsar Functions] Allow kubernetes runtime to customize function instance class path 7844 [Pulsar Functions] SinkRecord adds an overridden method 8038 [Pulsar Functions] Set dryrun of KubernetesRuntime is null 8064 [Pulsar Functions] Allow disabling forwarding source message properties 8158 [Pulsar Functions] Missed dryRun on maintenance of secrets 8286

    Pulsar Perf

    [Pulsar Perf] Support setting message key 7989 [Pulsar Perf] Make pulsar-perf ioThread number configurable 8090

    Pulsar Admin

    [Pulsar Admin] Support initial namespace of the cluster without startup the broker 7434 [Pulsar Admin] Fix some params on consumer broken by #4400 (regex, initialSouscriptionPosition) 7795 [Pulsar Admin] Return more informative error message when trying to create subscription on non-persistent through Rest API or pulsar-admin CLI 7831 [Pulsar Admin] Add cli command to get last message Id 8082 [Pulsar Admin] Support delete all data associated with a cluster 8133 [Pulsar Admin] Support delete schema ledgers when delete topics 8167 [Pulsar Admin] Add command to delete a cluster's metadata from ZK 8169 [Pulsar Admin] Support reset cursor to a batch index for Pulsar Admin 8329

    Tiered Storage

    [Tiered Storage] Refactored JCloud Tiered Storage 6335 [Tiered Storage] Remove duplicate updates 8198 [Tiered Storage] Make the field name in OffloadPolicies match with config file 8310

    Source code(tar.gz)
    Source code(zip)

  • v2.6.1(Aug 21, 2020)

    2.6.1 — 2020-08-21

    The following lists fixes and enhancements in 2.6.1 release.

    Broker

    • [Broker] Limit batch size to the minimum of the maxNumberOfMessages and maxSizeOfMessages #6865
    • [Broker] Fix hash range conflict issue in Key_Shared with sticky hash range #7231
    • [Broker] Fix the issue that get lookup permission error #7234
    • [Broker] Update Jetty to version 9.4.29 #7235
    • [Broker] Fix readers backlog stats after data is skipped #7236
    • [Broker] Fix the regression in isSupperUser #7241
    • [Broker] Avoid introducing null read position for the managed cursor #7264
    • [Broker] Fix permission operation check on setRetention admin operation #7281
    • [Broker] Fix error in creation of non-durable cursor #7355
    • [Broker] Fix bug related to managedLedger properties #7357
    • [Broker] Add tenant name check in list namespaces function #7369
    • [Broker] Avoid the NPE occurs in method ManagedLedgerImpl.isOffloadedNeedsDelete #7389
    • [Broker] Fix producer stuck issue due to NPE thrown when creating a new ledger #7401
    • [Broker] Avoid NPEs at ledger creation when DNS failures happen #7403
    • [Broker] Support decompression payload if needed in KeyShared subscription #7416
    • [Broker] Fix update-cluster cli updates proxy-url #7422
    • [Broker] Handle NotAllowed Exception at the client side #7430
    • [Broker] Shade jclouds to avoid Gson conflict #7435
    • [Broker] Consumer is registered on dispatcher even if hash range conflicts on Key_Shared subscription #7444
    • [Broker] Add pulsar-client-messagecrypto-bc into pulsar-client dependency to avoid method not found #7447
    • [Broker] Fix update partitions error for non-persistent topic #7459
    • [Broker] Use CGroup CPU usage when present #7475
    • [Broker] Fix ArrayIndexOutOfBoundsException when dispatch messages to consumer #7483
    • [Broker] Get last entry is trying to read entry -1 #7495
    • [Broker] Fix timeout opening managed ledger operation #7506
    • [Broker] Fixes the exception that occurred when the geo-replication policy is updated #7514
    • [Broker] Update Jackson to version 2.11.1 and ensure all dependencies are pinned #7519
    • [Broker] Fix protobuf generation on handling repeated long number #7540
    • [Broker] Add more logging to the auth operations on failure #7567
    • [Broker] Use Consume/Produce/Lookup interfaces for specific operations in allowTopicOperation #7587
    • [Broker] Support configuring DeleteInactiveTopic setting in namespace policy #7598
    • [Broker] Fix NPE when using advertisedListeners #7620
    • [Broker] Fix the issue that deduplication cursor can not be deleted after disabling message deduplication #7656
    • [Broker] Add missing AuthenticationDataSource to canConsumeAsync method call #7694
    • [Broker] Close the previous reader of the health check topic #7724
    • [Broker] Change some WebApplicationException log level to debug #7725
    • [Broker] Replay delayed messages in order #7731
    • [Broker] Fix the wrong returned URL for lookup when specify advertised listener #7737
    • [Broker] Fix topic getting recreated immediately after deletion #7524
    • [Broker] Set default root log level to debug #7789
    • [Broker] Fix producer stucks on creating ledger timeout #7319
    • [Broker] AllowTopicOperationAsync should check the original role is super user #7788

    Zookeeper

    • [Zookeeper] Use hostname for bookie rackawareness mapping #7361

    Pulsar SQL

    • [Pulsar SQL] Make Pulsar SQL get correct offload configurations #7701

    Pulsar Schema

    • [Schema] Fix the error that occurs when getting schemaName by partitioned topic name #7708

    Java Client

    • [Java Client] Fix the issue that the HTTP header used in Athenz authentication can not be renamed #7311
    • [Java Client] Add more detail information of retry errors #7341
    • [Java Client] Check NPE when a tombstone (null value) is produced. #7408
    • [Java Client] Fix batch ackset recycled multiple times. #7409
    • [Java Client] Support Oauth2 authentication #7420
    • [Java Client] Ensure the create subscription can be completed when the operation timeout happens #7522
    • [Java Client] Fix race condition on the close consumer while reconnecting to the broker. #7589
    • [Java Client] Fix validation never return false #7593
    • [Java Client] Make OAuth2 auth plugin to use AsyncHttpClient #7615
    • [Java Client] Support to set listener name for client CLI #7621
    • [Java Client] Fix batch index filter issue in Consumer #7654
    • [Java Client] Fix the backward compatibility issues with batch index acknowledgment. #7655
    • [Java Client] Fix the issue that batchReceiveAsync is not completed exceptionally when closing consumer #7661
    • [Java Client] Fix producer stats recorder time unit error #7670
    • [Java Client] Fix shutdown AsyncHttpConnector.delayer #7687

    CPP Client

    • [CPP Client] Fix partition index error in closing callback #7282
    • [CPP Client] Reduce log level for ack-grouping tracker #7373
    • [CPP Client] Support Oauth2 authentication #7467
    • [CPP Client] Fix segment crashes that caused by race condition of timer in cpp client #7572
    • [CPP Client] Fix multitopic consumer segfault on connect error #7588
    • [CPP Client] Add support to read credentials from file #7606
    • [CPP Client] Fix the issuer_url parsing failure in oauth2 #7791
    • [CPP Client] Fix reference leak when reader create #7793

    Pulsar Functions

    • [Pulsar Function] Use fully qualified hostname as default to advertise worker #7360
    • [Pulsar Function] Fix the function BC issue introduced in release 2.6.0 #7528
    • [Pulsar Function] Improve security setting of Pulsar Functions #7578
    • [Pulsar Function] Differentiate authorization between source/sink/function operations #7466

    Go Function

    • [Go Function] Fix Go instance config port #7322
    • [Go Function] Remove timestamp from metrics #7539

    Pulsar Perf

    • [Pulsar Perf] Supports tlsAllowInsecureConnection in pulsar-perf produce/consume/read #7300
    Source code(tar.gz)
    Source code(zip)

  • v2.6.0(Jun 17, 2020)

    2.6.0 — 2020-06-17

    Features

    PIPs
    • [PIP-37] Large message size support #4400
    • [PIP-39] Namespace change events (System Topic) #4955
    • [PIP-45] Switch ManagedLedger to use MetadataStore interface #5358
    • [PIP 53] Contribute DotPulsar to Apache Pulsar
    • [PIP-54] Support acknowledgment at batch index level #6052
    • [PIP-58] Support consumers set custom message retry delay #6449
    • [PIP-60] Support SNI routing to support various proxy-server #6566
    • [PIP-61] Advertise multiple addresses #6903
    • [PIP-65] Adapting Pulsar IO Sources to support Batch Sources #7090
    Broker
    • [Broker] Add threshold shedder strategy and metrics exporter for loadbalancer #6772
    • [Broker] Add consistent hashing in the Key_Shared distribution #6791
    • [Broker] Fixed ordering issue in KeyShared subscription dispatcher when adding consumer #7106 #7108 #7188
    • [Broker] Add support for key hash range reading in Key_Shared subscription #5928
    • [Broker] Allow for schema reader and writer registration on SchemaDefinition #6905
    • [Broker] Support use null key and null value in KeyValue Schema #7139
    • [Broker] Support multiple pulsar clusters to use the same bk cluster #5985
    • [Broker] Add a flag to skip broker shutdown on transient OOM #6634
    • [Broker] Make zookeeper cache expiry time configurable #6668
    • [Broker] Check replicator periodically to avoid issue due to zookeeper missing watch #6674
    • [Broker] Expose managedLedgerCache, managedLedger, loadBalance metrics to Prometheus #6705
    • [Broker] Optimize consumer fetch messages in case of batch message #6719
    • [Broker] Add configuration to limit max partitions for a partitioned topic #6794
    • [Broker] Change default FlushEntryLogBytes to 256MB to improve bookie io throughput #6915
    • [Broker] Introduce precise topic publish rate limiting #7078
    • [Broker] Expose new entries check delay in the broker.conf 7154
    • [Broker] Add broker interceptor for intercepting all Pulsar command and REST API requests #7143
    • [Broker] Only close active consumer for Failover subscription when seek() #7141
    • [Broker] Allow to delete topics that are failing to recover #7131
    • [Broker] Support set netty max frame size in bookkeeper.conf #7116
    • [Broker] Trigger rollover when meeting maxLedgerRolloverTimeMinutes #7111
    • [Broker] Cap the dispatcher batch size in bytes to fixed max #7097
    • [Broker] Support specify managedLedgerMaxSizePerLedgerMbytes in broker.conf #7085
    • [Broker] Allow to grant permissions when the authorization is disabled #7074
    • [Broker] Add messages and bytes counter stats to broker-stats/topics #7045
    • [Broker] Expose new entries check delay in the broker.conf #7154
    Function
    • [Function] Built-in functions support #6895
    • [Function] Add Go Function heartbeat (and gRPC service) for production usage #6031
    • [Function] Add custom property option to functions #6348
    • [Function] Separate TLS configuration of function worker and broker #6602
    • [Function] Added ability to build consumers in functions and sources #6954
    • [Function] Support DLQ on sources and sinks #7032
    Pulsar SQL
    • [SQL] KeyValue schema support #6325
    • [SQL] Multiple version schema support #4847
    • [SQL] Fix presto SQL does not start metrics service before queue execute #7030
    Pulsar IO
    • Added ability for sources to publish messages on their own #6941
    • [RabbitMQ] Allow routing key per message to RabbitMQ sink connector #5890
    • [RabbitMQ] Add passive config options #6679
    • [debezium] Upgrade from v0.10.0-Final to v1.0.0-Final #5972
    • [debezium] Support avro schema for debezium connector #6034
    • [influxdb2] Add support for influxdb2 in pulsar-influxdb-sink #6601
    • [jdbc] Add jdbc sinks: postgres, mariadb, clickhouse #6835
    Pulsar Proxy
    • [Proxy] Add REST API to get connection and topic stats #6473
    • [Proxy] Add advertised address option #6942
    • [Proxy] Add proxyLogLevel into config #6948
    Clients
    • [Java Client] Use pure-java Air-Compressor instead of JNI based libraries #5390
    • [Java Client] Change the time unit of patternAutoDiscoveryPeriod to seconds #5950
    • [Java Client] Support waiting for inflight messages while closing producer #6648
    • [Java Client] Add support to load TLS certs/key dynamically from input stream #6760
    • [Java Client] Support return sequence ID when throwing Exception for async send message #6825
    • [Java Client] Change the default value of maxLookupRedirects of Java client to 20 #7126
    • [Java Client] Limit the number of times lookup requests are redirected #7096
    • [CPP Client] Support seek by time on partitioned topic #7198
    • [CPP Client] Refresh authentication credentials #7070
    • [CPP Client] Fix Result can't be serialized to string inside the library #7034
    • [CPP Client] Support zstd and Snappy compression to C API #7014
    • [Python Client] Add deliver_at and deliver_after for the producer #6737
    Admin
    • [Admin] Support delete inactive topic when subscriptions caught up #6077
    • [Admin] Add configuration to disable auto-creation of subscriptions #6456
    • [Admin] Add maxUnackedMessagesPerSubscription and maxUnackedMessagesPerConsumer on namespaces policies #5936
    • [Admin] Support get a message by message ID in pulsar-admin #6331
    • [Admin] Support delete subscription forcefully #6383
    • [Admin] Add subscribe initial position for consumer CLI #6442
    • [Admin] Support to get managed ledger info of a partitioned topic #6532
    • [Admin] Support compact all partitions of a partitioned topic #6537
    • [Admin] Support multi-hosts in PulsarAdmin #6547
    • [Admin] Support to get internal stats for a partitioned topic #6624
    • [Admin] Support enable or disable subscription auto-creation at namespace level #6637
    • [Admin] Enable to set the subscription expiration time for each namespace #6851

    Fixes

    • [Broker] Fixed increasing number of partitions with attached readers #7077
    • [Broker] Make ZkBookieRackAffinityMapping work as expected #6917
    • [Broker] Fix backlog and backlog size stats keeps growing #7082
    • [Java Client] Fix connection leak #6524
    • [Java Client] Fix message id compare between MessageId and BatchMessageId #6621
    • [Java Client] Fix memory leak when create producer with not exsits topic #7120 #7124
    • [Java Client] Fix duplicated messages sent to dead letter topic #7021
    • [CPP Client] Fix deadlock of consumer for topics auto discovery #7206
    • [Managed Ledger] Fix NPE on opening non-durable cursors on an empty managed ledger #7133
    • [Websocket] Fix incorrect topic URL parse #6630
    • [Pulsar SQL] Fix problem with multiple zookeeper address #6947
    • [Docker] Do not apply env values to pulsar_env.sh and bkenv.sh implicitly 6579
    Source code(tar.gz)
    Source code(zip)

  • v2.5.2(May 19, 2020)

    2.5.2 — 2020-05-19

    Fixes and Enhancements

    Broker
    • [Broker] Implement AutoTopicCreation by namespace level override #6471
    • [Broker] Add custom deletionLag and threshold for offload policies per namespace #6422
    • [Broker] Invalidate managed ledgers zookeeper cache instead of reloading on watcher triggered #6659
    • [Broker] Retention policy should be respected when there is no traffic #6676
    • [Broker] Fixed double delete on a namespace #6713
    • [Broker] fix get batch message from http response, only get the first message#6715
    • [Broker] Fix Deadlock by Consumer and Reader#6728
    • [Broker] avoid backpressure by skipping dispatching if consumer channel is not writable #6740
    • [Broker] fix when producing encrypted messages, MessageMetadata objects are not released after they are created. #6745
    • [Broker] Bump netty version to 4.1.48.Final #6746
    • [Broker] Increase timeout for loading topics #6750
    • [Broker] Fix wrong cursor state for cursor without consumer #6766
    • [Broker] change nondurable cursor to active to improve performance #6769
    • [Broker] register loadbalance znode should attempt to wait until session expired #6788
    • [Broker] Fix some empty message related problems in the compacted topic. #6795
    • [Broker] Avoid creating partitioned topic for partition name #6846
    • [Broker] Add Tls with keystore type config support #6853
    • [Broker] fix consumer stuck when batchReceivePolicy maxNumMessages > maxReceiverQueueSize #6862
    • [Broker] use originalAuthMethod on originalAuthChecker in Proxy Authentication #6870
    • [Broker] Close producer when the topic does not exists. #6879
    • [Broker] Handle all exceptions from topic.addProducer #6881
    • [Broker] fix topicPublishRateLimiter not effective after restart broker #6893
    • [Broker] Expose pulsar_out_bytes_total and pulsar_out_messages_total for namespace/subscription/consumer. #6918
    • [Broker] Policy ttlDurationDefaultInSeconds not applying #6920
    • [Broker] Fix pulsar admin thread number explode bug. #6940
    Pulsar Schema
    • [Schema] Fix long field parse in GenricJsonRecord #6622
    • [Schema] Fix the leak of cursor reset if message encode fails in Avro schema. #6695
    • [Schema] fix Get schema by version can get the deleted schema info #6754 #6754
    • [Schema] Fix serialization of enums with json/avro schemas in python #6808
    • [Schema] Pulsar SQL Support Avro Schema ByteBuffer Type #6925
    CPP Client
    • [CPP Client] Auto update topic partitions #6732
    • [CPP Client] Subscription InitialPosition is not correctly set on regex consumers. #6810
    • [CPP Client] Fix message id is always the default value in send callback #6812
    • [CPP Client] Fix message id error if messages were sent to a partitioned topic #6938
    Python Client
    • [Python Client] Fix Python function protobuf missing field#6641
    Pulsar Functions
    • [Functions] Support function with format: Function<I, CompletableFuture>#6684
    • [Functions] Function endpoint admin/v3/functions/{tenant}/{namespace} always returns 404 #6767
    • [Functions] Ensure that all dangling consumers are cleaned up during failures #6778
    • [Functions] Fix localrunner netty dependency issue #6779
    • [Functions] Fix SerDe validation of function's update #6888
    Tiered Storage
    • [Tiered Storage] Extract common SerDe method in tiered storage to managed-ledger module #6533
    • [Tiered Storage] Make SchemaStorage accessible in Offloader #6567
    • [Tiered Storage] Avoid prefetch too much data causing OutOfMemory, when offloading data to HDFS #6717
    Pulsar IO
    • [IO] JDBC sink does not handle null in schema #6848
    Source code(tar.gz)
    Source code(zip)

  • v2.5.1(May 11, 2020)

    2.5.1 — 2020-04-20

    Features

    • PIP-55: Refresh Authentication Credentials #6074
    • Namespace level support offloader #6183
    • Upgrade Avro to 1.9.1 #5938
      • Avro 1.9.1 enables the JSR310 datetimes by default, which might introduce some regression problems if users use generated source code by Avro compiler 1.8.x and contains datetimes fields. It's better to use Avro 1.9.x compiler to recompile.
    • Support unload all partitions of a partitioned topic #6187
    • Supports evenly distribute topics count when splits bundle #6241
    • KeyValue schema support for pulsar sql #6325
    • Bump netty version to 4.1.45.Final #6424
    • Support BouncyCastle FIPS provider #6588
    • Improve Key_Shared subscription message dispatching performance. #6647
    • Add JSR310 logical type conversion. #6704
    • Introduce maxMessagePublishBufferSizeInMB configuration to avoid broker OOM #6178

    Fixes

    Broker
    • [Broker] Fixed NPE occurs when getting partitioned topic stats #6060
    • [Broker] Fixed zero queue consumer message redelivery #6076
    • [Broker] Fixed message redelivery for zero queue consumer while using async api to receive messages #6090
    • [broker] Fixed bug that backlog message that has not yet expired could be deleted due to TTL #6211
    • [Broker] Remove problematic semicolon from conf #6303
    • [Broker] Fixed broker to specify a list of bookie groups #6349
    • [Broker] Fixed create consumer on partitioned topic while disable topic auto creation #5572
    • [Broker] Fix maven broken link #6068
    • [Broker] Fixed java code errors reported by lgtm. #6398
    • [Broker] Fixed memory leak when running topic compaction. #6485
    • [Broker] Fixed admin getLastMessageId return batchIndex #6511
    • [Broker] Fixed topic with one partition cannot be updated #6560
    • [Broker] Fixed negative un-ack messages in consumer stats #5929
    • [broker] Fixed bug that tenants whose allowed clusters include global cannot be created/updated #6275
    • [Broker] Fixed log compaction for flow control/empty topic/last deletion #6237
    • [Broker] Fixed restore clusterDispatchRate policy for compatibility #6176
    • [Broker] Fix some async method problems at PersistentTopicsBase. #6483
    • [Broker] This "earlier" message should be avoided to emit when reset cursor.#6393
    Managed Ledger
    • [Managed Ledger] Fixed consumer received duplicated delayed messages upon restart #6404
    Pulsar Proxy
    • [Proxy] Fixed correct name for proxy thread executor name #6460
    • [Proxy] Fixed logging for published messages #6474
    • [Proxy] Fixed proxy routing to functions worker #6486
    Zookeeper
    • [Zookeeper] Fixed casting in ZooKeeperCache.getDataIfPresent() #6313
    Pulsar Functions
    • [Function] remove future.join() from PulsarSinkEffectivelyOnceProcessor #6361
    Pulsar SQL
    • [SQL] Fixed integration Pulsar SQL test failed #6279
    Security
    • Fixed publish buffer limit does not take effect #6431
    • Fixed the bug of authenticationData is't initialized. #6440
    Pulsar Schema
    • [Schema] Fixed get schema version in HttpLookupService. #6193
    • [Schema] Fixed avro schema decode error ClassCastException in Pulsar Function #6662
    • [Schema] Fixed channel write error handling for send get raw schema request #6650
    Java client
    • [Client] Fixed available permits may be greater than 1 even though queue size is 0. #6106
    • [Client] Fixed broker client tls settings error #6128
    • [Client]Fixed hasMessageAvailable() #6362
    • [Client] Fixed duplicate key to send propertys #6390
    • [Client] fixed deadlock on send failure #6488
    • [Client] Fixed NPE while call getLastMessageId #6562
    • [Client] Fixed the max backoff configuration for lookups #6444
    C++ client
    • [C++] Fixed static linking on C++ lib on MacOS #5581
    • [C++] Fixed memory corruption on ExecutorService destructor #6270
    • [C++] Fixed handling of canceled timer events on NegativeAcksTracker #6272
    • [C++] Fixed for possible deadlock when closing Pulsar client #6277
    • [C++] Fixed Unacked Message Tracker by Using Time Partition on C++ #6391
    • [C++] Fixed Redelivery of Messages on UnackedMessageTracker When Ack Messages . #6498
    Python Client
    • [Python Client]Fixed the enum34 package not found #6401
    Pulsar Websocket
    • [Websocket] Fixed Websocket doesn't set the correct cluster data #6102
    Deployments
    • [Helm] Autorecovery - Fixed could not find or load main class #6373
    • [Helm]: Start proxy pods when at least one broker pod is running #6158

    Enhancements

    Pulsar Broker
    • [Broker] close managed-ledgers before giving up bundle ownership to avoid bad zk-version #5599
    • [Broker] Add timeout to search for web service URLs to avoid web threads getting stuck #6124
    • [Broker] Flush the potential duplicated message when add messages to a batch. #6326
    • [Broker] Avoid getting partition metadata while the topic name is a partition name. #6339
    • [Broker] Fixed create partitioned topic with a substring of an existing topic name. #6478
    • [Broker] Do not retry on authorization failure #6577
    • [Broker]Handle BadVersionException thrown by updateSchemaLocator() #6683
    • [Broker] Expose bookkeeper expose explicit lac configuration in broker.conf #5822
    • [Broker] Allow to enable/disable delayed delivery for messages on namespace #5915
    • [Broker] Prevent creation of regular topic with the same name as existing partitioned topic #5943
    • [Broker] Reset cursor with a non-exists position #6120
    • [Broker] Use fully qualified hostname as default to advertise brokers #6235
    • [broker] Timeout API calls in BrokerService #6489
    • [Broker] Start namespace service and schema registry service before start broker. #6499
    • [Broker] Disable channel auto read when publish rate or publish buffer exceeded #6550
    • [Broker] Resume some servercnx method to public #6581
    • [Broker] Enable get precise backlog and backlog without delayed messages. #6310
    • [Broker] Avoid using same OpAddEntry between different ledger handles #5942
    • [Broker] Clean up closed producer to avoid publish-time for producer #5988
    • [Broker] Support delete inactive topic when subscriptions caught up #6077
    • [Broker] Add a message on how to make log refresh immediately when starting a component #6078
    • [Pulsar Admin] allow tenant admin to manage subscription permission #6122
    • [Broker] Output resource usage rate to log on broker #6152
    • [Broker] Creating a topic does not wait for creating cursor of replicators #6364
    • [Broker] Stop increase unacked messages for the consumer with Exclusive/Failover subscription mode. #6558
    • [Broker] Not allow sub auto create by admin when disable topic auto create #6685
    Zookeeper
    • [Zookeeper] Close ZK before canceling future with exception #6399
    • [ZooKeeper] Upgrade ZooKeeper to 3.5.7 #6329
    Pulsar IO
    • [IO] Adds integration test for RabbitMQ #6033
    Pulsar Functions
    • [Function] remove future.join() from PulsarSinkEffectivelyOnceProcessor #6361
    Stats & Monitoring
    • [Broker] Add backlogSize in topicStats #5914
    • [Broker] Expose lastConsumedTimestamp and lastAckedTimestamp to consumer stats #6051
    • Improve backlogSize stats in the topic. #6700
    Security
    • Validate tokens for binary connections #6233
    • Create namespace failed when TLS is enabled in PulsarStandalone #6457
    • Use more granular permissions for topics #6504
    Pulsar Schema
    • [Schema] Independent schema is set for each consumer generated by topic #6356
    • [Schema] Extract an original avro schema from the "$SCHEMA" field using reflection. If it doesn't work, the process falls back generation of the schema from POJO.#6406
    • [Schema] Add verification for SchemaDefinitionBuilderImpl.java #6405
    Java client
    • [Client] Start reader inside batch result in read first message in batch. #6345
    • [Client] Stop shade snappy-java in pulsar-client-shaded #6375
    • [Client] MultiTopics discovery is broken due to discovery task scheduled twice instead of pendingBatchReceiveTask #6407
    • [Client] Make SubscriptionMode a member of ConsumerConfigurationData #6337
    • [Client] Should set either start message id or start message from roll back duration. #6392
    • [Client] BatchReceivePolicy implements Serializable. #6423
    • [Client] Remove duplicate cnx method #6490
    • [Client] Pulsar Java client: Use System.nanoTime() instead of System.currentTimeMillis() to measure elapsed time #6454
    • [Client] Make tests more stable by using JSONAssert equals #6247
    • [Client] make acker in BatchMessageIdImpl transient #6064
    C++ client
    • [C++] Windows CMake corrections #6336
    • [C++] Avoid calling redeliverMessages() when message list is empty #6480
    • [C++] Improve cpp-client-lib: provide another libpulsarwithdeps.a in dep/rpm #6458
    Python Client
    • [Python Client] Support generate pulsar-client for python3.8#6741
    Deployments
    • [Helm] Explicit statement env-var 'BOOKIE_MEM' and 'BOOKIE_GC' for values-mini.yaml #6340
    • [Helm] Add missing check to dashboard-ingress #6160
    • Make kubernetes yamls for aws operational #6192
    • Ensure JVM memory and GC options are set for bookie #6201
    • Default functionAuthProvider when running in k8s #6203
    Adaptors
    • [Adaptor] Skip javadoc task for pulsar-client-kafka-compact modules #5836
    • [Flink-Connector] Get PulsarClient from cache should always return an open instance #6436
    Source code(tar.gz)
    Source code(zip)

  • v2.4.2(Dec 9, 2019)

    2.4.2 — 2019-12-04

    Fixes

    • Fixed don't set interrupt flag again after catching interrupt exception in Pulsar Client #5643
    • Fixed data is not deleted after expiration due to connected readers #5621
    • Fixed the go client docs missing in website #5595
    • Fixed schema def build error with protobuf schema #5569
    • Fixed docs about reset cursor #5551
    • Fixed repeated initialization of connectorsManager #5545
    • Fixed Functions unnecessarily restart during FunctionRuntimeManager init phase #5527
    • Fixed list non-persistent topics shows the persistent topics #5502
    • Return after triggering callback with empty result #5500
    • Fixed dispatcher skipping delivery of a batch during concurrent replays #5499
    • Fixed reader_listener option for Python API #5487
    • Fixed wrongly report "3600 messages have timed-out" #5477
    • Fixed broken custom auth-provider that uses authenticationData #5462
    • Fixed negative ack tracker constructor sequence #5453
    • Fixed StringSchema static initialization #5445
    • Fixed message corruption on OOM for batch messages #5443
    • Fixed couple functions related integration tests #5434
    • Fixed bug that namespace policies does not take effect due to NPE #5408
    • Fixed race condition : Failed to read-more entries on dispatcher #5391
    • Fixed potential deadlock that can occur in addConsumer #5371
    • Fixed proxy to be able to re-send request body #5361
    • Fixed pulsar can't load the customized SerDe #5357
    • Fixed instability in Pulsar Function window integration test #5337
    • Fixed bk write failure, use signals to resume writing #5322
    • Fixed bad_weak_ptr error when closing producer #5315
    • Fixed typo(massage->message) causing prometheus metrics display error #5299
    • Fixed invalidate cache on zk-cache timeout #5298
    • Fixed memory leak caused by not being executed ClientConnection destructor #5286
    • Fixed producer blocked after send an over size message while batch enabled #5282
    • Fixed race condition while triggering message redelivery after an ack-timeout event #5276
    • Fixed behavior when getting a key from functions state that doesn't exist #5272
    • Fixed Cmake to build _pulsar for osx #5263
    • Fixed client backoff setting does not take effect #5261
    • Fixed memory leak caused by deadline_timer holding object reference #5246
    • Fixed in Message Deduplication that may cause incorrect client/broker interaction #5243
    • Fixed bug that fails to search namespace bundle due to NPE #5191
    • Fixed bug that message delivery stops after resetting cursor for failover subscription #5185
    • Fixed exception type check order bug #5174
    • Fixed spark receiver to account for all the consumer config options #5152
    • Fixed broker fails to start with function worker enabled and broker client using TLS #5151
    • Fixed deadlock when resetting cursor #5136
    • Fixed windowed functions were broken when we changed java function instance to use classloaders #5117
    • Fixed storage size always 0 without subscription #5108

    Enhancements

    • Add subscribe position param for consumer of sink #5532
    • Efficiency improvements for delay delivery tracker #5498
    • Add is_read_compacted to create_reader() in python API #5483
    • Make some common use method of ManagedLedger public #5472
    • Avoid leak on publish failure on batch message #5442
    • Record message failure and avoid exiting from process on publish failure #5441
    • Add support for partitioned topic consumer seek by time #5435
    • Add default loader for latest pyyaml #5432
    • Trim messages which less than mark delete position for message redelivery #5378
    • Make skip all messages async #5375
    • Set default ensemble size to 2 in service conf, to match broker.conf #5339
    • Only seek when reading unexpected entry #5356
    • Don't require both region and endpoint to be specified #5355
    • If cursor is not durable, close dispatcher when all consumers are removed from subscription #5340
    • Disable stickyRead by default #5321
    • Allow to specify delivery delay in C++ client #5317
    • Add debug log + fix thread-factory name #5302
    • Don't attempt to append on read-only cursor ledger #5297
    • Close previous dispatcher when subscription type changes #5288
    • Improve error handling logic for effectively once #5271
    • Upgrade dependencies for security fixes #5232
    • Allow for topic deletions with regex consumers #5230
    • Ensure consumer background tasks are cancelled after subscribe failures #5204
    • Added missing enum value KeyShared on the python wrapper #5196
    • Make some member variables of Dispatcher volatile #5193
    • Ensure getting list of topics for namespace is handled asynchronously #5188
    • Close RateLimiter instance #5155
    • Throw an error if the key was not specified for querying state #5145
    • Allow configuring region aware placement related settings #5100
    • DeleteBookieRack should remove the rack info from zookeeper #5084
    • Use "info" as the default root logger level #5079
    • Modify BatcherBuilder interface and it's subs to implement java.io.Serializable, otherwise java.io.NotSerializableException occurs when we use plusar-flink-connector #5068
    • Don't return DEFAULT_RACK if ZkBookieRackAffinityMapping can't resolve network location #5067
    • Reload zk cache asynchronously #5049
    • Add different cache flags to ConcurrentOpenLongPairRangeSet for size() and toString() #5040
    • Introduce number of threads in perf producer program #5036
    • Completing connector configuration #4999
    • Add checkstyle validation and fix style violations in the common module #4989
    • Trim deleted entries after recover cursor #4987
    • Expose getLastMessageId method in ConsumerImpl #4911
    • Add a documentation page for metrics reference #4910
    • Provide a convenient method for C++ client producer batch container #4885
    • Add schema admin api get schema info with schema version #4877
    • Return Message ID for send for cpp and cgo client #4811
    • Add the schema admin api #4800
    • Clarify how retention interacts with readers #4780
    • Get schema info with topic partition #4751
    • Remove failed stale producer from the connection #4741
    • Update logic for picking active consumer for failover subscription on non-partitioned topic #4604
    Source code(tar.gz)
    Source code(zip)

  • v2.4.1(Sep 4, 2019)

    2.4.1 — 2019-08-30

    Fixes

    • Fixed wrong serialize of batchBuilder in ProducerConfigurationData #4620
    • Fixed SchemaInfo properties losing when registering schema using admin api #4617
    • Fixed wrong topic domain returned by get partitioned topic #4613
    • Fixed pulsar-dashboard wrong count number of bundles #4637
    • Fixed partitionIndex error in consumer for a single partitioned topic #4591
    • Fixed deadlock on get-status rest-api call in broker #4616
    • Fixed deadlock in subscribeAsync demo #4649
    • Fixed C++ client lookup error over HTTP in standalone #4625
    • Fixed NPE at managed-ledger when fetch reader internal-stats #4615
    • Fixed C++ client producer sendAsync() hang when no enough batched message #4657
    • Fixed issue when submitting NAR via file url #4577
    • Renamed C++ logger enum names to avoid conflicts with compiler macros #4664
    • Fixed leaking of pulsar-io-influxdb in distribution #4678
    • Fixed the default port for https and http in admin client #4623
    • Fixed wrong schema delete when checking compatibility #4669
    • Fixed docker/publish.sh doesn't work for pulsar-all and pulsar-standalone images #4705
    • Fixed integration-test failure when execute pip3 install pulsar_client #4754
    • Added log folder in pulsa-function-go back #4736
    • Fixed Pulsar SQL NPE when predicate pushdown for publish_time #4744
    • Fixed redelivered message logic of partition topic #4653
    • Fixed C++ log level names in Log4cxxLogger #4735
    • Fixed go function package not executable #4743
    • Added anonymous role to proxy configuration #4733
    • Added kubernetes namespace to function instance url #4701
    • Fixed go function not parse conf content first #4746
    • Made PulsarKafkaProducer thread safe #4745
    • Fixed messages not get acked if there is not sink topic #4815
    • Fixed bug when function package jar/py/go and runtime is not set #4814
    • Fixed topic loading in rest-api not time out in zooKeeperOperationTimeoutSeconds #4805
    • Fixed default retention policy miss upload to zk #4810
    • Added checking function implements correct interface #4844
    • Fixed retention size policy bug #4825
    • Catch throwable in interceptors of consumer and producer #4860
    • Fixed first position in managedLedger is ahead of the last #4853
    • Fixed concurrent access of uninitializedCursors in ManagedLedgerImpl.asyncOpenCursor #4837
    • Fixed schema not found handling in pulsar-sql #4890
    • Fixed requests not respect no_proxy env variable in dashboard #4867
    • Fixed broken replication msg to specific cluster #4930
    • Fixed dashboard peek parse message error #4918
    • Fixed StructSchema reader cache loading logic #4962
    • Fixed fd leakage in FunctionActioner.downloadFile #4970
    • Fixed python function str bytes convert in example custom_object_function.py #4946
    • Fixed NPE while cleaning up namespace node #4965
    • Fixed the getSchema logic in pulsar proxy #4975
    • Fixed warning by add default loader for latest pyyaml #4974
    • Fixed snappy compressor compile error in pulsar-client-cpp #4972
    • Reinitialize certain components for externally managed runtimes when moving functions #5007
    • Upgraded jackson-databind #5011
    • Fixed the problem of missing dependencies less in docker #5034
    • Fixed duplicated Prometheus TYPE in broker metrics #4183
    • Fixed pulsar sink and source state by init state earlier #5046

    Enhancements

    • Re-factored Component implementation #4541
    • Provided a clock for generating publish timestamp for producers #4562
    • Reduced unnecessary track message calls #4595
    • Supported dynamic configure with escape char #4611
    • Added null check for function/source/sink configs #4627
    • Added delete dynamic config api #4614
    • Made broker replication mtls configuration dynamic #4609
    • Added authorization to function worker REST endpoints #4628
    • Improved and add authorization to function download and upload #4644
    • Allowed consumer retrieve the sequence id that the producer set #4645
    • Added perPartition parameter to partitioned-stats API #4639
    • Supported Pulsar schema for pulsar kafka client wrapper #4534
    • Supported delete and update event for JDBC Sink #4358
    • Cleaned up tests in the presto module #4683
    • Added allowAutoTopicCreation to broker.conf and related configuration #4694
    • Changed to use classloaders to load Java functions #4685
    • Removed fixed server type check in kerberos #4758
    • Changed type of publish_time to timestamp #4757
    • Added read-timeout to admin-request #4762
    • Added checking of deleted schema when adding schema #4731
    • Added strand to C++ client for exclusive control #4750
    • Added support to create partitioned topic with 1 partition #4764
    • Added getters and setters to PulsarService & BrokerService #4709
    • Added configure ack-timeout tick time #4760
    • Added options to rewrite namespace delimiter for pulsar sql #4749
    • Made Pulsar SQL supports pulsar's primitive schema #4728
    • Added basic authentication capabilities to Pulsar SQL #4779
    • Improved SchemaInfoProvider to fetch schema info asynchronously #4836
    • Supported KeyValue schema use AUTO_CONSUME as key/value schema #4839
    • Provided action type for insert in pulsar-io-jdbc #4862
    • Made partition as internal column in pulsar-sql #4888
    • Added option to disable authentication for proxy /metrics #4921
    Source code(tar.gz)
    Source code(zip)

  • v2.4.0(Jul 5, 2019)

    Apache

    2.4.0 — 2019-06-30

    PIPs

    Fixes

    • Fix possible message loss using peer-cluster feature #3426
    • Fix pulsar standalone does not read zk port from conf/standalone.conf #3790
    • Fix some issues of ZkIsolatedBookieEnsemblePlacementPolicy #3917 #3918
    • Fix NPE when unload non-existent topic #3946
    • Fix race condition while deleting global topic #4173
    • Fix deadlock on skip messages #4411
    • Fix NPE when closing batch during a reconnection #4427
    • Fix race condition of read-timeout task in managed ledger #4437
    • Disable sticky read by default #4526
    • Fix race condition between timeout-task and add-call complete #4455

    Enhancements

    • Optimize message replay for large backlog consumer #3732
    • Added support for websocket produce/consume command #3835
    • Added support for TTL config in broker.conf #3898
    • Reduce memory used in ClientCnx for pending lookups #4104
    • Reduce number of hashmap sections for ledger handles cache #4102
    • Added backlog and offloaded size in Prometheus stats #4150
    • Added support for configure the managed ledger cache eviction frequency #4066
    • Added support to avoid payload copy when inserting into managed ledger cache #4197
    • Added support to cache unack-messageId into OpenRangeSet #3819
    • Added support configure static PulsarByteBufAllocator to handle OOM errors #4196
    • Auto refresh new tls certs for jetty webserver #3645
    • Create non-persistent topic by pulsar-admin/rest api #3625
    • Consumer priority-level in Failover subscription #2954

    Security

    • Added support for other algorithms in token auth #4528

    Namespace Policies

    • Added support for tenant based bookie isolation #3933
    • Added support for secondary bookie isolation group at namespace #4458
    • Added support for secondary bookie-isolation-group #4261
    • Added support for replicator rate limit between clusters #4273
    • Disable backlog quota check by default #4320

    Tiered Storage

    • Added support for Pulsar SQL to read data from tiered storage #4045

    Pulsar Schema

    • Added schema versioning to support multi version messages produce and consume #3876 #3670 #4211 #4325 #4548
    • Added TRANSITIVE schema check strategies to support compatibility check over all existing schemas #4214
    • Added schema data validator #4360
    • Added support for delete schema when deleting a topic #3941
    • Added generic record builder #3690

    Pulsar IO

    • Added IO connector for flume source and sink #3597
    • Added IO connector for redis sink #3700
    • Added IO connector for solr sink #3885
    • Hide kafka-connecter details for easy use debezium connector #3825
    • Added IO connector for debezium PostgreSQL source #3924
    • Enhancements for RabbitMQ source configuration #3937
    • Added IO connector for RabbitMQ sink #3967
    • Added IO connector for InfluxDB sink #4017

    Pulsar functions

    • Added support for authentication #3735 #3874 #4198
    • Fix NPE when stats manager not initialized #3891
    • Added async state manipulation methods #3798
    • Fix fail to update functions in effectively-once mode #3993
    • Added labels to function statefulsets and services #4038
    • Added support for set key for message when using function publish #4005
    • Use negative acknowledge to instead ackTimeout #4103
    • Fix backward compatibility with 2.2 auth not working #4241

    Java client

    • Added negative acks #3703
    • Added support for backoff strategy configuration #3848
    • Added support for configure TypedMessageBuilder through a Map conf object #4015
    • Added interceptor for negative ack send #3962
    • Added support for seek operate on reader #4031
    • Store key part of KeyValue schema into pulsar message keys #4117
    • Added interceptor for ack timeout #4300
    • Added support for snappy compression #4259
    • Added support for key based batcher #4435

    Python client

    • Added negative acks #3816
    • Added support for snappy compression #4319

    Go client

    • Added negative acks #3817
    • Added support for go schema #3904
    • Added support for snappy compression #4319
    • Added support for Key_Shared subscription #4465

    C++ client

    • Added negative acks #3750
    • Fix ack timeout when subscribing to regex topic #3897
    • Added support for Key_Shared subscription #4366

    Adaptors

    • In Kafka client wrapper, added some configurations #3753 #3797 #3843 #3887 #3991
    • In Apache Flink connector, allow to specify a custom Pulsar producer #3894 and client authentication #3949
    • In Apache Flink connector, added support for accept ClientConfigurationData, ProducerConfigurationData, ConsumerConfigurationData #4232
    • In Apache Storm connector, fix NPE while emitting next tuple #3991 and some add some enhancements #4280 #4239 #4238 #4236 #4495 #4494

    For a complete list of issues fixed, see

    https://github.com/apache/pulsar/milestone/20?closed=1

    https://github.com/apache/pulsar/releases/tag/v2.4.0

    Source code(tar.gz)
    Source code(zip)

  • v2.3.2(May 31, 2019)

    2.3.2 — 2019-05-30

    Fixes

    • Validate admin operation on topic with authoritative parameter #4270
    • fix bug with source local run #4278
    • fix cannot use size (type _Ctype_int) as type _Ctype_ulong #4212
    • Fix segfault in c++ producer #4219
    • AlwaysCompatible doesn't use AlwaysSchemaValidator in 2.3.1 component/schemaregistry #4181
    • Avoid potentially blocking method during topic ownership check #4190
    • [pulsar-broker]Fix: client-producer can't reconnect due to failed producer-future on cnx cache #4138
    • Removing # TYPE comment from topic metrics in Prometheus #4136
    • For functions metrics in prometheus also remove TYPE #4081
    • Fix: set receive queue size for sinks #4091
    • Fix: Exception when switch cluster from auth enabled to auth disabled #4069
    • Fix update cli source sink #4061
    • Fix connectors nested configs #4067
    • For functions metrics, avoid having HELP #4029
    • Fix Python functions state which is completely broken #4027
    • [issue #3975] Bugfix NPE on non durable consumer #3988
    • Fix: Function auth should ignore exception because it might be anonymous user #4185
    • [pulsar-function] fix worker-stats broken admin-api #4068
    • fix errors in sql doc #4030
    • Fix the swagger files generated by removing troublesome class #4024
    • [pulsar-function] fix broken backward compatibility with v1-namespace while registering function #4224
    • Revert dup consumer and related code #4142
    • [issue 4274][pulsar-io]Add double quotation marks for metrics with remote_cluster #4295

    Enhancements

    • By default, auto configure the size of Bookie read/write cache #4297
    • Upgrade to BookKeeper 4.9.2 #4288
    • [pulsar-function] support bookie authentication from function-worker #4088
    • Optimizing performance for Pulsar function archive download #4082
    • allow users to update output topics for functions and sources #4092
    • improve data-generator source performance #4058
    • [client] Set actual topic name to partitioned consumer #4064
    • ack records in datagenerator print sink #4052
    • [security] Upgrade athenz libraries #4056
    • [python client] Handle subrecords in JsonSchema encoding #4023
    • [Issue 3987][pulsar-broker]Handle config is null when create tenant #4019
    • Add bookkeeper client version constraint #4013
    • Improve error handling for triggering function when there is a schema mismatch #3995
    • [pulsar-broker] add producer/consumer id in error-logging #3961
    Source code(tar.gz)
    Source code(zip)

  • v2.3.1(Apr 19, 2019)

    Fixes

    • Fixed C++ batch acks tracker to evict message from sendList array. This was causing a slowdown in C++ consumers #3618
    • Allow publishing messages >5MB with batching (when they compress to <5MB) #3673 and #3718
    • Use at least 8 threads in Jetty thread pool. This fixed deadlocks in Jetty requests handling #3776
    • Fixed Reader.HasNext() in Go client #3764
    • Fixed increasing consumer permits after ack dedup operation. #3787
    • Set the dedup cursor as "inactive" after recovery #3612
    • Fix read batching message by pulsar reader #3830
    • Fix submit function with code specified via URL #3934
    • Fixed reader reading from a partition #3960
    • Fixed issue with Authorization header missing after client gets redirected #3869

    Enhancements

    • Added producer.flush() on Python #3685
    • Introduced schema builder to define schema #3682
    • Support passing schema definition for JSON and AVRO schemas #3766
    • Exposing InitialPosition setting in Python consumer #3714

    For a complete list of issues fixed, see

    https://github.com/apache/pulsar/milestone/21?closed=1

    Source code(tar.gz)
    Source code(zip)

  • v2.3.0(Feb 21, 2019)

    General

    • Support for schema definitions in the Pulsar Python client library
    • PIP-25: Token based authentication #2888, #3067 and #3089
    • Updated to Apache BookKeeper 4.9.0
    • ZStandard compression codec #3159. Note that when a producer choose Zstd compression, a consumer will need to be at least at version 2.3.0 to be able to correctly receive the messages.
    • Support for Java 11 #3006
    • Added Schema.AUTO_PRODUCE type to allow to publish serialized data and validate it against the topic schema #2685
    • Added Schema.KeyValue to allow for schema to be validated on message keys as well as payloads. #2885
    • Support TLS authentication and authorization in standalone mode #3360
    • When creating namespace, use local cluster by default #3571
    • Tag BookKeeper ledgers created by Pulsar with topic/subscription names for info/debug purposes #3525
    • Enabled sticky reads in BooKeeper reads to increase IO efficiency with read-ahead #3569
    • Several optimization in Pulsar SQL Presto connector (#3128, #3135, #3139, #3144, #3143)
    • Configure Pulsar broker data cache automatically from JVM settings #3573
    • Reuse the SSL context objects #3550
    • Automatic schema update can be disabled through admin interface #2691
    • Support Dead-Letter-Queue from WebSocket proxy #2968
    • Pull-mode for WebSocket proxy #3058
    • Export Jetty stats to Prometheus #2804
    • Added stats for Pulsar proxy #2740
    • Allow subscribers to access subscription admin-api #2981
    • Make brokers read on closest Bookie in a multi-region deployment #3171

    Fixes

    • Fixed deadlock in reusing ZookKeeper event thread #3591
    • In functions log topic appender, don't set producer name #3544
    • When cursor recovery encounters empty cursor ledger, fallback to latest snapshot #3487
    • Fixed C++ regex-consumer when using HTTP service URL #3407
    • Fix race condition: broker not scheduling read for active consumer #3411

    Pulsar IO

    • Added Debezium connector for Change-Data-Capture into Pulsar #2791
    • Added MongoDB connector #3561
    • Added Elastic Search connector #2546
    • Added HBase sink #3368
    • Added Local files connector #2869
    • Report source/sink stats in Prometheus #3261
    • Allow filtering in Twitter Firehose connector #3298
    • Sources/Sinks can be launched using fat jars as well #3166

    Pulsar Functions

    • Added Kubernetes runtime #1950
    • Secrets interface #2826
    • Cleanup subscriptions when deleting functions #3299
    • Add Windowfunction interface to functions api #3324
    • Support for accessing state in Python #2714
    • Support submitting Python functions as wheel file
    • Support submitting Python functions as Zip file with dependencies included #3321
    • Add minimum amount of resources to run setting for functions #3536
    • Fixed the behavior of Function start/stop #3477

    Java client

    • Moved Pulsar v1 client API into separate artifact #3228.
      Applications that are using the Pulsar v1 API, deprecated since 2.0 release, need to update the Maven dependency to use the pulsar-client-1x artifact instead of pulsar-client. Eg. 
      <dependency>
    <groupId>org.apache.pulsar</groupId>
    <artifactId>pulsar-client-1x</artifactId>
    <version>2.3.0</version>
</dependency>
    • Fixed shading issues with Javadoc bundled in client jars by separating the API in a different Maven module #3309
    • Improve Javadocs #3592
    • Support specifying multiple hosts in pulsar service url and web url #3249
    • Automatically discover when partitions on a topic are increased #3513
    • Added Client.getPartitionsForTopic() #2972 (Javadoc)
    • Added Consumer.pauseMessageListener() and Consumer.resumeMessageListener() #2961
    • Removed shading relocations for Circe-checksum and lz4 libraries, to ensure native libraries are correctly loaded when using shaded client lib. #2191

    Python client

    • Fixed Message.properties() #3595

    Go client

    • Added Producer.flush() to flush all outstanding messages #3469
    • Support Consumer.Seek() #3478
    • Added Message.Topic() #3346
    • Allow to specify SubscriptionInitPos option in ConsumerOptions #3588
    • Added TLS hostname verification #3580
    • Allow to link statically against libpulsar.a#3488
    • Expose Producer.LastSequenceID() and Message.SequenceID() #3416

    C++ client

    • Enable batching by default when using sendAsync() #2949
    • Allow to specify schema info in Avro format #3354
    • Added Producer.flush() to flush all outstanding messages #3020
    • Added TLS hostname verification #2475
    • Allow to specify SubscriptionInitialPosition #3567
    • Added Message.getTopicName() #3326
    • Added Cosnsumer.receiveAsync() #3389
    • Build libpulsar.a with all required dependencies #3488
    • Removed Boost from Pulsar API headers #3374

    Adaptors

    For a complete list of issues fixed, see

    https://github.com/apache/pulsar/milestone/18?closed=1

    Downloads at https://pulsar.apache.org/download

    Source code(tar.gz)
    Source code(zip)
Owner
The Apache Software Foundation
The Apache Software Foundation
GitHub https://pulsar.apache.org/
FLiP: StreamNative: Cloud-Native: Streaming Analytics Using Apache Flink SQL on Apache Pulsar

StreamingAnalyticsUsingFlinkSQL FLiP: StreamNative: Cloud-Native: Streaming Analytics Using Apache Flink SQL on Apache Pulsar Running on NVIDIA XAVIER

Timothy Spann 5 Dec 19, 2021
A high available,high performance distributed messaging system.

#新闻 MetaQ 1.4.6.2发布。更新日志 MetaQ 1.4.6.1发布。更新日志 MetaQ 1.4.5.1发布。更新日志 MetaQ 1.4.5发布。更新日志 Meta-ruby 0.1 released: a ruby client for metaq. SOURCE #介绍 Meta

dennis zhuang 1.3k Dec 12, 2022
HornetQ is an open source project to build a multi-protocol, embeddable, very high performance, clustered, asynchronous messaging system.

HornetQ If you need information about the HornetQ project please go to http://community.jboss.org/wiki/HornetQ http://www.jboss.org/hornetq/ This file

HornetQ 245 Dec 3, 2022
High Performance Inter-Thread Messaging Library

LMAX Disruptor A High Performance Inter-Thread Messaging Library Maintainer LMAX Development Team Support Open a ticket in GitHub issue tracker Google

LMAX Group 15.5k Jan 9, 2023
A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues

Nakadi Event Broker Nakadi is a distributed event bus broker that implements a RESTful API abstraction on top of Kafka-like queues, which can be used

Zalando SE 866 Dec 21, 2022
An Open-Source, Distributed MQTT Message Broker for IoT.

MMQ broker MMQ broker 是一款完全开源，高度可伸缩，高可用的分布式 MQTT 消息服务器，适用于 IoT、M2M 和移动应用程序。 MMQ broker 完整支持MQTT V3.1 和 V3.1.1。 安装 MMQ broker 是跨平台的，支持 Linux、Unix、macOS

Solley 60 Dec 15, 2022
Carbyne Stack secret sharing distributed object store

Carbyne Stack Amphora Secret Share Store Amphora is an open source object store for secret shared data and part of Carbyne Stack. DISCLAIMER: Carbyne

Carbyne Stack 6 Dec 1, 2022
SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

SeaTunnel SeaTunnel was formerly named Waterdrop , and renamed SeaTunnel since October 12, 2021. SeaTunnel is a very easy-to-use ultra-high-performanc

The Apache Software Foundation 4.4k Jan 2, 2023
Mirror of Apache Kafka

Apache Kafka See our web site for details on the project. You need to have Java installed. We build and test Apache Kafka with Java 8, 11 and 15. We s

The Apache Software Foundation 23.9k Jan 5, 2023
Mirror of Apache RocketMQ

Apache RocketMQ Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level c

The Apache Software Foundation 18.5k Dec 28, 2022
Apache Camel is an open source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data.

Apache Camel Apache Camel is a powerful, open-source integration framework based on prevalent Enterprise Integration Patterns with powerful bean integ

The Apache Software Foundation 4.7k Dec 31, 2022
Mirror of Apache ActiveMQ

Welcome to Apache ActiveMQ Apache ActiveMQ is a high performance Apache 2.0 licensed Message Broker and JMS 1.1 implementation. Getting Started To hel

The Apache Software Foundation 2.1k Jan 2, 2023
Mirror of Apache ActiveMQ Artemis

ActiveMQ Artemis This file describes some minimum 'stuff one needs to know' to get started coding in this project. Source For details about the modify

The Apache Software Foundation 824 Dec 26, 2022
An XMPP server licensed under the Open Source Apache License.

Openfire About Openfire is a real time collaboration (RTC) server licensed under the Open Source Apache License. It uses the only widely adopted open

Ignite Realtime 2.6k Jan 3, 2023
Kryptonite is a turn-key ready transformation (SMT) for Apache Kafka® Connect to do field-level 🔒 encryption/decryption 🔓 of records. It's an UNOFFICIAL community project.

Kryptonite - An SMT for Kafka Connect Kryptonite is a turn-key ready transformation (SMT) for Apache Kafka® to do field-level encryption/decryption of

Hans-Peter Grahsl 53 Jan 3, 2023
Dagger is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data.

Dagger Dagger or Data Aggregator is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processi

Open DataOps Foundation 238 Dec 22, 2022
Template for an Apache Flink project.

Minimal Apache Flink Project Template It contains some basic jobs for testing if everything runs smoothly. How to Use This Repository Import this repo

Timo Walther 2 Sep 20, 2022
System Of E-Commerce

SystemOfECommerce System Of E-Commerce Simulation of Login the System With Google Account E-mail Verification Simulation Operation of Register To Syst

Barış Aslan 4 Jan 30, 2022
Kafka is not for event sourcing, isn't it? Kafka alone is not an event store, but Kafka and ksqlDB together allow building full-featured event stores. This repository provides a sample of event sourced system that uses Kafka and ksqlDB as event store.

Event Sourcing with Kafka and ksqlDB Introduction Example Domain Event Sourcing and CQRS 101 State-Oriented Persistence Event Sourcing CQRS Advantages

Evgeniy Khyst 54 Dec 28, 2022