Fixes

• logback: Append context tags/keys to the MDC, even when the MDC is empty. This issue was introduced on 2.5.11. Fixed by @ivantopo via #1227. Many thanks to @scheleaap for providing a reproducible!

Improvements

• all: Bump OkHttp version to 4.10.0. Contributed by @azolotko via #1226
• akka-grpc: Propagage context to service implementations when handling unary requests on the Java API. Contributed by @ivantopo via #1229

Fixes

• apm: The Kamon APM reporter was not closing OkHttp's response body, leading to a warning about leaked connections. Fixed by @ivantopo via #1224

Improvements:

• redis, caffeine, and annotation: These modules are now published for Scala 3 as well. Contributed by @hughsimpson via #1123 and #1221.

Fixes:

• logback: #1212 Avoid stackoverflow errors when using Logback 1.3.x / 1.4.x. Fixed by @ivantopo via #1220

Improvements:

• bundle: The bundle now ships with Kanela 1.0.16. Now you can use Kanela on folders that have white spaces on their path.
• apm: Moved to the v2 ingestion API on the Kamon APM reporter. All timestamps on spans (start, finish, and marks) are now reported using the full nanosecond precision available on Kamon Telemetry. Contributed by @ivantopo via #1211

New Features

• cats-io-3: There is a new kamon-cats-io-3 module with automatic instrumentation for Cats Effect 3 fibers :tada: This was a huge effort with tons of support from @m50d, @dispalt, @hughsimpson, @cmcmteixeira, @dpsoft, and @ivantopo via #1196. Thank you so much to everyone involved :heart:

Improvements

• jaeger: Bumped jaeger-thrift dependency to address security issues. Contributed by @mvollmar via #1209
• common: Increase default range for the http.server.connection.lifetime metric to a week. Contributed by @ivantopo via #1204
• jvm: Add memory utilization metrics and support ZGC/Shenandoah. Contributed by @ivantopo via #1203
• scala-future and kafka: Make the tests pass and publish artifacts for Scala 3. Contributed by @hughsimpson via #1208

Improvements

• tapir: The Tapir instrumentation is now compatible with Tapir 1.x. Contributed by @Grandys and @hughsimpson via #1188. Thanks for your help!

Fixes

• core: Ensure that ignored operations are taken into account when doing delayed sampling. Fixed by @ivantopo via #1198

Ignoring Metrics and Spans for Specific Operations

We get asked pretty often: how can we disable sampling for health checks and status endpoints? This Kamon Telemetry version comes with a new setting for that:

kamon.trace.ignored-operations = ["/status", "/health", "/ready"]

This has a few advantages over the groups configuration on the adaptive sampler:

• It works with all samplers, including always and random.
• It comes with a track-metrics-on-ignored-operations setting, which allows disabling metrics for these operations too (the adaptive sampler would only disable sampling but still track metrics for all operations).

Improvements

• instrumentation-common: This module is now published for Scala 3 as well. Special thanks to @vaslabs for going the extra mile on that PR :heart_eyes:. Contributed by @vaslabs via #1191.
• core: Introduce a new setting to ignore operations and disable metrics for them. Contributed by @ivantopo via #1197

Akka Cluster Metrics

This release comes with experimental support for a bunch of Akka Cluster-related metrics! We can break down the metrics into per-state member counts that help you get a general view of the cluster status, and per-member status counts, which let you see exactly which nodes are on each state.

Per-state Member Counts

These metrics track how many cluster members are on each of the possible states in the cluster. The metric names for these states are:

• akka.cluster.members.joining.count
• akka.cluster.members.weakly-up.count
• akka.cluster.members.up.count
• akka.cluster.members.leaving.count
• akka.cluster.members.exiting.count
• akka.cluster.members.down.count
• akka.cluster.members.removed.count
• akka.cluster.members.total.count

All members of the cluster expose these metrics with the number of members they see in each state. Since these metrics are exposed as gauges, you could always make a chart of the max(akka.cluster.members.[state].count) to see how many members are on a given state. For example, for members in the Up state you can see how a cluster starts growing in members:

Per-member Status Counts

The metrics we mentioned above will help you know, for example, that there are 3 members in the Joining state, but what if you want to know whichare those members? That's what the per-member metrics are for. These metrics have a member tag that identifies the member being measured, and the value of the metric tells you the state it is currently in. Here is a breakdown of the possible values:

akka.cluster.member.status

• Joining = 1
• WeaklyUp = 2
• Up = 3
• Leaving = 4
• Exiting = 5
• Down = 6
• Removed = 7

akka.cluster.member.reachability

• Reachable = 0
• Unreachable = 1

The same as with the status metrics, you can plot the max(akka.cluster.members.status) and see what's the latest on each member of the cluster! In the example below, all members are on state 3 (Up):

Enabling Cluster Metrics

This feature isn't fully tested yet so we are not enabling it by default. To enable Akka Cluster Metrics, add this setting to your application.conf file:

kamon.instrumentation.akka.cluster.track-cluster-metrics=yes

New Features

• akka: Publish Akka Cluster metrics for members' status and reachability. Contributed by @ivantopo via #1189

Improvements

• play: We are now using the Play Environment's class loader to create all Kamon-related objects. This ensures that custom classes (like samples or SlowStatementProcessors) can be loaded from the classpath when running on development mode. Contributed by @ivantopo via #1185

Fixes

• bundle: The new kamon-aws-sdk module had a Compile dependency on the Dynalite test container that was brining a bunch of extra classes into the Bundle and messing with user-provided dependencies :facepalm:. Reported by @fggarcia on #1171 and fixed by @ivantopo via #1178
• aws-sdk: Duplicate execution attribute definitions when using more than one AWS SDK Client. Reported by @hhalex on #1173 and fixed by @ivantopo on #1175

:fire: :rotating_light: :fire: Skip this release! This release has a dependency packaging issue described in #1171. Please jump straight to the latest and greatest Kamon Telemetry version!

AWS SDK for Java Support

This release ships automatic instrumentation for the AWS SDK for Java, both 1.x and 2.x! The support is pretty basic: just tracing interactions with the SDK client libraries. Of course, the automatic metrics collected from spans will be there to help, but that's it.

The immediate next steps for the AWS SDK instrumentation are to ensure context propagation through SQS (same as we do for Kafka) and start extracting more request-specific information to enrich those spans. Feel free to stop by our Discord or open an issue to tell us what you are missing!

New Features

• aws-sdk: We are shipping a new kamon-aws-sdk instrumentation module that comes with basic tracing capabilities for the AWS Java SDK. Contributed by @ivantopo via #1167
• alpakka-kafka: We are shipping a new kamon-alpakka-kafka instrumentation module that includes automatic context propagation for ProducerMessage instances. Contributed by @ivantopo via #1168

Improvements:

• system-metrics": This module is now built and published for Scala 3 as well. Contributed by @ornicar via #1155
• otel: The OpenTelemetry reporter now supports the OTEL_RESOURCE_ATTRIBUTES environment variable. Contributed by @hughsimpson via #1157
• otel: The OpenTelemetry exporter dependencies are now bumped to 1.13.0. Contributed by @hughsimpson via #1158

Fixes:

• system-metrics: No network metrics were reported by the System Metrics module when running inside docker containers. Fixed by @jypma via #1165

New Features

• core: This release ships with a new datadog Span propagation scheme that uses the x-datadog-trace-id, x-datadog-parent-id, and x-datadog-sampling-priority headers for all tracing information, exactly as Datadog does it. Contributed by @seglo and @leonwlaw via #1145
• core: It is now possible to filter the metrics sent to a metrics reporter via configuration. You can use kamon.modules.reporter-name.metric-filters to configure an includes/excludes filter with the metrics you want to be exported. Contributed by @ivantopo via #1148
• otel: The OpenTelemetry traces reporter now supports http/protobuf protocol. The entire OTel reporter was rewritten on top of the official OTel exporter and should make it really easy to get into exporting metrics soon :tada:. Contributed by @hughsimpson via #1102

Fixes

• twitter-future: Fixed context propagation issues that were haunting us for ages in the Twitter Future. Many thanks to @dispalt for providing reproducibles for the issue and helping debug this issue. Fixed by @ivantopo via #1147
• akka-http: We were unable to trace HTTP/1 requests on Akka HTTP servers that had HTTP/2 enabled, now we can. Many thanks to @jtjeferreira for writing a failing test for the issue. Fixed by @ivantopo via #1094

Akka gRPC Server Support

Starting on this release, Kamon Telemetry ships with support for tracing server-side Akka gRPC requests implemented with Akka HTTP, Play Framework, and Lagom. This instrumentation is the first step towards full Akka gRPC support and we will soon add:

• gRPC client tracing
• Context propagation across gRPC channels

Improvements

• core: Add the error.type tag to traces, getting us closer to follow the OpenTelemetry semantic conventions. Contributed by @hughsimpson via #1112
• kafka: Prevent the Kafka producer instrumentation from starting traces when there is no current trace on the application. There is a new kamon.instrumentation.kafka.client.tracing.start-trace-on-producer setting to control this behavior. Contributed by @ivantopo via #1125
• akka-grpc: New instrumentation for server-side Akka gRPC. Contributed by @ivantopo via #1119
• akka: #1093 New instrumentation for Akka's Scheduler scheduler.scheduleOnce function, which should bring proper context propagation for pattern.retry and pattern.after as well. Contributed by @ivantopo via #1135
• executors: #641 Avoid reflective access on the executors instrumentation. Contributed by @ivantopo via #1136

Fixes

• influxdb: #1129, #1131 There two bugs regarding encoding and configuration settings on the InfluxDB Authorization Token support introduced on Kamon Telemetry 2.4.8, those are gone now. Fixed by @duese via #1130
• redis: #1082 The Jedis instrumentation wasn't measuring the entire roundtrip time to Redis, only the time it took to send out the commands. Fixed by @ivantopo via #1117
• redis: #1072 Ensure the Akka instrumentation takes the Rediscala actors' exclusions into account. Reported by @jtjeferreira and fixed by @ivantopo via #1122
• akka:" #1116 Stop generating spans for Akka Streams-related actors. Fixed by @ivantopo via #1133
• akka: #1132 Fix module registration warnings for Akka Dispatchers. Fixed by @ivantopo via #1138

Breaking Changes :rotating_light:

• We removed the Play gRPC-specific instrumentation that was included in kamon-play to the new kamon-akka-grpc hooks at a lower level that will work with both, coming with two important changes:
  • If you are not using the Kamon Bundle but instead adding dependencies one by one, you should add the new kamon-akka-grpc dependency for to your classpath
  • There is no custom operation name generator support in kamon-play anymore. All gRPC operations are now instrumented with the tag names defined by the OpenTelemetry semantic conventions and there shouldn't be any need to deviate from that with a custom generator
• The operation tag in Play gRPC requests will no longer have a / prefix, meaning that operations that previously were reported as /ReplyService/SayHello will now show up as ReplyService/SayHello. You might need to update your dashboards and alert conditions to account for that
• Kamon can no longer instrument Executors created by these helper functions from java.util.concurrent.Executors:
  • newSingleThreadExecutor
  • newSingleThreadScheduledExecutor
  • unconfigurableExecutorService
  • unconfigurableScheduledExecutorService

Starting on this release you can use API Token authentication when sending data to InfluxDB using this setting:

kamon.influxdb.authentication {
  token = "your-glorious-token-here"
}

Improvements

• influxdb. Introduced a new authentication.token configuration setting for cases where the InfluxDB reporter needs to authenticate with an API token. Contributed by @ivantopo via #1110.

You can now use the kamon.instrumentation.jdbc.add-db-statement-as-span-tag setting to control the db.statement tag in JDBC calls. This is especially important when your JDBC calls might contain sensitive information that you don't want to share with third parties, like your tracing vendor. The possible setting values are:

• always: (default) will always add the db.statement tag to spans
• prepared: will only add the the db.statement tag when the traced execution was made with a PreparedStatement, which should ensure no placeholder values are leaked to the tracing backend
• never: completely disables adding the db.statement tag to spans

Improvements

• jdbc: control what kind of JDBC statements will be added to the db.statement tag in JDBC spans. Contributed by @ivantopo via #1103

Namespaced Environment Tags

Now you can add namespaced (i.e. tags with dots in their name) environment tags to the kamon.environment.tags settings in your configuration file.

kamon.environment.tags {
  container.id = "4444444444"
  k8s {
    namespace.name = "production"
    cluster {
      name = "strong-cluster"
    }
  } 
}

For example, the settings above in your application.conf file would add the container.id, k8s.namespace.name, and k8s.cluster.name tags to the environment. Most reporters will automatically add all of these tags to metrics and spans before they get sent out.

Improvements

• core: It is now possible to have namespaced tags inside the kamon.environment.tags configuration. Contributed by @ivantopo via #1101.

New Instrumentation

• finagle: Now you can add a new HTTP filter to trace Finagle Client requests. Contributed by @seglo via #1096
• lagom: We are now shipping a Lagom CircuitBreakerMetricsProvider that keeps track of all circuit breakers by default. Contributed by @ihostage via #760

New Features

• runtime-attacher: We are now shipping a new artifact called kamon-runtime-attacher that contains the Kanela agent and the logic to attach it in a running JVM. It is basically the bundle, but without the dependencies. Contributed by @ivantopo via d5ce436
• apm: The Kamon APM reporter now sends environment tags as part for metrics and spans ingestion requests. Contributed by @ivantopo via 2235b79

Fixes

• akka-http: Fix a "requirement failed: HTTP/1.0 responses must not have a chunked entity" error message when using HttpEntity.Default in responses. Fixed by @ivantopo via d8a57912fe9a6a56c6b589311caec9554148dcec

This improvement was originally meant to go out with v2.4.0, but it slipped through the cracks. Sorry!

Improvements

• system: Add a new jvm.threads.states gauge tracking the number of threads on each possible state. Contributed by @getArtemUsername via #1075

Introducing Local Tail Sampling

This release comes with a new way of ensuring you will get traces for slow and failed requests, at least locally: the Local Tail-based Sampler.

The quick summary is that the local tail sampler can keep finished spans in memory for a few seconds and then override their sampling decision if they reach a configurable latency or error count threshold. If you want the full story check out our post about how to keep traces for slow and failed requests.

You can enabled the local tail sampler with these settings:

kamon.trace {
  span-reporting-delay = 20 seconds

  local-tail-sampler {
    enabled = yes
    error-count-threshold = 1
    latency-threshold = 1 second
  }
}

Improvements

• datadog: The OkHttp dependency is now shaded into the Datadog reporter jar to avoid classpath clashes. Contributed by @seglo via #1069
• prometheus: Get easy access to the Prometheus scrape data through PrometheusReporter.latestScrapeData() in case you need it for exposing on your own HTTP server. Contributed by @ivantopo via #1079
• redis: The Jedis instrumentation now works with Jedis 2.x as well. There is still a known issue in #1082 that should get solved soon after. Contributed by @jtjeferreira via #1076
• apm: The Kamon APM reporter now drops old Spans that might be accumulated in the reporters' executor and comes with lower retries and backoff default settings. Contributed by @ivantopo via eeb05960b6fe249fba562042f8285e7b595410db and 49aef33b7c7e344be7f4c632e31bb5c4303d196a

Fixes

• bundle: Remove a misplaced logback.xml from kamon-mongo that was ending up in the bundle. Thanks @PavelPenkov for reporting the issue. Fixed by @ivantopo via d37cb48
• general: @Philippus helped us get a bunch of typos fixed!

Fixes

• play: The GuiceModule we include in the kamon-play instrumentation was not updated to start Kamon's scheduler after the updates we introduced in Kamon 2.3.0. This release adds a new Kamon.initWithoutAttaching(...) API that does the right initialization. This was reported and fixed by @ihostage, then @ivantopo just committed the lines via #1065.

GraalVM

This is the first Kamon release that can be used in applications that target native images with GraalVM :tada:

There was a non-trivial amount of work under the hood to ensure we are not starting any threads on static initializers, and ensuring that everything that requires a thread is delayed until Kamon.init(...) is called, and it works! All the core APIs, the Status Page, and all the reporters are now available to native image users.

The main driver for this change was bringing support for Quarkus, and that's where we are going next! Creating a quarkus extension for Kamon is going to be a bit more challenging than we expected, particularly because we can't use automatic instrumentation in there. Also, Quarkus encourages using SmallRye Context Propagation, but Kamon has its own Context Propagation mechanism in place. It will be a fun ride :).. Join our Discord server if you want to participate on that adventure.

Scala 3

Starting on this release, we are also publishing the Core, Status Page, and Reporter artifacts for Scala 3! We still need to put some work on publishing all of our instrumentation for Scala 3. If you want to give a hand, check out #1064 as a starting point.

New Features

• Support for Scala 3 was brought up by @dpsoft, building on the work started by @bplommer. Thanks a lot for this contribution!
• GraalVM Native Image support was contributed by @ivantopo via #1055

Fixes

• cassandra: Guard against unknown query formats in the Cassandra client instrumentation. Reported as #1058 and fixed by @ivantopo via c5ac1ea.
• akka-http: Handle sub-route rejections on the path matchers instrumentation. Thanks very much to @seglo for providing a reproducible for the issue via #1063. Fixed by @ivantopo on that same #1063 PR.

Deprecation Removals

• We completely removed the Scala Future Chaining instrumentation that was deprecated in v2.1.21. This included a few deprecated functions for creating Spans with the old instrumentation, and a tiny bit of instrumentation for Akka HTTP's FastFuture that is no longer necessary. Those instrumentation pieces were already disabled since v2.2.0, and shouldn't be missed at all after the upgrade. Full diff here: 39191f8e6753ead1b55c5069528dedb1b9052625.

• kamon-redis: Add tracing for the rediscala library. Contributed by @SimunKaracic #1052
• kamon-jdbc: Add SQL parsing to get better operation names. Disabled by default because the performance impact is yet unclear. If you'd like to test it and report back, set kamon.instrumentation.jdbc.parse-sql-for-operation-name to true. Contributed by @Falmarri via #1013

• kamon-opentelemetry: added missing span marks to otel exporter. Contributed by @pnerg via #1047
• kamon-core: Repair W3C span propagation, now with way more spec compliance. Contributed by @SimunKaracic, @the-overengineer and @dguggemos via #1045.
• kamon-system-metrics: Bump oshi-core to latest version. Contributed by @SimunKaracic via #1045
• kamon-caffeine: new module adding instrumentation for Caffeine synchronous caches. Tracing is done automatically, and for cache metrics, KamonStatsCounter is available, which needs to be added manually. Contributed by @SimunKaracic via #1051

• kamon-datadog: Truncate 128-bit trace IDs before reporting to DataDog. Contributed by @dvgica via #1040. Fixes #1032
• kamon-bundle: Ensure that the temporary Kanela agent file extracted by the Bundle will be deleted after the JVM exits. Contributed by @ivantopo via #1036
• kamon-jdbc: Add support for HyperSQL. Contributed by @ivantopo via #1041
• kamon-jdbc: Bump hikariCP to the latest version. Contributed by @simunkaracic via #1039
• kamon-status-page: Try to bind Status Page on a different port if the configured port is busy. Contributed by @ivantopo via #1042
• kamon-redis: Add tracing for the lettuce library. Contributed by @simunkaracic via #1037

:rotating_light: This release changes the instrumentation behavior for Scala Futures :rotating_light:

This release changes the way Kamon instruments Scala Futures, going back to the same behavior we had in Kamon 1.x. This change is going to be transparent for most users because most of them only use the automatic instrumentation, but there might be a small difference in how Spans and Context are related to each other after. Broadly speaking:

• In Kamon 2.x, a Scala Future and all the transformations applied to them (.map/.flatmap/etc) would form a "chain", and changes to the current context in any of the transformations would be carried on to the next transformation, as long as those changes are performed with the functions included in the ScalaFutureInstrumentation companion object. This change brought two challenges:
  • It makes it impossible to cache Future values because the Future itself was tied to the context available when the Future was created for the first time. This would manifest as traces with Spans that don't belong to them.
  • It is possible to leave dirty threads or get unexpected relationships between Spans when not using the helper functions from the ScalaFutureInstrumentation companion object.
• In Kamon 1.x, a Scala Future would only be tied to its "execution context", making it possible to cache Future values and create Spans without concerns of leaving dirty threads, but context updates cannot be propagated through all transformations.

There is more information about the motivations for this change on https://github.com/kamon-io/Kamon/issues/1021.

The Future Chaining instrumentation is still included in Kamon 2.2.0, but it is marked as deprecated and disabled by default. If you need to, you can bring it back with these configuration settings:

kanela.modules {
  executor-service {
    exclude += "scala.concurrent.impl.*"
  }

  scala-future {
    enabled = true
  }
  
  akka-http {
    instrumentations += "kamon.instrumentation.akka.http.FastFutureInstrumentation"
  }  
}

The Future Chaining instrumentation will be fully removed with Kamon 2.3.0.

Changes

• Rollback the Scala Future instrumentation to the same behavior we had in Kamon 1.x. Contributed by @ivantopo via #1035.

New Features

• We got a new Kamon.span function that simplifies creating Spans and automatically handle Scala Futures and CompletionStage instances, ensuring that Spans will only be finished after the async computation is done. Contributed by @ivantopo via #1033.

Deprecations

• We deprecated the Future Chaining instrumentation introduced with Kamon 2.x. There is more info at #1021. Contributed by @ivantopo via #1034.

• kamon-redis: Fix #1027 by changing instrumentation so it does not use Jedis classes during loading, which leads to failures when Jedis is not on the classpath. Contributed by @SimunKaracic and @ivantopo via #1028 .

:warning: :rotating_light: Jump straight to Kamon 2.1.20! This version has a bug when Jedis is not on the classpath!

• kamon-core: Introduce span reporting delay, allowing users to keep Spans in memory for a period of time. Contributed by @ivantopo via #1026
• kamon-akka-http: Fix #1014. Contributed by @SimunKaracic via #1018
• kamon-sqlite: Fix #1008, and update sqlite version. Contributed by @SimunKaracic and @dpsoft via #1023
• kamon-redis: New module, adding tracing for the popular Jedis library. Contributed by @SimunKaracic via #1022

• kamon-prometheus: It's now possible to generate gauge metrics from distribution based metrics. Contributed by @pnerg via #1011
• kamon-core: Add filtering to context tag propagation. Contributed by @pnerg via #1015
• kamon-status-page: Fix status page indicator. Contributed by @the-overengineer via #999.
• kamon-system-metrics: Fix typos in metric description. Contributed via #1007 by @moznion
• kamon-core: Added resetDistribution method to RangleSampler. Contributed by @SimunKaracic via #1017
• kamon-mongo: Update to driver version 4.x. If you're using kamon-mongo (instead of the kamon-bundle), beware. That package now supports only the 4.x.x versions of mongo drivers. For older version of drivers, please use kamon-mongo-legacy. Contributed by @SimunKaracic via #1001
• kamon-mongo-legacy: new instrumentation module for older versions of mongo drivers. https://github.com/kamon-io/Kamon/tree/master/instrumentation/kamon-mongo-legacy Contributed by @SimunKaracic via #1001
• kamon-core: Added flag to disable starting new spans in client instrumentation when there is no current span. Contributed by @SimunKaracic via #1006