OpenTelemetry
Introduction
Collect Jenkins monitoring data through OpenTelemetry.
Architecture
Using the OpenTelemetry Collector, you can use many monitoring backends to monitor Jenkins such as Jaeger, Zipkin, Prometheus, Elastic Observability and many others listed here.
Here are few examples of architecture:
Features
Monitoring of Jenkins jobs execution using distributed tracing principles
- Understand where time is spent, including time spent waiting to schedule the job (time spent in the build queue)
- The time spent in the built queue waiting for a build agent is visualised with the span "Phase : Start"
- Detect increasing time spent in steps like
- Invocations of external systems (e.g. git checkout...)
- Built in integration with Elastic Observability, Jaeger, and Zipkin. Other OpenTelemetry compatible distributed tracing solutions are also supported.
Attributes
Transactions
| Attribute | Description | Type |
|---|---|---|
| ci.pipeline.id | Job name | String |
| ci.pipeline.name | Job name (user friendly) | String |
| ci.pipeline.type | Job type | Enum (freestyle, workflow, multibranch, unknown) |
| ci.pipeline.multibranch.type | Multibranch type | Enum (branch, tag, change_request) |
| ci.pipeline.node.id | Name of the node | String |
| ci.pipeline.run.completed | Is this a complete build? | Boolean |
| ci.pipeline.run.durationMillis | Build duration | Long |
| ci.pipeline.run.description | Build description | String |
| ci.pipeline.run.number | Build number | Long |
| ci.pipeline.run.result | Build result | Enum (aborted, success, failure, not_build and unstable) |
| ci.pipeline.run.url | Build URL | String |
| ci.pipeline.run.user | Who triggered the build | String |
| ci.pipeline.parameter.sensitive | Whether the information contained in this parameter is sensitive or security related. | Boolean |
| ci.pipeline.parameter.name | Name of the parameter | String |
| ci.pipeline.parameter.value | Value of the parameter | String |
Spans
| Attribute | Description | Type |
|---|---|---|
| jenkins.pipeline.step.name | Step name (user friendly) | String |
| jenkins.pipeline.step.type | Step name | String |
| jenkins.pipeline.step.id | Step id | String |
| jenkins.pipeline.step.node.label | Labels attached to the node | String |
| git.branch | Git branch name | String |
| git.repository | Git repository | String |
| git.username | Git user | String |
| jenkins.url | Jenkins URL | String |
| jenkins.computer.name | Name of the node | String |
Metrics on Jenkins health indicators
| Metrics | Description |
|---|---|
| ci.pipeline.run.active | Gauge of active jobs |
| ci.pipeline.run.launched | Job launched |
| ci.pipeline.run.started | Job started |
| ci.pipeline.run.completed | Job completed |
| ci.pipeline.run.aborted | Job aborted |
| jenkins.queue.waiting | Number of waiting items in queue |
| jenkins.queue.blocked | Number of blocked items in queue |
| jenkins.queue.buildable | Number of buildable items in queue |
| jenkins.queue.left | Total count of left items |
| jenkins.queue.time_spent_millis | Total time spent in queue by items |
| jenkins.agents.total | Number of agents |
| jenkins.agents.online | Number of online agents |
| jenkins.agents.offline | Number of offline agents |
Jenkins metrics can be visualised with any OpenTelemetry compatible metrics solution such as Prometheus or Elastic Observability
Getting started
- Setup an OpenTelemetry endpoint such as the OpenTelemetry Collector
- Install the Jenkins OpenTelemetry plugin
- Configure the Jenkins OpenTelemetry plugin navigating to the "Manage Jenkins / Configure System" screen
- In the OpenTelemetry section define
- "OTLP GRPC Endpoint": the hostname and port of the OpenTelemetry GRPC Protocol (OTLP GRPC) endpoint, typically an OpenTelemetry Collector or directly an Observability backend that supports the OTLP GRPC protocol
- "Header Authentication" : name of the authentication header if header based authentication is used.
- "Bearer Token Authentication": Bearer token when using header based authentication.
- Visualization: the backend used to visualize job executions as traces.
- Elastic Observability
- Jaeger
- Zipkin
- Custom Observability backend for other visualisation solution
Screenshots
Sample of traces collected for various flavors of pipelines
Scripted Pipeline
Scripted pipeline status page
node {
stage('Prepare') {
echo("Prepare")
}
stage('Build') {
git 'https://github.com/jglick/simple-maven-project-with-tests.git'
sh "mvn -Dmaven.test.failure.ignore=true clean package"
}
stage('Post Build') {
echo("this is the post build phase")
}
}
Scripted pipeline visualized with Elastic Observability
Scripted pipeline visualized with Jaeger
Scripted pipeline visualized with Zipkin
Declarative Pipeline
pipeline {
agent any
stages {
stage('Build') {
steps {
git 'https://github.com/jglick/simple-maven-project-with-tests.git'
sh "mvn -Dmaven.test.failure.ignore=true clean package"
}
post {
success {
echo "success"
}
}
}
}
}
Scripted Pipeline with Error
node {
stage('Prepare') {
echo("Prepare")
}
stage('Build') {
git 'https://github.com/jglick/simple-maven-project-with-tests.git'
sh "mvn -Dmaven.test.failure.ignore=true clean package"
}
stage('Post Build') {
error 'Fail'
}
}
Scripted Pipeline with Parallel Step
node {
stage('Prepare') {
echo("Prepare")
}
stage('Build') {
git 'https://github.com/jglick/simple-maven-project-with-tests.git'
sh "mvn -Dmaven.test.failure.ignore=true clean package"
}
stage('Parallel Post Build') {
parallel parallBranch1: {
echo("this is the post build parallel branch 1")
} ,parallBranch2: {
echo("this is the post build parallel branch 2")
echo("this is the post build parallel branch 2")
}
}
}
Freestyle Job
Ideas
- Collect labels of build agents
- Detect outages caused by upgrades. Report on the version of the plugin of each plugin being used as a step
Configuration as code
This plugin supports configuration as code. Add to your yaml file:
unclassified:
openTelemetry:
authentication: "noAuthentication"
endpoint: "otel-collector-contrib:4317"
exporterIntervalMillis: 60000
exporterTimeoutMillis: 30000
ignoredSteps: "dir,echo,isUnix,pwd,properties"
observabilityBackends:
- elastic:
kibanaBaseUrl: "http://localhost:5601"
- jaeger:
jaegerBaseUrl: "http://localhost:16686"
- customObservabilityBackend:
metricsVisualisationUrlTemplate: "foo"
traceVisualisationUrlTemplate: "http://example.com"
- zipkin:
zipkinBaseUrl: "http://localhost:9411/"
See the jcasc folder with various samples.
For more details see the configuration as code plugin documentation: https://github.com/jenkinsci/configuration-as-code-plugin#getting-started
Contributing
Refer to our contribution guidelines
LICENSE
Licensed under Apache Software License 2, see LICENSE
These are the configurations I've made. and on elk server i have only installed apm-server all are 8.4.2 version. The jenkins overview dashboard isnt loading
1.1k Sep 16, 2022
665 Jan 8, 2023
2.3k Dec 27, 2022
22 Aug 28, 2022
522 Dec 24, 2022
33 Oct 25, 2022
300 Dec 4, 2022
54 Nov 22, 2022
3 Sep 2, 2021
6 Dec 22, 2022
1 Feb 16, 2022
4 Apr 6, 2022
1 Jan 18, 2022
7.8k Jan 5, 2023
5 Jun 19, 2022
708 Dec 23, 2022
939 Dec 23, 2022
3.7k Dec 30, 2022
20.4k Jan 5, 2023