SAURON - VERSION AND DEPLOYMENT TRACKER
DESCRIPTION
Sauron, the all seeing eye! It is a service to generate automated reports and track migrations, changes and dependency versions for backend services also report on known CVE and security issues. A detailed description can be found in the internal RFC document.
COMPONENTS
Sauron Service is segregated into a few components that are described below:
-
Sauron Core: Sauron common library to be used by all sauron plugins. More details here
-
Sauron Service: Sauron entrypoint controller described in more details in this README
-
Sauron Plugin System: Sauron has an embedded plugin support that allows anyone to introduce its own logic without the need to rebuild, regenerate and stop/restart Sauron Service. More details in further sections.
-
Elasticsearch Cluster: Sauron uses Elasticsearch as a data storage. It is currently deployed in AWS and can be accessed here.
-
Kibana: Elasticsearch data can be explored using the built-in Kibana instance that can be accessed here.
-
Dependencytrack: Sauron includes an instance of Dependencytrack, a platform that allows organizations to identify and reduce risk from the use of third-party and open source components.
ARCHITECTURE OVERVIEW
RUNNING
Sauron can be deployed using Docker and Docker-Compose. It provides a Dockerfile that can be built using the following commands:
Build Sauron project using maven and ship it to a docker image:
make
Start Sauron stack and load the local plugin repository
docker-compose -f docker-compose.yml --compatibility up
This command will start three components:
- elasticsearch: http://localhost:9200
- kibana: http://localhost:5601
- sauron-service: http://localhost:8080
Note: Since Sauron needs your maven, gradle and nodejs configuration, the docker-compose.yml creates a volume for each configuration folder/file. If there is no configuration already created, please create one before running the command above. For more details please refer to docker-compose.yaml > Volumes
Docker
In order to run Sauron with you predefined configuration using docker, use the command below:
docker run \
-e SPRING_CONFIG_LOCATION="/sauron/config/sauron-service.yml" \
-e M2_HOME="/usr/share/maven" \
-e SPRING_PROFILES_INCLUDE="local" \
-e SPRING_CLOUD_CONFIG_ENABLED="false" \
--mount type=bind,source=${PWD}/sauron-service/docker/config/sauron-service.yml,destination=/sauron/config/sauron-service.yml,readonly \
--mount type=bind,source=${PWD}/sauron-service/plugins,destination=/sauron/plugins \
--mount type=bind,source=${HOME}/.m2,destination=/root/.m2 \
--mount type=bind,source=${HOME}/.gradle,destination=/root/.gradle \
--mount type=bind,source=${HOME}/.npmrc,destination=/root/.npmrc \
--mount type=bind,source=${HOME}/.ssh,destination=/root/.ssh,readonly \
--name=sauron \
-p 8080:8080 \
ghcr.io/freenowtech/sauron/sauron-service:latest
If you need to use a specific version, please refer to Sauron Packages
CONFIGURATION
Sauron configuration can be set via application.properties file. The file path can be provided using the environment variable:
- SPRING_CONFIG_LOCATION: it must contain the path pointing to the local file e.g.
/path/to/config/my-properties.propertiesor/path/to/config/my-properties.yaml
Sauron supports Spring Cloud Config Server as a configuration provider. In order to set Config Server url, please use the environment variable below:
- SPRING_CLOUD_CONFIG_URI: it must contain the URL pointing to the remote repository e.g.
https://my-repository.com/my-config
For a Sauron configuration file example, please refer to sauron-service.yml
USAGE
Sauron Elasticsearch Index Template
Before start using Sauron, it is import to define the index template that will be used by Elasticsearch to create Sauron's index. It can be done by running the command line below:
elasticsearch/sauron-template.sh
elasticsearch/dependencies-template.sh
This template increases the number of fields, since usually the amount of dependencies and thus the amount of fields is huge, and some other minor optimizations.
Triggering Sauron Service
Sauron Service provides a REST api that allows one to trigger a new build. The detailed parameters can be found in its swagger documentation. Once a new build has been triggered, Sauron's pipeline will run applying all plugins pre-configured. The output will be stored in elasticsearch and can be queried afterwards using the Kibana Installation.
For more details, check in the next section how the plugin system works.
A build request example can be found below:
curl --verbose --location --request POST 'http://localhost:8080/api/v1/build' \
--header 'Content-Type: application/json' \
--data-raw '{
"serviceName": "MyService",
"repositoryUrl": "https://github.com/gazgeek/springboot-helloworld.git",
"commitId": "41c7823dddbef43680a0726ccea0631519b9d3c1",
"buildId": "2b8caa6c-8b55-4b57-b654-5a00d519f409",
"owner": "Sauron",
"eventTime": 1586962717770,
"rollback": false,
"returnCode": 0,
"environment": "production",
"release": "0.0.1",
"user": "sauron-user",
"dockerImage": "helloworld:0.0.1",
"platform": "K8S"
}'
Visualizing Sauron Data
As mentioned before, Sauron stack uses Elasticsearch + Kibana to respectively store and visualize data. Once the Sauron build is done, the information will be available and can be queried using Kibana.
In order to do that, a index pattern must be created in Kibana using this HOWTO.
Once it is done, you are able to use your data. Sauron provides a default Kibana Dashboard available here. Import it and voilà! Enjoy your data!
Kibana Dashboard Example
SAURON PLUGIN SYSTEM
Sauron has an embedded plugin system that allows anyone to insert its own business logic to extract information during the building/deploy process. It uses the PF4J which is a plugin framework written in Java, and provides a nice interface to implement an integration in your service.
During the startup Sauron loads all available plugins, and updates them every 5 minutes using the pre-defined plugin repository (Local or Artifactory). For more details please refer to sauron-service.yml.
Official Plugins
console Output
Prints the DataSet content to sysout.
data-sanitizer
Sanitizes the data before being processed by Sauron pipeline.
git-checkout
Checkout the project source code.
dependency-checker
Extracts dependencies information in CycloneDX and insert in DataSet.
dependencytrack-publisher
Publishes the dependencies to our internal Dependency Track instance.
maven-report
Retrieves information from pom.xml file.
elasticsearch-output
Stores the DataSet content into Elasticsearch.
protocw-checker
Checks whether a service is using protoc, and the protoc wrapper.
logs-report
Tells if your service has any logs on elasticsearch. You can configure indexes and time range to search for logs.
kubernetesapi-report
Allows Sauron to query Kubernetes API to retrieve the annotations and labels assigned to the resources, specified in the configuration, and stores the values into the DataSet.
sonarapi-report
Query Sonar API to retrieve your service related data, like Code Coverage, and stores into the DataSet.
thanosapi-report
Query Thanos API to retrieve your service related data(like RPM, Circuit Breaker), and stores into the DataSet.
readme-checker
Checks whether a service has or not a README.md file in its root folder.
bcrypt-passwordencoder-checker
Checks whether a service might be using Bcrypt to encode the passwords in the API
jaegerapi-report
Query Jaeger API to extract tracing information
Creating a New Sauron Plugin
Sauron provides a maven archetype in order to create a standard skeleton of a new plugin. To use that follow the steps below.
Install maven archetype
Checkout the sauron-plugin-archetype project:
$ git clone https://github.com/freenowtech/sauron.git
Install the archetype locally and add it to the local archetype catalog:
$ cd sauron-plugin-archetype
$ mvn clean install
Create new plugin skeleton
Generate the new plugin using the installed archetype:
$ mvn org.apache.maven.plugins:maven-archetype-plugin:3.1.2:generate
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0)
Choose archetype:
...
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : sauron-plugin-archetype
Choose archetype:
1: remote -> com.freenow.sauron:sauron-plugin-archetype (Sauron Plugin Archetype)
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1
[INFO] Using property: groupId = com.free-now.sauron.plugins
Define value for property 'artifactId': my-plugin
[INFO] Using property: version = 0.0.1-SNAPSHOT
[INFO] Using property: package = com.freenow.sauron.plugins
Define value for property 'className': MyPlugin
Confirm properties configuration:
groupId: com.free-now.sauron.plugins
artifactId: my-plugin
version: 0.0.1-SNAPSHOT
package: com.freenow.sauron.plugins
className: MyPlugin
Y: :
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: sauron-plugin-archetype:0.0.1-SNAPSHOT
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.free-now.sauron.plugins
[INFO] Parameter: artifactId, Value: my-plugin
[INFO] Parameter: version, Value: 0.0.1-SNAPSHOT
[INFO] Parameter: package, Value: com.freenow.sauron.plugins
[INFO] Parameter: packageInPathFormat, Value: com/freenow/sauron/plugins
[INFO] Parameter: package, Value: com.freenow.sauron.plugins
[INFO] Parameter: version, Value: 0.0.1-SNAPSHOT
[INFO] Parameter: groupId, Value: com.free-now.sauron.plugins
[INFO] Parameter: className, Value: MyPlugin
[INFO] Parameter: artifactId, Value: my-plugin
[INFO] Project created from Archetype in dir: /Users/sergio/Documents/sauron/my-plugin
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.041 s
[INFO] Finished at: 2019-04-08T15:10:44+02:00
[INFO] ------------------------------------------------------------------------
It will generate a new maven project that follows the structure below:
Fill in the MyPlugin.java class with your desired logic:
package com.freenow.sauron.plugins;
import com.freenow.sauron.model.DataSet;
import com.freenow.sauron.properties.PluginsConfigurationProperties;
import org.pf4j.Extension;
@Extension
public class MyPlugin implements SauronExtension
{
@Override
public DataSet apply(PluginsConfigurationProperties properties, DataSet input)
{
// PLUGIN LOGIC
return input;
}
}
Configuring your new Plugin
In order to provide extra configuration to your plugin, refer to CONFIGURATION SECTION. After your configuration has been added, it will be available to be used by your plugin in PluginsConfigurationProperties object. See below an example of how you can use the configuration properties:
The following configuration provides an url to my-plugin plugin generated in step above:
sauron:
plugins:
my-plugin:
url: https://my-plugin.com
So in order to retrieve this configuration uses the following java code:
@Extension
public class MyPlugin implements SauronExtension
{
@Override
public DataSet apply(PluginsConfigurationProperties properties, DataSet input)
{
properties.getPluginConfigurationProperty("my-plugin", "url").ifPresent(url -> System.out.println(url) );
return input;
}
}
Documenting your Plugin
Sauron Plugin Archetype provides a template of a README that must be filled to document what your plugin does, inputs, outputs and possible configuration.
Deploying your new Plugin
Once the developing and testing process has been done, you can deploy a new version of your plugin, using the pre-defined plugin repository (Local or Artifactory). For more details please refer to sauron-service.yml.
The plugin reloading process runs every 5 minutes. To force a reloading use the /api/v1/reload method.
Check Swagger Documentation for more information.
Using your new Plugin
Once your plugin has been deployed and loaded by Sauron Service you can use it in a pipeline which will be then applied to new deployments. Refer to Sauron Usage for detailed information.
5 Feb 2, 2022
3 Dec 15, 2022
2 Sep 21, 2021
2 Mar 3, 2022
5.2k Dec 31, 2022
2 Jan 26, 2022
6 Oct 4, 2022
2 Jun 13, 2022
3 Sep 8, 2022
8 Dec 22, 2022
19 Nov 21, 2022
5 Oct 19, 2022
1 Jan 31, 2022
1 Feb 12, 2022
7 Dec 15, 2022
18 Dec 29, 2022
5.9k Dec 29, 2022
1.2k Dec 18, 2022
389 Dec 31, 2022