DEPRECATION NOTICE
This repo is no longer actively maintained. While it should continue to work and there are no major known bugs, we will not be improving Heroic or releasing new versions.
Heroic
A scalable time series database based on Bigtable, Cassandra, and Elasticsearch. Go to https://spotify.github.io/heroic/ for documentation.
This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.
Install
Docker
Docker images are available on Docker Hub.
$ docker run -p 8080:8080 -p 9091:9091 spotify/heroic
Heroic will now be reachable at http://localhost:8080/status.
In production it's advised to use a tagged version.
Configuration
For help on how to write a configuration file, see the Configuration Section of the official documentation.
Heroic has been tested with the following services:
- Cassandra (
2.1.x,3.5) when using metric/datastax. - Cloud Bigtable when using metric/bigtable.
- Elasticsearch (
7.x) when using metadata/elasticsearch or suggest/elasticsearch. - Kafka (
0.8.x) when using consumer/kafka.
Developing
Building from source
In order to compile Heroic, you'll need:
- A Java 11 JDK
- Maven 3
- Gradle
The project is built using Gradle:
# full build, runs all tests and builds the shaded jar
./gradlew build
# only compile
./gradlew assemble
# build a single module
./gradlew heroic-metric-bigtable:build
The heroic-dist module can be used to produce a shaded jar that contains all required dependencies:
./gradlew heroic-dist:shadowJar
After building, the entry point of the service is com.spotify.heroic.HeroicService. The following is an example of how this can be run:
./gradlew heroic-dist:runShadow <config>
which is the equivalent of doing:
java -jar $PWD/heroic-dist/build/libs/heroic-dist-0.0.1-SNAPSHOT-shaded.jar <config>
Building with Docker
$ docker build -t heroic:latest .
This is a multi-stage build and will first build Heroic via a ./gradlew clean build and then copy the resulting shaded jar into the runtime container.
Running heroic via docker can be done:
$ docker run -d -p 8080:8080 -p 9091:9091 -v /path/to/config.yml:/heroic.yml spotify/heroic:latest
Logging
Logging is captured using SLF4J, and forwarded to Log4j.
To configure logging, define the -Dlog4j.configurationFile=<path> parameter. You can use docs/log4j2-file.xml as a base.
Testing
We run tests with Gradle:
# run unit tests
./gradlew test
# run integration tests
./gradlew integrationTest
or to run a more comprehensive set of checks:
./gradlew check
This will run:
- unit tests
- integration tests
- Checkstyle
- Coverage Reporting with Jacoco
It is strongly recommended that you run the full test suite before setting up a pull request, otherwise it will be rejected by Travis.
Full Cluster Tests
Full cluster tests are defined in heroic-dist/src/test/java.
This way, they have access to all the modules and parts of Heroic.
The JVM RPC module is specifically designed to allow for rapid execution of integration tests. It allows multiple cores to be defined and communicate with each other in the same JVM instance.
- See AbstractClusterQueryIT
- JVM-based JvmClusterQueryIT
- gRPC-based GrpcClusterQueryIT
Code Coverage
There's an ongoing project to improve test coverage. Clicking the above graph will bring you to codecov.io, where you can find areas to focus on.
Bypassing Validation
To bypass automatic formatting and checkstyle validation you can use the following stanza:
// @formatter:off
final List<String> list = ImmutableList.of(
"Welcome to...",
"... The Wild West"
);
// @formatter:on
To bypass a FindBugs error, you should use the @SupressFBWarnings annotation.
@SupressFBWarnings(value="FINDBUGS_ERROR_CODE", justification="I Know Better Than FindBugs")
public class IKnowBetterThanFindbugs() {
// ...
}
Module Orientation
The Heroic project is split into a couple of modules.
The most critical one is heroic-component. It contains interfaces, value objects, and the basic set of dependencies necessary to glue different components together.
Submodules include metric, suggest, metadata, and aggregation. The first three contain various implementations of the given backend type, while the latter provides aggregation methods.
heroic-core contains the com.spotify.heroic.HeroicCore class which is the central building block for setting up a Heroic instance.
heroic-elasticsearch-utils is a collection of utilities for interacting with Elasticsearch. This is separate since we have more than one backend that needs to talk with elasticsearch.
Finally there is heroic-dist, a small project that depends on all module. Here is where everything is bound together into a distribution — a shaded jar. It also provides the entry-point for services, namely com.spotify.heroic.HeroicService or through an interactive shell com.spotify.heroic.HeroicShell. The shell can either be run standalone or connected to an existing Heroic instance for administration.
Contributing
Guidelines for contributing can be found here.
![[WIP] Add distribution support to spotify100 and the ability to fetch distribution from bigTable.](https://avatars.githubusercontent.com/u/14685441?v=4)
1.7k Dec 17, 2022
4.8k Dec 26, 2022
9.9k Jan 1, 2023
367 Oct 11, 2022
46.3k Jan 10, 2023
495 Dec 1, 2022
230 Dec 9, 2022
322 Dec 29, 2022
286 Dec 7, 2022
12.3k Jan 1, 2023
3.6k Jan 2, 2023
4 Dec 28, 2022
363 Dec 30, 2022
6.9k Jan 9, 2023
4.6k Dec 30, 2022
11.4k Jan 5, 2023
1k Dec 14, 2022