ThirdEye is an integrated tool for realtime monitoring of time series and interactive root-cause analysis.

Related tags

Miscellaneous thirdeye
Overview

Introduction to ThirdEye

Build Status license

ThirdEye is an integrated tool for realtime monitoring of time series and interactive root-cause analysis. It enables anyone inside an organization to collaborate on effective identification and analysis of deviations in business and system metrics. ThirdEye supports the entire workflow from anomaly detection, over root-cause analysis, to issue resolution and post-mortem reporting.

What is it for? (key features)

Online monitoring and analysis of business and system metrics from multiple data sources. ThirdEye comes batteries included for both detection and analysis use cases. It aims to minimize the Mean-Time-To-Detection (MTTD) and Mean-Time-To-Recovery (MTTR) of production issues. ThirdEye improves its detection and analysis performance over time from incremental user feedback.

Detection

  • Detection toolkit based on business rules and exponential smoothing
  • Realtime monitoring of high-dimensional time series
  • Native support for seasonality and permanent change points in time series
  • Email alerts with 1-click feedback for automated tuning of detection algorithms

Root-Cause Analysis

  • Collaborative root-cause analysis dashboards
  • Interactive slice-and-dice of data, correlation analysis, and event identification
  • Reporting and archiving tools for anomalies and analyses
  • Knowledge graph construction over time from user feedback

Integration

  • Connectors for continuous time series data from Pinot, Presto, MySQL and CSV
  • Connectors for discrete event data sources, such as holidays from Google calendar
  • Plugin support for detection and analysis components

What isn't it? (limitations)

ThirdEye maintains a dedicated meta-data store to capture data sources, anomalies, and relationships between entities but does not store raw time series data. It relies on systems such as Pinot, Presto, MySQL, RocksDB, and Kafka to obtain both realtime and historic time series data.

ThirdEye does not replace your issue tracker - it integrates with it. ThirdEye supports collaboration but focuses on the data-integration aspect of anomaly detection and root-cause analysis. After all, your organization probably already has a well-oiled issue resolution process that we don't want to disrupt.

ThirdEye is not a generic dashboard builder toolkit. ThirdEye attempts to bring overview data from different sources into one single place on-demand. In-depth data about events, such as A/B experiments and deployments, should be kept in their respective systems. ThirdEye can link to these directly.

Getting Involved

Documentation

Detailed documentation can be found at ThirdEye documentation for a complete description of ThirdEye's features.

Comments
  • Feature: Adds notification support via slack

    Feature: Adds notification support via slack

    Description

    The current notification options available only around email and jira. This PR adds support for notifications and subscription based alerting via slack.

    Upgrade Notes

    Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

    • [ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

    Does this PR fix a zero-downtime upgrade introduced earlier?

    • [ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

    Does this PR otherwise need attention when creating release notes? Things to consider:

    • New configuration options
    • Deprecation of configurations
    • Signature changes to public methods/interfaces
    • New plugins added or old plugins removed
    • [ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

    Release Notes

    If you have tagged this as either backward-incompat or release-notes, you MUST add text here that you would like to see appear in release notes of the next release.

    If you have a series of commits adding or enabling a feature, then add this section only in final commit that marks the feature completed. Refer to earlier release notes to see examples of text

    Documentation

    If you have introduced a new feature or configuration, please add it to the documentation as well.

    opened by venkatvghub 1
  • Docs: fix syntax issues

    Docs: fix syntax issues

    Description

    Addresses syntax errors in the documentation reported in #9.

    Upgrade Notes

    Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

    • [ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

    Does this PR fix a zero-downtime upgrade introduced earlier?

    • [ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

    Does this PR otherwise need attention when creating release notes? Things to consider:

    • New configuration options
    • Deprecation of configurations
    • Signature changes to public methods/interfaces
    • New plugins added or old plugins removed
    • [ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

    Release Notes

    N/A

    Documentation

    N/A

    opened by bocytko 0
  • Feature: Google Authentication

    Feature: Google Authentication

    Description

    This PR adds a custom authentication via google authentication

    Upgrade Notes

    Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

    • [ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

    Does this PR fix a zero-downtime upgrade introduced earlier?

    • [ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

    Does this PR otherwise need attention when creating release notes? Things to consider:

    • New configuration options
    • Deprecation of configurations
    • Signature changes to public methods/interfaces
    • New plugins added or old plugins removed
    • [ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

    Release Notes

    If you have tagged this as either backward-incompat or release-notes, you MUST add text here that you would like to see appear in release notes of the next release.

    If you have a series of commits adding or enabling a feature, then add this section only in final commit that marks the feature completed. Refer to earlier release notes to see examples of text

    Documentation

    If you have introduced a new feature or configuration, please add it to the documentation as well.

    enhancement 
    opened by venkatvghub 0
  • a fault in quick start

    a fault in quick start

    hi, I found this in quick start: git clone https://github.com/apache/incubator-pinot.git cd incubator-pinot/thirdeye chmod +x install.sh run-frontend.sh run-backend.sh reset.sh ./install.sh

    there is no thirdeye in incubator-pinot. So I git clone thirdeye directly. I'd like to ask the relationships between incubator-pinto and thirdeye. And will the thirdeye be maintained?

    opened by drepig 0
  • Fix errors when installing from scratch

    Fix errors when installing from scratch

    Hey Team,

    We encountered the following issues when trying to install from scratch from the code base here

    Root-causes for initial failures:

    • frontend-maven-plugin used in thirdeye-frontend/pom.xml needed to use a later version;
    
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.508 s
[INFO] Finished at: 2021-11-15T16:22:17+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.9.1:install-node-and-yarn (install node and npm) on project thirdeye-frontend: Could not download Node.js: Got error code 404 from the server. -> [Help 1]
Cannot download node - 404 · Issue #952 · eirslett/frontend-maven-plugin
    • missing logger dependency in thirdeye-dashboard/pom.xml which was resulting in error when running run-frontend.sh
    Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
	at org.apache.pinot.thirdeye.dashboard.ThirdEyeDashboardApplication.<clinit>(ThirdEyeDashboardApplication.java:193)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 1 more

    • https://dzone.com/articles/adding-slf4j-your-maven

    • https://mvnrepository.com/artifact/org.slf4j/slf4j-simple/1.7.32

    • run-frontend.sh was pointing to the wrong jar file (i.e. the one in thirdeye-dashboard directory whereas all the classes in the in the project are compiled in thirdeye-dist/target/thirdeye-dist-1.0.0-SNAPSHOT

    this was resulting in Could not find or load main class org.apache.pinot.thirdeye.dashboard.ThirdEyeDashboardApplication

    • install.sh needs to use mvn clean install -DskipTests to ensure there are no conflicts with existing target directories in case of multiple installs from scratch

    These errors are being addressed in pull request Update Maven dependencies by shrechak · Pull Request #11 · project-thirdeye/thirdeye - - https://github.com/project-thirdeye/thirdeye/pull/11

    opened by ateneva 0
  • Update Maven dependencies

    Update Maven dependencies

    Description

    Update dependency versions in pom.xml to successfully package the jar.

    Upgrade Notes

    Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

    • [ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

    Does this PR fix a zero-downtime upgrade introduced earlier?

    • [ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

    Does this PR otherwise need attention when creating release notes? Things to consider:

    • New configuration options
    • Deprecation of configurations
    • Signature changes to public methods/interfaces
    • New plugins added or old plugins removed
    • [ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

    Release Notes

    If you have tagged this as either backward-incompat or release-notes, you MUST add text here that you would like to see appear in release notes of the next release.

    If you have a series of commits adding or enabling a feature, then add this section only in final commit that marks the feature completed. Refer to earlier release notes to see examples of text

    Documentation

    If you have introduced a new feature or configuration, please add it to the documentation as well.

    opened by shrechak 0
  • Edit on GitHub is outdated since repo split

    Edit on GitHub is outdated since repo split

    The "Edit on GitHub" link is outdated and links to the pinot repo, e.g. in the About page to pinot/.../about.rst. It's likely that the docs have not been published since the repo was split.

    There are minor syntax errors I spotted as well:

    ThirdEye UI Mocks:

    Screen Shot 2021-09-25 at 00 55 53

    About ThirdEye:

    Screen Shot 2021-09-25 at 00 56 02

    opened by bocytko 0
  • Custom Config reader for reading configs and secrets from environment variables

    Custom Config reader for reading configs and secrets from environment variables

    Description

    This PR adds a custom config reader to read configurations from environment variables.

    Why is this needed

    Specifically in container environments like docker and kubernetes, we might have multiple environments and also possibly want to control secrets and configurations using kubernetes secrets / configmaps. In addition, we do not want to commit these credentials into code as well. So, in essence, instead of directly file mounting the data-sources-config.yml as a secret, we can selectively define some of the configs like hostname, port, username, password etc as kubernetes or docker secrets. E.g.: Our default data-sources-config.yml might look like this:

    ...
properties:
      MySQL:
        - db:
            mysql: jdbc:mysql://?somemysqlhost:3306/?thirdeye
          user: thirdeye_user
          password: thirdeye_password
...

    With this PR, we can do the following:

    - className: org.apache.pinot.thirdeye.datasource.sql.SqlThirdEyeDataSource
    properties:
      MySQL:
        - db:
            mysql: jdbc:mysql://?$MYSQL_HOSTNAME:?$MYSQL_PORT/?$THIRDEYE_DATABASE
          user: ?$MYSQL_USERNAME
          password: ?$MYSQL_PASSWORD

    In essence, for all the above configs, add ?$<ENVIRONMENT_VARIABLE_NAME> to ensure we can pass these as environment variables And then the kubernetes deployment will look like the following:

    ...
- name: MYSQL_USERNAME
   valueFrom:
      secretKeyRef:
        key: MYSQL_USERNAME
        name: thirdeye
- name: MYSQL_PASSWORD
   valueFrom:
      secretKeyRef:
        key: MYSQL_PASSWORD
        name: thirdeye
 ....

    Upgrade Notes

    Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

    • [ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

    Does this PR fix a zero-downtime upgrade introduced earlier?

    • [ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

    Does this PR otherwise need attention when creating release notes? Things to consider:

    • New Config Options : As mentioned earlier, the new data store configs are provided above
    • TODO: Update the thirdeye documentation to provide examples for the same
    • [X] Yes (Please label this PR as release-notes and complete the section on Release Notes)

    Release Notes

    • This Section needs a bit of update based on updates to the documentation.

    Documentation

    TODO

    enhancement 
    opened by venkatvghub 3
Owner
null
GitHub
Function allowing you to find duplicate in an array in linear time and constant space

Floyd's tortoise and hare Function allowing you to find duplicate in an array in linear time and constant space ! Floyd's method is a pretty amazing,

null 7 Mar 29, 2022
Serverless Reference Architecture for Real-time File Processing

Serverless Reference Architecture: Real-time File Processing The Real-time File Processing reference architecture is a general-purpose, event-driven,

AWS Samples 436 Oct 7, 2022
A Java to iOS Objective-C translation tool and runtime.

J2ObjC: Java to Objective-C Translator and Runtime Project site: https://j2objc.org J2ObjC blog: https://j2objc.blogspot.com Questions and discussion:

Google 5.9k Dec 29, 2022
The open-source Java obfuscation tool working with Ant and Gradle by yWorks - the diagramming experts

yGuard yGuard is an open-source Java obfuscation tool. With yGuard it is easy as pie ( ?? ) to configure obfuscation through an extensive ant task. yG

yWorks GmbH 265 Jan 2, 2023
A simplified and multi-functional tool for spigot developers

A simplified and multi-functional tool for spigot developers. There are dozens of features you can use in it, and it is completely open source code. hCore supports all versions from 1.8.x to 1.18.2. Also you can find all these APIs usages from here.

Hakan Kargın 57 Jan 1, 2023
Helper tool to calculate the price for a given amount of users within a graduated pricing model.

Lithic Hi Lithic, This small piece of software is a coding exercise I asked some candidates to solve and is based on the pricing model of Atlassian pr

Stefan Antal 1 Feb 27, 2022
A tool to help Barcelonian people get various immigration-related appointments via the Cita previa process

NIEBot A tool to help Barcelonian people get various immigration-related appointments via the Cita previa process. Requirements A Windows or Linux PC.

Conor Rynne 49 Dec 10, 2022
Sauron, the all seeing eye! It is a service to generate automated reports and track migrations, changes and dependency versions for backend services also report on known CVE and security issues.

SAURON - VERSION AND DEPLOYMENT TRACKER DESCRIPTION Sauron, the all seeing eye! It is a service to generate automated reports and track migrations, ch

FREENOWTech 20 Oct 31, 2022
A calculator that performs various operations such as addition, subtraction, multiplication and division of positive and negative values

A calculator that performs various operations such as addition, subtraction, multiplication and division of positive and negative values The calculator also does percentages, square roots and squares

Andrey Fabricio 1 Jan 31, 2022
A Local implementation of a java library functions to create a serverside and clientside application which will communicate over TCP using given port and ip address.

A Local implementation of java library functions to create a serverside and clientside application which will communicate over TCP using given port and ip address.

Isaac Barry 1 Feb 12, 2022
Java Notes & Codes for better understanding and it contains all the funtions with examples and also added Cheat Sheet for Revision

Java Notes & Codes for better understanding and it contains all the funtions with examples and also added Cheat Sheet for Revision...

Ujjawal Singh 1 Nov 30, 2022
Automatically discover and tag PII data across BigQuery tables and apply column-level access controls based on confidentiality level.

Automatically discover and tag PII data across BigQuery tables and apply column-level access controls based on confidentiality level.

Google Cloud Platform 18 Dec 29, 2022
Make Slack and Facebook Bots in Java.

JBot Make bots in Java. JBot is a java framework (inspired by Howdyai's Botkit) to make Slack and Facebook bots in minutes. It provides all the boiler

Ram 1.2k Dec 18, 2022
API gateway for REST and SOAP written in Java.

Membrane Service Proxy Reverse HTTP proxy (framework) written in Java, that can be used as an API gateway as a security proxy for HTTP based integrati

predic8 GmbH 389 Dec 31, 2022
An extensible Java framework for building XML and non-XML streaming applications

Smooks Framework This is the Git source code repository for the Smooks Project. Build Status Building Pre-requisites JDK 8 Apache Maven 3.2.x Maven gi

Smooks Framework 353 Dec 1, 2022
TChart Simple and fast charts

TChart Simple and fast charts. Current version Beta 0.9.1 Preview Import jitpack.io gradle allprojects

null 30 Sep 20, 2022
A lightweight staff chat plugin for BungeeCord and Spigot.

A lightweight staff chat plugin for BungeeCord and Spigot with 2-way Discord chat support and many other great features.

null 4 Mar 30, 2022
Masterful Machinery is a mod about creating customisable and complex multiblocks machines.

Masterful Machinery Masterful Machinery is a mod about creating customisable and complex multiblocks machines. Everything in the mod is customisable w

null 23 Nov 12, 2022
This service checks the Co-WIN public API at a specific interval and send update to users specified telegram bot.

COVID VACCINE TELEGRAM BOT USING SPRING BOOT This application is a covid vaccine slot notifier via telegram bot. This application uses public CO-WIN A

Hardeek Sharma 6 Oct 4, 2022