Tackle Data-intensive Validity Analyzer

Overview

Tackle-DiVA (Data-intensive Validity Analyzer)

Tackle-DiVA is a command-line tool for data-centric application analysis. It imports a set of target application source files (*.java/xml) and provides following analysis result files.

  • Service entry (exported API) inventory
  • Database inventory
  • Transaction inventory
  • Code-to-Database dependencies (call graphs)
  • Database-to-Database dependencies
  • Transaction-to-Transaction dependencies
  • Transaction refactoring recommendation.

Overview

Prerequisites

  • Docker runnable environment (e.g. RHEL, Ubuntu, macOS)

Getting Started

Supposed that tackle-diva is cloned to /tmp,

  1. Build docker image for diva.
$ cd /tmp/tackle-diva/
$ docker build . -t diva
  1. Prepare source codes of target Java applications whose Java framework is supported in DiVA, such as DayTrader7 and TradingApp.
$ cd /tmp
$ git clone https://github.com/WASdev/sample.daytrader7.git
  1. Move to tackle-diva/distrib/bin/ directory, and execute diva_docker command attaching directory full path.
$ cd /tmp/tackle-diva/distrib/bin/
$ ./diva_docker /tmp/sample.daytrader7/
  1. Check tackle-diva/distrib/output directory and confirm analysis result files
$ ls /tmp/tackle-diva/distrib/output
contexts.yml            transaction.json        transaction_summary.dot
database.json           transaction.yml         transaction_summary.pdf

Analysis Results

An output directory for storing analysis result files:

  • contexts.yml: Extracted application service entries
  • database.json: Extracted database tables
  • transaction.json/yml: Extracted transactions that expresses as a sequence of service entry, call graphs, SQLs
  • transaction_summary.dot/pdf: Extracted database-to-database/transaction-to-transaction dependencies and recommended transaction refactoring.

Supported Java Frameworks

The other frameworks to be supported.

Comments
  • Performance problem with diva-windup

    Performance problem with diva-windup

    Currently diva-windup rule (DivaLauncher.perform) is very slow. We guess it's due to our construction of janus-graph. The arquillian test log (*) repeats the following message while running the diva rule.

    Sep 14, 2021 9:33:05 AM org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3 execute
    WARNING: Query requires iterating over all vertices [()]. For better performance, use indexes
    Sep 14, 2021 9:33:06 AM org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3 execute
    WARNING: Query requires iterating over all vertices [()]. For better performance, use indexes
    Sep 14, 2021 9:33:06 AM org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3 execute
    WARNING: Query requires iterating over all vertices [()]. For better performance, use indexes
    ...
    

    Diva standalone runs faster in few seconds.

    Here are steps to run the arquillian test.

    $ git clone https://github.com/WASdev/sample.daytrader7.git
    $ cd tackle-diva
    $ ./gradlew publishToMavenLocal -x test
    $ cd windup
    $ JAVA_HOME=<path-to-java8-runtime> mvn verify -Dtest=DaytraderTest
    

    *) https://github.com/konveyor/tackle-diva/runs/3597051370

    opened by akihikot 5
  • DOA for Oracle

    DOA for Oracle

    Implement preliminary function to support PL/SQL, which is a format used in Oracle DBMS.

    • As a first step, DOA structurally parses PL/SQL using our parser and provides assessment results for automatic/manual conversion.
    • In the next step, DOA performs minimum conversion of source SQLs so that psql accepts converted files without errors.
      • #72
      • #75
    doa 
    opened by shinsa82 2
  • Post-release minor fixes for DiVA-DOA v2.0.0

    Post-release minor fixes for DiVA-DOA v2.0.0

    • Remove usage of Makefile
    • Introduce template-based document/code generation (using jinja2)
    • Change name and structure of the generated files
    • Update documents to match the implementation
    • Brush up documents
    doa 
    opened by shinsa82 1
  • Added JitPack repo and the proper diva-core dependency

    Added JitPack repo and the proper diva-core dependency

    • added the Jitpack registry
    • declared the diva-core dependency following the Jitpack's naming convention

    Consider Jitpack works also if the version tag references a commit or a tag so we can use it in a very productive way: right now it's using the snapshot version based on the latest main branch.

    opened by mrizzi 1
  • Implement DOA frontend in Python and DOA for Oracle (#87, #90)

    Implement DOA frontend in Python and DOA for Oracle (#87, #90)

    • Implement DOA frontend in Python and package it in python-doa-cli directory.
    • Enhance DOA for Oracle including semantic verification and package it in python-doa-lib directory.
    • Deploy the built DOA Docker image on quay.io.
    doa 
    opened by shinsa82 0
  • Rewrite DOA frontend scripts in Python

    Rewrite DOA frontend scripts in Python

    Current DOA frontend (that invokes DOA Docker images) are implemented as bash scripts. Using bash to implement complex CLI, including handling of CLI options, whitespace in arguments and exceptions, is error-prone.

    Thus we plan to rewrite them as a Python script, that adds Python to dependencies to use DOA.

    kind/feature doa 
    opened by shinsa82 0
  • fix #80. version of DOA bumped to 2.2.1

    fix #80. version of DOA bumped to 2.2.1

    commit 1383ebb6fd77643a802897199cb5be60739d5201 Author: Shin Saito [email protected] Date: Wed May 11 20:10:39 2022 +0900

    fix #80. version of DOA bumped to 2.2.1
    

    commit bb34f3c304c524841878b6979a73fd526c7e89c0 Author: Shin Saito [email protected] Date: Wed Apr 20 04:48:22 2022 +0000

    update document
    

    commit 0655f9a1e0247ac21021be6153d5b91370d5a56d Author: Shin Saito [email protected] Date: Wed Apr 20 04:43:06 2022 +0000

    update document
    

    commit 3aa1599e0214a02ec1d6940d8196d9b3355bec8d Author: Shin Saito [email protected] Date: Wed Apr 20 04:09:54 2022 +0000

    update document
    

    commit ac0a661e0cf3a19ecacde22bc219b3f3c9ffb22b Author: Shin Saito [email protected] Date: Wed Apr 20 04:06:25 2022 +0000

    update document
    

    commit 7f6cb4bbfcd961da08b1501c6300ca17d7e7cb3a Author: Shin Saito [email protected] Date: Tue Apr 19 16:09:00 2022 +0000

    update document
    

    commit 2078edff2745783ce8a26b0cba0462ee96e6d934 Author: Shin Saito [email protected] Date: Tue Apr 19 16:00:09 2022 +0000

    add rew document on Oracle dialects
    
    kind/bug doa 
    opened by shinsa82 0
  • DOA fails when the input directory containing whitespaces is specified

    DOA fails when the input directory containing whitespaces is specified

    DOA (run-doa.sh) fails when -f is specified and the input directory contains whitespaces:

    Example:

    bash ./run-doa.sh -l oracle -f "/tmp/a b/plsql-example"
    

    It makes docker run, especially mounting of the input directory, fail.

    See for -f and -l usage: https://github.com/konveyor/tackle-diva/blob/main/doa/README-Oracle.md

    kind/bug doa 
    opened by shinsa82 0
  • Tackle DiVA Docker image should be hosted on quay.io with other Konveyor images

    Tackle DiVA Docker image should be hosted on quay.io with other Konveyor images

    Users of Tackle DiVA should not have to clone a repo and build their own Docker image. We should host an "official" image on quay.io with all of the other Konveyor tools.

    I propose that we call the image tackle-diva and change the documentation and scripts to point to: quay.io/konveyor/tackle-diva for all commands related to running DiVA stand-alone. This should simplify its use.

    Along with this the README.md and ./distrib/bin/diva_docker script should be updated to use the new image.

    opened by rofrano 0
  • Replace `System.out.println` with logging framework

    Replace `System.out.println` with logging framework

    While testing the Windup Diva addon, it turned out the console output has a lot of System.out.println [link to one sample below] entries that would be great if moved into a logging framework at (my suggestion) DEBUG level.

    https://github.com/konveyor/tackle-diva/blob/ea087c515c50b1832abc73dc8b14dfeddf4895d5/core/src/main/java/io/tackle/diva/analysis/JPAAnalysis.java#L110

    opened by mrizzi 0
  • Documentation needed to interface with tackle-diva from Tackle Inventory

    Documentation needed to interface with tackle-diva from Tackle Inventory

    Hi , @YasuKatsuno @akihikot I'm onboarding to the Tackle project along with @jortel and @mansam right now.

    (Note: I'm not sure if tackle-diva is meant to be integrated with tackle-inventory or if it's more tied to windup. Looking for information on this)

    We are working to learn how the Tackle App Inventory can interface with other tools in the Tackle suite, and trying to define the requirements of the App Inventory data model.

    It would be helpful to have a reference doc on the public interface of tackle-diva (TD) so that we can get an idea of how TD this would tie in with Tackle Inventory, and what changes might be needed to the Tackle Inventory data model:

    • Expected inputs / outputs of TD (examples would be useful)
    • Methods for invoking TD (REST? CLI? Java API? Something else?)
    • Is there a Quay.io container auto-build set up so we can consume TD easily running as a Pod on OpenShift?
    • Is there a deploy YAML available with a deployment definition for TD and any required auxiliary OpenShift resources (e.g. namespaces, services, routes). This would let us quickly play around with TD using other info provided. We may be able to assist here if needed.

    Do documents detailing the public interface of TD exist? Even some simple examples would be useful to get started.

    cc @rromannissen @jwmatthews @PhilipCattanach

    documentation 
    opened by djwhatle 3
Owner
Konveyor
Open Source Migration Tools for Kubernetes
Konveyor
An extensible multilanguage static code analyzer.

PMD About PMD is a source code analyzer. It finds common programming flaws like unused variables, empty catch blocks, unnecessary object creation, and

PMD 4.1k Jan 2, 2023
:coffee: SonarSource Static Analyzer for Java Code Quality and Security

Code Quality and Security for Java This SonarSource project is a code analyzer for Java projects. Information about the analysis of Java features is a

SonarSource 976 Jan 5, 2023
Java bytecode static analyzer

This project is abandoned and unlikely will be supported in future HuntBugs 0.0.11 New Java bytecode static analyzer tool based on Procyon Compiler To

Tagir Valeev 302 Aug 13, 2022
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Apache Gobblin Apache Gobblin is a highly scalable data management solution for structured and byte-oriented data in heterogeneous data ecosystems. Ca

The Apache Software Foundation 2.1k Jan 4, 2023
A static analyzer for Java, C, C++, and Objective-C

Infer Infer is a static analysis tool for Java, C++, Objective-C, and C. Infer is written in OCaml. Installation Read our Getting Started page for det

Facebook 13.7k Dec 28, 2022
An extensible multilanguage static code analyzer.

PMD About PMD is a source code analyzer. It finds common programming flaws like unused variables, empty catch blocks, unnecessary object creation, and

PMD 4.1k Jan 2, 2023
:coffee: SonarSource Static Analyzer for Java Code Quality and Security

Code Quality and Security for Java This SonarSource project is a code analyzer for Java projects. Information about the analysis of Java features is a

SonarSource 976 Jan 5, 2023
The IK Analysis plugin integrates Lucene IK analyzer into elasticsearch, support customized dictionary.

IK Analysis for Elasticsearch The IK Analysis plugin integrates Lucene IK analyzer (http://code.google.com/p/ik-analyzer/) into elasticsearch, support

Medcl 14.6k Jan 5, 2023
Java bytecode static analyzer

This project is abandoned and unlikely will be supported in future HuntBugs 0.0.11 New Java bytecode static analyzer tool based on Procyon Compiler To

Tagir Valeev 302 Aug 13, 2022
Bank Statement Analyzer Application that currently runs in terminal with the commands: javac Application.java java Application [file-name].csv GUI coming soon...

Bank Statement Analyzer Application that currently runs in terminal with the commands: javac Application.java java Application [file-name].csv GUI coming soon...

Hayden Hanson 0 May 21, 2022
Java - Packet Analyzer Application based on Java, Networking and Swing UI

Network-Packet-Tracer-using-Java Java - Packet Analyzer / Sniffing System Application based on Java, Networking and Swing UI Java - Packet Analyzer Ap

Muhammad Asad 6 Feb 3, 2022
Redstone analyzer and debugger

Minecraft Multimeter Mod Install and use This mod runs on Fabric, so make sure you have that installed. Download the Fabric API if you don't have it a

Antoni Spaanderman 1 Dec 28, 2021
A scientific charting library focused on performance optimised real-time data visualisation at 25 Hz update rates for data sets with a few 10 thousand up to 5 million data points.

ChartFx ChartFx is a scientific charting library developed at GSI for FAIR with focus on performance optimised real-time data visualisation at 25 Hz u

GSI CS-CO/ACO 386 Jan 2, 2023
A scientific charting library focused on performance optimised real-time data visualisation at 25 Hz update rates for data sets with a few 10 thousand up to 5 million data points.

ChartFx ChartFx is a scientific charting library developed at GSI for FAIR with focus on performance optimised real-time data visualisation at 25 Hz u

GSI CS-CO/ACO 385 Dec 30, 2022
Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

Firehose - Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

Open DataOps Foundation 279 Dec 22, 2022
Infinispan is an open source data grid platform and highly scalable NoSQL cloud data store.

The Infinispan project Infinispan is an open source (under the Apache License, v2.0) data grid platform. For more information on Infinispan, including

Infinispan 1k Dec 31, 2022
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An

Cossack Labs 1.6k Dec 29, 2022
Dex : The Data Explorer -- A data visualization tool written in Java/Groovy/JavaFX capable of powerful ETL and publishing web visualizations.

Dex Dex : The data explorer is a powerful tool for data science. It is written in Groovy and Java on top of JavaFX and offers the ability to: Read in

Patrick Martin 1.3k Jan 8, 2023
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

Apache Zeppelin Documentation: User Guide Mailing Lists: User and Dev mailing list Continuous Integration: Contributing: Contribution Guide Issue Trac

The Apache Software Foundation 5.9k Jan 8, 2023