A near-real-time Mesh Join Algorithm Implementation provided with a Complete Data warehouse for METRO

Overview

Mesh Join Algorithm and Data Warehouse

A complete Mesh-Join Algorithm Implementation as provided in the paper R-MESHJOIN . This is demonstrated by the use of a single Data warehouse that is a simplified version of a WallMart-style Shopping and Retail Business. The Repository provides code for both the Warehouse and the MESHJOIN algorithm Code.

Introduction

Problem Statement

This project aims to create a centralized Data Warehouse for the analysis and querying of various business aspects of the store. The warehouse needs to operate in real-time, meaning that any changes in the ODS stores should almost immediately be reflected in the warehouse for querying. This is done by creating an ETL Service that can run on a server and can monitor the ODS to check for updates and insertions. The service then fetches and prepares data to be aggregated and then loaded into the warehouse. We implement such a service using the Java Programming Language in this project.

Scope

The scope of the project is limited to only the transactional data of the ODS. The transactional data is provided in a small file that amounts to 10,000 transactions. The project also provides us with a “MasterData” Table, which contains information on the products catalogues, and their prices. The ETL Service, therefore, needs to perform a real-time join operation to calculate the total sale of each transaction, and store them in the warehouse. This is the primary goal of the project.

Data Warehouse

Schema

image

The schema includes a single Fact table called TRANSCATIONS. The Schema design follows the Star Schema template, and is made on the basis of the ODS Schema provided below:

image

Usage

File Structure

  • Java Project

    • Contains the complete source code of the Eclipse and IntelliJ Java Project
    • The code is found in the src/com/dwh folder
    • The main function is found in the meshJoin.java file
  • MySQL Project

    • Contains files for the MySQL Workbench project
    • This project file creates a model that is used throughout the project
  • SQL Files

    • Contains all the files for SQL Functions

    • createDW.sql

      • Run this SQL File to implement the Data Warehouse
      • This file is executed and tested on MySQL Workbench
    • queriesDW.sql

      • Contains all the OLAP queries for the project
    • mainSQL.sql

      • Contains all the OLAP queries for the project
      • Also contains addtional queries used for testing and debugging
    • Transaction_and_MasterData_Generator.sql

      • Generates Data for the MESHJOIN algorithm to work
      • This is the ODS schema that is used to load the warehouse into MySQL Projects

Code Compilation

  1. Using the MySQL Workbench File provided, Open the Project
  2. You can either use Forward Enigneering feature, or execute DWH Script on the connected database
  3. Execute the TransactionGenerator.sql File provided
  4. Execute the createDW.sql File provided to create the warehouse
  5. Load the Eclipse/IDEA Project into the IDE
  6. Compile and Build the program
  7. If a JDBC error occurs, then add the provided jdbc-connection.jar file into the project build settings
  8. When the program starts it's execution, it asks for use input on the number of transactions already loaded into the warehouse
    • Enter 0 if it is the first time running the script, and we need to load fresh data
    • Enter the number of transactions already loaded, if new data has been added into the tables
  9. The Menu asks to perform ETL or quit
    • Press 1 for ETL, and the script executes showing the progress
    • Press 0 for Exiting
You might also like...

Hashids algorithm v1.0.0 implementation in Java

Hashids.java A small Java class to generate YouTube-like hashes from one or many numbers. Ported from javascript hashids.js by Ivan Akimov What is it?

Dec 29, 2022

Search API with spelling correction using ngram-index algorithm: implementation using Java Spring-boot and MySQL ngram full text search index

Search API with spelling correction using ngram-index algorithm: implementation using Java Spring-boot and MySQL ngram full text search index

Search API to handle Spelling-Corrections Based on N-gram index algorithm: using MySQL Ngram Full-Text Parser Sample Screen-Recording Screen.Recording

Dec 4, 2021

Person Apri development in real time

Digital Innovation: Expert class - Desenvolvendo um sistema de gerenciamento de pessoas em API REST com Spring Boot Nesta live coding vamos desenvolve

Nov 11, 2021

This app corrects your sitting posture and provides feedback in real time in conjunction with the app. A sensor of 31 cells detects your posture to get better life-wellness

This app corrects your sitting posture and provides feedback in real time in conjunction with the app. A sensor of 31 cells detects your posture to get better life-wellness

Notichair 실시간 자세분석 및 교정 스마트체어 🏆 상명대학교 PRIME 경진대회 수상 🏆 요구사항 31-cell sensor (mdxs-16-5610) 목차 1. 소개 프로젝트 내용 소개 2. 개발 환경 사전 설정 및 환경 구축 3. 기능 Sensors Ap

Jan 15, 2022

corrects your sitting posture and provides feedback in real time in conjunction with the app. A sensor of 31 cells detects your posture to get better life-wellness

corrects your sitting posture and provides feedback in real time in conjunction with the app. A sensor of 31 cells detects your posture to get better life-wellness

Notichair 실시간 자세분석 및 교정 스마트체어 🏆 상명대학교 PRIME 경진대회 수상 🏆 요구사항 31-cell sensor (mdxs-16-5610) 목차 1. 소개 프로젝트 내용 소개 2. 개발 환경 사전 설정 및 환경 구축 3. 기능 Sensors Ap

Jan 15, 2022

LOQUI - Real-time chat application built using Apache Kafka, Java, Spring Boot, SockJS and React

LOQUI - Real-time chat application built using Apache Kafka, Java, Spring Boot, SockJS and React

LOQUI is a simple real-time chat application that demonstrates how to use Apache Kafka as a message broker along with Java, Spring Boot and React on the front-end

Jun 5, 2022

A real-time ray tracer in Java, accelerated on heterogeneous hardware using TornadoVM

A real-time ray tracer in Java, accelerated on heterogeneous hardware using TornadoVM

Java Ray Tracer for TornadoVM Description This project aims to build a real-time ray tracer in Java, accelerated on heterogeneous hardware using Torna

May 23, 2022

Source code of course - Building Real-Time REST APIs with Spring Boot

springboot-blog-rest-api Learn how to build real-time REST APIs with Spring Boot by building a complete Blog App. Source code of Popular Building Real

Jan 6, 2023

A complete and performing library to highlight text snippets (EditText, SpannableString and TextView) using Spannable with Regular Expressions (Regex) for Android.

A complete and performing library to highlight text snippets (EditText, SpannableString and TextView) using Spannable with Regular Expressions (Regex) for Android.

Highlight A complete and performing library to highlight text snippets (EditText/Editable and TextView) using Spannable with Regular Expressions (Rege

Dec 22, 2022
Owner
null
Warehouse management REST API with Java & Spring Boot

Spring Warehouse Spring Warehouse is a Java Spring Boot REST API that has the purpose of managing products and articles in your warehouse. This projec

Josep Bigorra 40 Dec 15, 2022
Presti 5 Nov 19, 2022
Welcome to the EHS robotics club's GitHub repository, this will also be used as our primary community center and means of communication. Also be sure to join our remind for on the go updates @EHSFTC21

NOTICE This repository contains the public FTC SDK for the Ultimate Goal (2020-2021) competition season. Formerly this software project was hosted her

null 5 Feb 25, 2022
InterfaceMaker is a modern plugin to handle and customize join items, hotbars and menus with a developer and administrator friendly API.

Interface Maker InterfaceMaker is a modern plugin to handle and customize join items, hotbars and menus with a developer friendly API. Features Simple

2LStudios - Minecraft 10 Nov 27, 2022
Discourse-java is a platform where users can freely discuss on topics they want to, and like-minded people can join in and contribute

Discourse is the 100% open source discussion platform built for the next decade of the Internet. Use it as a: mailing list discussion forum long-form

Infosys Ltd 16 Oct 19, 2022
Parallel programming quick sort and parallel sum examples with Fork-join, RecursiveTask, RecursiveAction

QuickSortMultiThreading Parallel programming quick sort and parallel sum examples with Fork-join, RecursiveTask<T>, RecursiveAction Fork-Join Fork-Joi

Güven TUNCAY 4 Jun 12, 2022
A Minecraft Plugin For 1.8.8 - 1.19.2 For Giving A Player A Custom Item On Join Which Executes A Command On Right Click

PixelPlayz Custom Item A Minecraft Plugin For 1.8.8 - 1.19.2 For Giving A Player A Custom Item On Join Which Executes A Command On Right Click Install

null 2 Sep 9, 2022
Representational State Transfer + Structured Query Language(RSQL): Demo application using RSQL parser to filter records based on provided condition(s)

Representational State Transfer + Structured Query Language: RSQL Demo application using RSQL parser to filter records based on provided condition(s)

Hardik Singh Behl 9 Nov 23, 2022
Community extension to generate a Java client from the provided Camunda 7 OpenAPI descitpion and also warp it into Spring Boot

Camunda Engine OpenAPI REST Client Java and Spring Boot This community extension is a convenience wrapper around the generated Java client from the Ca

Camunda Community Hub 29 Dec 28, 2022
The VAST ad sample code provided by HUAWEI Ads Kit describes how to display linear ads by integrating the HUAWEI VAST SDK into your app.

HMS Ads Demo for VAST English | 中文 Table of Contents Introduction Installation Configuration Supported Environments Sample Code Result License Introdu

HMS 11 Jul 16, 2022