联邦学习系统,包括常用算法和通用训练推理系统框架 | Fedlearn Main System, Including Algorithms and Frameworks for Training / Inference.

Overview

fedlearn 京东科技联邦学习系统

系统包含包含控制端(即前端)、协调端、单点客户端和分布式客户端等

1.代码结构

代码分为多个模块

  • assembly 整体代码打包模块,无实际功能
  • client 单机版客户端
  • common 公共包,实体和工具定义
  • coordinator 协调端,负责协调多个参与方数据交互
  • core 核心算法
  • frontend 业务逻辑提交和前端页面
  • manager 分布式客户端管理单元
  • worker 分布式客户端计算单元

2.环境要求

  • 最低硬件配置--4核CPU,8G内存,50G硬盘;
  • 操作系统--Centos7/8,ubuntu 16/18/20
  • Java环境--JDK1.8
  • maven--3.6

3.下载代码和编译打包

  • clone代码
  • package
mvn clean package
  • 打包后是.zip文件,位于 ./assembly/target/fedlearn-all-assembly.zip 运行时需要把文件解压缩,文件夹内会有多个目录
fedlearn-all
  --conf  配置文件,端口号、相关路径的修改
  --bin   启动、停止脚本
  --lib   相关依赖
  --readme 文档

4.部署

系统组件包含协调端、控制端、客户端三部分,其中客户端部署分为两种,单点客户端和分布式客户端。

4.1 协调端部署
  • 数据初始化 服务端依赖数据库保存持久化数据,所以需要创建数据库和初始化表结构。我们现有python项目支持数据库和表的初始化工作; 依照元数据存储方式的不同,目前支持mysql和sqlite两种方式,其中sqlite为系统安装包自带。
  • 修改配置,根据实际情况配置元数据和日志等目录。 -命令启动
cd ./fedlearn-all
bash bin/start-coordinator.sh -c ./conf/coordinator.properties
4.2 界面部署
  • 修改配置
  • 命令启动
cd ./fedlearn-all
bash bin/start-frontend.sh -c ./conf/application.yml
4.3单机版客户端部署

-修改配置 根据实际情况修改conf/client.properties文件 -命令启动

cd ./fedlearn-all
bash bin/start-client.sh -c ./conf/client.properties
4.4 分布式客户端部署

分布式客户端包括manager和worker两部分,

cd ./fedlearn-all
bash bin/start-worker.sh -c ./conf/worker.properties
4.5 区块链版本部署

区块链系统依赖京东的区块链jdchain,请先安装jdchain,参考 https://github.com/blockchain-jd-com

  • 修改各项配置文件中的区块链项目为
jdchain.available=true

并根据实际情况修改配置文件中的区块链地址

  • 参照标准版模式启动区块链系统,

5. FAQ

You might also like...

Algorithms Made Easy May 10 Challenge

Algorithms Made Easy May 10 Challenge

Algorithms-Made-Easy-May-Challenges Algorithms Made Easy May 10 day 30 problems Challenge Hi 👨‍🎓 , I'm Rohit Kumar Singh All Leetcode Soluton Connec

May 24, 2021

Graph Algorithms Repository for Coding Minutes Course.

Graph Algorithms Repository for Coding Minutes Course.

graph-algorithms-for-competitive-coding Graph Algorithms Repository for Coding Minutes Course. This is the repository for Graph Algorithms Course for

Dec 28, 2022

This is a tool to visualize search algorithms

This is a tool to visualize search algorithms

Path-Finding-Visualizer Purpose This is a tool to visualize search algorithms Algorithms featured Breadth First Search Deapth First Search Gready Best

Oct 20, 2022

Algorithms in Java

Algorithms-In-Java 'Algorithms in Java' keeps programs from the "Algorithms and data structures" held at the University of Technology in Wrocław (Poli

Jan 25, 2022

One-Stop Destination for codes of all Data Structures & Algorithms

CodingSimplified_GK This repository is aimed at creating a One stop Destination of codes of all Data structures and Algorithms along with basic explai

Sep 26, 2022

CompreFace is a free and open-source face recognition system from Exadel

 CompreFace is a free and open-source face recognition system from Exadel

CompreFace can be easily integrated into any system without prior machine learning skills. CompreFace provides REST API for face recognition, face verification, face detection, landmark detection, age, and gender recognition and is easily deployed with docker

Dec 31, 2022

Workflow management system fully supporting low-code language Petriflow.

Netgrif Application Engine MySQL Create NETGRIF user: CREATE USER 'netgrif_nae'@'localhost' IDENTIFIED BY 'netgrif_nae'; GRANT ALL PRIVILEGES ON * . *

Dec 23, 2022

👄 The most accurate natural language detection library for Java and the JVM, suitable for long and short text alike

👄 The most accurate natural language detection library for Java and the JVM, suitable for long and short text alike

Quick Info this library tries to solve language detection of very short words and phrases, even shorter than tweets makes use of both statistical and

Dec 28, 2022
Owner
null
Bazel training materials and codelabs focused on beginner, advanced and contributor learning paths

Bazel-learning-paths This repo has materials for learning Bazel: codelabs, presentations, examples. We are open sourcing the content for training engi

null 18 Nov 14, 2022
This repository holds the famous Data Structures (mostly abstract ones) and Algorithms for sorting, traversing, and modifying them.

Data-Structures-and-Algorithms About Repo The repo contains the algorithms for manipulating the abstract data structures like Linked List, Stacks, Que

Zaid Ahmed 14 Dec 26, 2021
java deep learning algorithms and deep neural networks with gpu acceleration

Deep Neural Networks with GPU support Update This is a newer version of the framework, that I developed while working at ExB Research. Currently, you

Ivan Vasilev 1.2k Jan 6, 2023
This repository consists of the code samples, assignments, and the curriculum for the Community Classroom complete Data Structures & Algorithms Java bootcamp.

DSA-Bootcamp-Java Subscribe to our channel Complete Playlist Syllabus Discord for discussions Telegram for announcements Connect with me     Follow Co

Kunal Kushwaha 10.2k Jan 1, 2023
Welcome 🙌! This repository encourages daily contributions from anyone intending to learn Data Structures and Algorithms every day

?? DSA-Community Welcome ?? ! This repository encourages daily contributions from anyone intending to learn Data Structures and Algorithms consistentl

Bishal Mohari 4 Sep 9, 2022
oj! Algorithms - ojAlgo - is Open Source Java code that has to do with mathematics, linear algebra and optimisation.

oj! Algorithms oj! Algorithms - ojAlgo - is Open Source Java code that has to do with mathematics, linear algebra and optimisation. General informatio

Optimatika 403 Dec 14, 2022
A visual representation of labyrinth solving with common traversal and heuristic algorithms + basic AI patterns

Path-finder A visual representation of labyrinth solving algorithms using common traversal algorithms such as BFS, DFS, A*. Plus there are some basic

Janez Sedeljšak 2 Jan 19, 2022
This JAVA repository contains solutions for common algorithms and problems.

JAVA-Algorithms ?? Description Beep Boop! Boop Beep!. I have created this repository to improve my Logical thinking skills & Knowledge in programming.

VINU 3 Apr 11, 2022
Data Structures and Algorithms (DSA) - Java Language Using Integrated Development Environments NetBeans

Data Structures and Algorithms (DSA) Course Code : CSC211 Credit Hours : 4 Language : JAVA Integrated development environments : NETBEANS Topic Covere

Ossama Mehmood 샘 2 Oct 1, 2022