Binary Artifact Management Tool

Overview

EO principles respected here DevOps By Rultor.com We recommend IntelliJ IDEA

Build Status Docker Pulls License Hits-of-Code Docker Image Version (latest by date) PDD status

Artipie is an experimental binary artifact management tool, similar to Artifactory, Nexus, Archiva, ProGet, and many others. The following set of features makes Artipie unique among all others:

The fastest way to start using Artipie is via Docker. First, create a new directory artipie and repo sub-directory inside it. Then, put your YAML config file into the repo sub-dir. Make sure that the name of your config file is the name of repository you are going to host, and its name matches [a-z0-9_]{3,32}. For example foo.yaml:

repo:
  type: maven
  storage:
    type: fs
    path: /var/artipie

Now, go back to artipie and start the container:

$ docker run -p 8080:80 artipie/artipie:latest

You should be able to use it with Maven at http://localhost:8080.

More examples are here.

We recommend you read the "Architecture" section in our White Paper to fully understand how Artipie is designed.

Contents

Storage configuration

For now, we support two storage types: file system and S3 storages. To configure file system storage it is enough to set the path where Artipie will store all the items:

storage:
  type: fs
  path: /urs/local/aripie/data

S3 storage configuration requires specifying bucket and credentials:

storage:
  type: s3
  bucket: my-bucket
  region: my-region # optional
  endpoint: https://my-s3-provider.com # optional
  credentials:
    type: basic
    accessKeyId: xxx
    secretAccessKey: xxx

Storages can be configured for each repository individually in repo configuration yaml or in the _storages.yaml file along with aliases:

storages:
  default:
    type: fs
    path: ./.storage/data 

Then default storage alias can be used to configure a repository:

repo:
  type: maven
  storage: default

Repository permissions

Permissions for repository operations can be granted in the repo configuration file:

repo:
  ...
  permissions:
    jane:
      - read
      - write
    admin:
      - "*"
    /readers:
      - read

All repositories support read and write operations, other specific permissions may be supported in certain repository types.

Group names should start with /, is the example above read operation is granted for readers group and every user within the group can read from the repository, user named jane is allowed to read and write. We also support asterisk wildcard for "any operation" or "any user", user admin in the example can perform any operation in the repository.

If permissions section is absent in repo config, then any supported operation is allowed for everyone, empty permissions section restricts any operations for anyone.

Multitenancy

You may want to run Artipie for your company, which has a few teams. Each team may want to have its own repository. To do this, you create a global configuration file /etc/artipie/artipie.yml:

meta:
  layout: org
  storage:
    type: fs
    path: /tmp/artipie/data/my-docker
  credentials:
    type: file
    path: _credentials.yml

If the type is set to file, another YAML file is required in the storage, with a list of users who will be allowed to create repos (type is password format, plain and sha256 types are supported):

credentials:
  jane:
    type: plain
    pass: qwerty
    email: [email protected] # Optional
  john:
    type: sha256
    pass: xxxxxxxxxxxxxxxxxxxxxxx
    groups: # Optional
      - readers
      - dev-leads

Users can be assigned to some groups, all repository permissions granted to the group are applied to the users participating in this group.

If the type is set to env, the following environment variables are expected: ARTIPIE_USER_NAME and ARTIPIE_USER_PASS. For example, you start Docker container with the -e option:

docker run -d -v /var/artipie:/var/artipie` -p 80:80 \
  -e ARTIPIE_USER_NAME=artipie -e ARTIPIE_USER_PASS=qwerty \
  artipie/artipie:latest

Single repository on port

Artipie repositories may run on separate ports if configured. This feature may be especially useful for Docker repository, as it's API is not well suited to serve multiple repositories on single port.

To run repository on its own port port parameter should be specified in repository configuration YAML as follows:

repo:
  type: <repository type>
  port: 54321
  ...

NOTE: Artipie scans repositories for port configuration only on start, so server requires restart in order to apply changes made in runtime.

Metrics

You may enable some basic metrics collecting and periodic publishing to application log by adding metrics to meta section of global configuration file /etc/artipie/artipie.yml:

meta:
  metrics:
    type: log # Metrics type, for now only `log` type is supported
    interval: 5 # Publishing interval in seconds, default value is 5

Artipie REST API

Artipie provides a set of APIs to manage repositories and users. The current APIs are fully documented here.

Additional configuration

You may want configure it via environment variables:

  • SSL_TRUSTALL - trust all unknown certificates

To configure repository config files location, add to the global configuration file /etc/artipie/artipie.yml:

meta:
  repo_configs: configs

Location is the storage key relatively to the main storage, or, in file system storage terms, subdirectory where repo configs are located relatively to the storage.

Thanks to FreePik for the logo.

Comments
  • NPM authentication documentation is missed

    NPM authentication documentation is missed

    I tried to create NPM account using npm tool for Central registry with this command:

    npm adduser --loglevel silly --registry=https://central.artipie.com/npm --always-auth
    

    I've entered valid username and password but got 401 error:

    npm verb login before first PUT {
    npm verb login   _id: 'org.couchdb.user:artipie',
    npm verb login   name: 'artipie',
    npm verb login   password: 'XXXXX',
    npm verb login   type: 'user',
    npm verb login   roles: [],
    npm verb login   date: '2020-05-27T13:34:21.893Z'
    npm verb login }
    npm http fetch PUT 401 https://central.artipie.com/npm/-/user/org.couchdb.user:artipie 796ms
    npm verb adduser before first PUT {
    npm verb adduser   _id: 'org.couchdb.user:artipie',
    npm verb adduser   name: 'artipie',
    npm verb adduser   password: 'XXXXX',
    npm verb adduser   email: '[email protected]',
    npm verb adduser   type: 'user',
    npm verb adduser   roles: [],
    npm verb adduser   date: '2020-05-27T13:34:22.690Z'
    npm verb adduser }
    npm http fetch PUT 401 https://central.artipie.com/npm/-/user/org.couchdb.user:artipie 271ms
    npm verb stack Error: 401 Unauthorized - PUT https://central.artipie.com/npm/-/user/org.couchdb.user:artipie
    npm verb stack     at /usr/lib64/node_modules/npm/node_modules/npm-registry-fetch/check-response.js:104:15
    npm verb stack     at processTicksAndRejections (internal/process/task_queues.js:97:5)
    npm verb statusCode 401
    

    It seems I configured auth wrong.

    scope role/DEV 
    opened by g4s8 28
  • Support /secutiry API of Artifactory

    Support /secutiry API of Artifactory

    Support Artifactory API at /api/secutiry/users and /api/secutiry/permissions: API clients should be able to create update and delete users and permissions.

    enhancement 0crat/role/DEV api/artifactory 
    opened by g4s8 26
  • Helm proof is broken

    Helm proof is broken

    When data is being posted to helm repo, the 100 response is started to being printed and then stuck. Try helm proof for details:

    curl -i -X POST --data-binary @tomcat-0.4.1.tgz http://localhost:8080/example_helm_repo/
    HTTP/1.1 100 Continue
    
    
    0crat/role/DEV 
    opened by Sammers21 23
  • Get rid of path config parameter

    Get rid of path config parameter

    Some repositories have path config parameter, some don't have it. I think we can remove it safely from config, since we always know what is the path of current request. Even if we switch to another routing schema #178 we still be able to get the path from request on top level. So we can remove it from config and specify directly. @artipie/contributors do you agree?

    scope role/DEV 
    opened by g4s8 23
  • Collect Artipie metrics

    Collect Artipie metrics

    Add interface to collect Artipie metrics for some common events. This interface is going to be connected to third-party services later. For the first stage, we may just print metrics to log output. Let's start with these metrics:

    • artifact uploaded
    • artifact downloaded
    • artifact upload error
    • artifact download error
    • not authenticated
    • not authorized

    Feel free to suggest more metrics or discuss existing.

    enhancement role/DEV quality/good 
    opened by g4s8 18
  • Repository configuration cache

    Repository configuration cache

    Artipie performs a lot of read operations for repository configs, it would be better to cache these configs in memory. The config can be updated at any time, so we need to drop cached value after some time, e.g. after few minutes. Consider using soft references to avoid OOM errors when working with many repos.

    scope role/DEV 
    opened by g4s8 17
  • Do not build assembly jar-with-dependencies by default

    Do not build assembly jar-with-dependencies by default

    Running mvn verify -Pqulice is extremely long because of the task make-assembly.

    It shouldn't be enabled in a normal build but only via a profile.

    I suppose the CI should take that into account also (depending on where this jar is needed).

    role/DEV quality/good 
    opened by victornoel 17
  • Catch repo file reading error and return 404

    Catch repo file reading error and return 404

    Resolving issue #74 Catching exceptions when reading repo config file. It is catching any exception when reading file and return 404 with an error message.

    role/DEV 
    opened by HDouss 16
  • Restructure Artipie components

    Restructure Artipie components

    Currently, it seems we have some "reactive overhead" in Artipie module for Dashboard and API - these components are not good candidates for applying reactive streams on them, because:

    1. API and Dashboard data traffic is very low and request frequency is rare, comparing to repository traffic.
    2. almost all requests contains only one chunk of data, so using Publisher of ByteBuffer for processing one ByteBuffer is overhead
    3. using non-blocking reactive streams approach requires a lot of additional code and work, which is OK if we pay it for better performance or resources consumption for high-load traffic, but in case with Dashboard and API, the traffic is low.

    Proposal

    I'm suggesting restructure Dashboard and API code and move it to separate software components (dedicated servers). It allows us to:

    • Simplify the code in artipie/artipie module by removing the code related to API management and Dashboard.
    • Use other approaches (maybe another web frameworks or even languages) for other modules.
    • Simplify changes in Dashboard and API, since these changes will not depend on main Artipie components (makes coupling lower).
    • Better (granular) resources control for these components in big applications in cluster.

    Possibly, API and dashboard could be one component, not two.

    Implementation

    Dashboard and API components may expose endpoints for /api and /dashboard URL paths; primary load-balancer routes requests based on this path: /api paths goes to API service, /dashboard path to Dashboard service and other requests to main Artipie service (see diagram below).

    artipie-component drawio

    Questions

    Using this approach, some aspects are not clear:

    1. How to provide simple deployment for local usage (now we have Docker with Dashboard and API included). Maybe Docker-compose or something like this can be used for that (if users needs dashboard and API for tiny local deployments).
    2. Auth mechanism for API and Dashboard - now it's rely on artipie/http authorization implementation, we may need to copy-paste this logic if we switch to multiservice components.
    enhancement question 
    opened by g4s8 15
  • Pie.java:49-52: Create a unit test for 404 response....

    Pie.java:49-52: Create a unit test for 404 response....

    The puzzle 74-e630ca71 from #74 has to be resolved:

    https://github.com/artipie/artipie/blob/2ef2028f29ee2b5fa4b794091c9ed57408373f13/src/main/java/com/artipie/Pie.java#L49-L52

    The puzzle was created by @g4s8 on 08-May-20.

    Estimate: 30 minutes, role: DEV.

    If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

    role/DEV quality/good 
    opened by 0pdd 15
  • Vertx context warning on each request

    Vertx context warning on each request

    Artipie prints this warning on each incoming HTTP request:

    WARNING: You're already on a Vert.x context, are you sure you want to create a new Vertx instance?
    

    It seems we're creating new Vertx instance for storage on each request.

    scope role/DEV quality/good 
    opened by g4s8 15
  • PKCS 12 credentials

    PKCS 12 credentials

    Hi Team,

    I was wondering if Artipie supports PKCS 12 if I supplied the cert with the file option(see doc below) | https://github.com/artipie/artipie/wiki/Configuration-Credentials#credentials

    My use case is I would want to connect/proxy to a upstream pypi repo with PKCS 12 credentials instead of username and password

    https://github.com/artipie/artipie/wiki/pypi-proxy

    Thanks!

    enhancement 
    opened by jmcgrath207 2
  • General token authentication

    General token authentication

    Several adapters (at least npm and conda) requires from us to support token authentication. Let's use vert.x jwt auth for these purposes as a) jwt are stateless, b) we do not need to invent any bicycles. Users will be able to use the token in all artipie: as for our rest api and for some repository.

    enhancement 
    opened by olenagerasimova 0
  • RPM packaging

    RPM packaging

    It would be nice to be able to install/update Artipie to RHEL-based servers with yum install/update artipie :D

    The RPM package could include:

    • The JAR file
    • A SystemD unit file
    • A sample configuration file
    • ???

    I could probably help with writing a spec file. GitHub should have an Action that will build the actual package. The public Artipie instance could then host the packages. How cool is that? :)

    enhancement 
    opened by vassilevsky 0
  • Maven group repo

    Maven group repo

    We have repository type maven-group in SliceFromConfig, the repository is build right in the SliceFromConfig. Let's figure out what is this repository idea, how it works and either add proper documentation and integration test or remove it.

    opened by olenagerasimova 0
Releases(v0.28.0)
Owner
Artipie
Binary Artifact Management Toolkit
Artipie
Classpy is a GUI tool for investigating Java class file, Lua binary chunk, Wasm binary code, and other binary file formats.

Classpy Classpy is a GUI tool for investigating Java class file, Lua binary chunk, Wasm binary code, and other binary file formats. Inspiration This t

null 1k Dec 17, 2022
ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.

ANTLR v4 Build status ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating

Antlr Project 13.6k Dec 28, 2022
ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.

ANTLR v4 Build status ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating

Antlr Project 13.6k Jan 3, 2023
log4j2-scan is a single binary command-line tool for CVE-2021-44228 vulnerability scanning and mitigation patch

log4j2-scan is a single binary command-line tool for CVE-2021-44228 vulnerability scanning and mitigation patch. It also supports nested JAR file scan

Logpresso GitHub 839 Dec 29, 2022
Simple Binary Encoding (SBE) - High Performance Message Codec

Simple Binary Encoding (SBE) SBE is an OSI layer 6 presentation for encoding and decoding binary application messages for low-latency financial applic

Real Logic 2.8k Jan 3, 2023
Java binary serialization and cloning: fast, efficient, automatic

Kryo is a fast and efficient binary object graph serialization framework for Java. The goals of the project are high speed, low size, and an easy to u

Esoteric Software 5.7k Jan 5, 2023
binary serialization format

Colfer Colfer is a binary serialization format optimized for speed and size. The project's compiler colf(1) generates source code from schema definiti

Pascal S. de Kloe 680 Dec 25, 2022
Simple Binary Encoding (SBE) - High Performance Message Codec

Simple Binary Encoding (SBE) SBE is an OSI layer 6 presentation for encoding and decoding binary application messages for low-latency financial applic

Real Logic 2.8k Dec 28, 2022
Apache POI - A Java library for reading and writing Microsoft Office binary and OOXML file formats.

Apache POI A Java library for reading and writing Microsoft Office binary and OOXML file formats. The Apache POI Project's mission is to create and ma

The Apache Software Foundation 1.5k Jan 1, 2023
RSocket is a binary protocol for use on byte stream transports such as TCP, WebSockets, and Aeron

RSocket RSocket is a binary protocol for use on byte stream transports such as TCP, WebSockets, and Aeron. It enables the following symmetric interact

RSocket 2.2k Dec 30, 2022
GalaxyCDC is a core component of PolarDB-X which is responsible for global binary log generation, publication and subscription.

中文文档 What is ApsaraDB GalaxyCDC ? GalaxyCDC is a core component of PolarDB-X which is responsible for global binary log generation, publication and su

null 56 Dec 19, 2022
Apollo is a reliable configuration management system suitable for microservice configuration management scenarios.

English | 中文 Apollo - A reliable configuration management system Apollo is a reliable configuration management system. It can centrally manage the con

Apollo 27.6k Jan 5, 2023
Backend for Saunah Management App provides a REST-API for the Saunah management app

?? ?? Saunah Backend Backend for Saunah Management App. This application provides a REST-API for the Saunah management app. ????‍?? Technology Stack T

null 2 Jun 13, 2022
Modern tool for microservice configuration management

Quick start We recommend to start with Microconfig Features guide and then continue reading this documentation. Microconfig overview and features Micr

Microconfig 262 Dec 19, 2022
APM, (Application Performance Management) tool for large-scale distributed systems.

Visit our official web site for more information and Latest updates on Pinpoint. Latest Release (2020/01/21) We're happy to announce the release of Pi

null 12.6k Jan 4, 2023
APM, (Application Performance Management) tool for large-scale distributed systems.

Visit our official web site for more information and Latest updates on Pinpoint. Latest Release (2020/01/21) We're happy to announce the release of Pi

null 12.5k Dec 29, 2022
APM, (Application Performance Management) tool for large-scale distributed systems.

Visit our official web site for more information and Latest updates on Pinpoint. Latest Release (2020/01/21) We're happy to announce the release of Pi

null 12.6k Jan 6, 2023
inspectIT is the leading Open Source APM (Application Performance Management) tool for analyzing your Java (EE) applications.

?? inspectIT OpenCensus Edition has been released ?? The inspectIT OCE project provides an improved Java agent newly developed from scratch focusing o

inspectIT 531 Dec 13, 2022
Keycloak - an Open Source Identity and Access Management tool

Keycloak is an Open Source Identity and Access Management tool. You can use it to add authentication to applications and secure services with minimum effort. No need to deal with storing users or authenticating users.

Erdem Günay 9 Sep 20, 2022
A maven plugin to include features from jmeter-plugins.org for JMeterPluginsCMD Command Line Tool to create graphs, export csv files from jmeter result files and Filter Result tool.

jmeter-graph-tool-maven-plugin A maven plugin to create graphs using the JMeter Plugins CMDRunner from JMeter result files (*.jtl or *.csv) or using F

Vincent DABURON 6 Nov 3, 2022