3 Repositories
Java tokenizer Libraries
Java implementation of GPT2 tokenizer.
GPT2 Tokenizer Java Java implementation of GPT2 tokenizer Requirements Please install the following dependencies to use the library. implementation 'c
Jan 4, 2023
Tokenisation of contents in Magnolia CMS
Magnolia tokenizer magnolia-tokenizer-module is a module of Magnolia CMS that allows the tokenization of contents in NFT's. Contents can be assets and
Nov 1, 2022
The fast scanner generator for Java™ with full Unicode support
JFlex JFlex is a lexical analyzer generator (also known as scanner generator) for Java. JFlex takes as input a specification with a set of regular exp
Dec 18, 2022