Simple Binary Encoding (SBE) - High Performance Message Codec

Overview

Simple Binary Encoding (SBE)

Javadocs GitHub

Actions Status Total Alerts Code Quality: Java

SBE is an OSI layer 6 presentation for encoding and decoding binary application messages for low-latency financial applications. This repository contains the reference implementations in Java, C++, Golang, and C#.

More details on the design and usage of SBE can be found on the Wiki.

An XSD for SBE specs can be found here. Please address questions about the specification to the SBE FIX community.

For the latest version information and changes see the Change Log with downloads at Maven Central.

The Java and C++ SBE implementations work very efficiently with the Aeron messaging system for low-latency and high-throughput communications. The Java SBE implementation has a dependency on Agrona for its buffer implementations. Commercial support is available from [email protected].

Binaries

Binaries and dependency information for Maven, Ivy, Gradle, and others can be found at http://search.maven.org.

Example for Maven:

<dependency>
    <groupId>uk.co.real-logic</groupId>
    <artifactId>sbe-all</artifactId>
    <version>${sbe.tool.version}</version>
</dependency>

Build

Build the project with Gradle using this build.gradle file.

Full clean build:

$ ./gradlew

Run the Java examples

$ ./gradlew runJavaExamples

Distribution

Jars for the executable, source, and javadoc for the various modules can be found in the following directories:

sbe-benchmarks/build/libs
sbe-samples/build/libs
sbe-tool/build/libs
sbe-all/build/libs

An example to execute a Jar from command line using the 'all' jar which includes the Agrona dependency:

java -Dsbe.generate.ir=true -Dsbe.target.language=Cpp -Dsbe.target.namespace=sbe -Dsbe.output.dir=include/gen -Dsbe.errorLog=yes -jar sbe-all/build/libs/sbe-all-${SBE_TOOL_VERSION}.jar my-sbe-messages.xml

C++ Build using CMake

NOTE: Linux, Mac OS, and Windows only for the moment. See FAQ. Windows builds have been tested with Visual Studio Express 12.

For convenience, the cppbuild script does a full clean, build, and test of all targets as a Release build.

$ ./cppbuild/cppbuild

If you are comfortable using CMake, then a full clean, build, and test looks like:

$ mkdir -p cppbuild/Debug
$ cd cppbuild/Debug
$ cmake ../..
$ cmake --build . --clean-first
$ ctest

Note: The C++ build includes the C generator. Currently, the C generator is a work in progress.

Golang Build

First build using Gradle to generate the SBE jar and then use it to generate the golang code for testing.

$ ./gradlew
$ ./gradlew generateGolangCodecs

For convenience on Linux, a gnu Makefile is provided that runs some tests and contains some examples.

$ cd gocode
# make # test, examples, bench

Users of golang generated code should see the user documentation.

Developers wishing to enhance the golang generator should see the developer documentation

C# Build

Users of CSharp generated code should see the user documentation.

Developers wishing to enhance the CSharp generator should see the developer documentation

License (See LICENSE file for full license)

Copyright 2013-2021 Real Logic Limited.
Copyright 2017 MarketFactory Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Comments
  • [C++] Undefined behaviour in generated code

    [C++] Undefined behaviour in generated code

    The C++ code generator typically generates code like this for accessing integer fields:

        std::int32_t securityID(void) const
        {
            return SBE_LITTLE_ENDIAN_ENCODE_32(*((std::int32_t *)(m_buffer + m_offset + 24)));
        }
    
        NewOrderSingle &securityID(const std::int32_t value)
        {
            *((std::int32_t *)(m_buffer + m_offset + 24)) = SBE_LITTLE_ENDIAN_ENCODE_32(value);
            return *this;
        }
    

    The compiler assumes that a pointer to a int32_t has the correct alignment. With this generated code the alignment requirement might not fulfilled. On amd64 this is fine as long as the compiler doesn't try to use SSE or AVX instructions, but it's not safe in general to assume it will work.

    The solution is to use memcpy like this (https://chromium.googlesource.com/chromium/src.git/+/master/base/bit_cast.h):

    template <class Dest, class Source>
    inline Dest bit_cast(const Source& source) {
       static_assert(sizeof(Dest) == sizeof(Source),
       "bit_cast requires source and destination to be the same size");
       static_assert(base::is_trivially_copyable<Dest>::value,
       "bit_cast requires the destination type to be copyable");
       static_assert(base::is_trivially_copyable<Source>::value,
       "bit_cast requires the source type to be copyable");
        
       Dest dest;
       memcpy(&dest, &source, sizeof(dest));
       return dest;
    }
    

    This should optimize to a single load load on amd64, guaranteed not to use instructions requiring alignment.

    opened by rigtorp 21
  • C++ Generated code encoding and decoding issue.

    C++ Generated code encoding and decoding issue.

    I found an issue in the c++ generated code. The positionptr is not correctly updated after a string is written to the buffer, this will cause the first repeating to be written out of position. The byte positions for the root items and the first repeating group should be constant, therefore the need to use the positionptr is unnecessary. The following repeating groups position should be calculated based on the size of the previous repeating group. This will make all the fields accessible without priming the positionptr

    opened by terryvaldez 21
  • New Rust code generator

    New Rust code generator

    This PR provides full SBE code generation support for the Rust programming language.

    • generates 100% safe rust code
    • the rust code generated has zero dependencies on any external rust libs
    • both lib.rs and Cargo.toml files are generated so projects can easily add codec as a dependency
    • Fixes #813

    Testing

    # following will write generated code to "./generated/rust"
    ./gradlew generateRustCodec
    
    # assuming you have "rust" and "cargo" already installed
    cd ./rust
    cargo test
    
    opened by mward 20
  • sbe-tool nuget package is DOA

    sbe-tool nuget package is DOA

    When using the sbe-tool from the latest sbe-tool nuget package: 0.1.12-1-beta-1, code generation immediately fails

    Exception in thread "main" java.lang.NoClassDefFoundError: org/agrona/MutableDirectBuffer
            at uk.co.real_logic.sbe.SbeTool.<clinit>(SbeTool.java:82)
    Caused by: java.lang.ClassNotFoundException: org.agrona.MutableDirectBuffer
            at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
            at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
            at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
            ... 1 more
    

    It seems that the nuget package contains the WRONG jar, it should have contained the jar from the: sbe-all/build/libs/ folder which contains all the required java deps, but instead it uses the jar from the sbe-tool/build/libs/ folder which is partial as far as running the complete sbe-tool is...

    Would it be OK for me submit a PR that fixes this and for someone who has access to the nuget account (I presume @billsegall) to publish an updated nuget package?

    As far as I can tell, this thing is dead out of water from nuget....

    opened by damageboy 20
  • C++ Segmentation fault printing XML sub group (parent message prints fine)

    C++ Segmentation fault printing XML sub group (parent message prints fine)

    I'm processing an MDIncrementalRefreshBook46 message. It contains two XML sub-groups: NoMDEntries followed-by NoOrderIDEntries.

    I create the MDIncrementalRefreshBook46 object and print it. I then iterate through the second XML sub-group and read the referenceID and OrderUpdateAction:

    mktdata::MDIncrementalRefreshBook46 msg;
    msg.wrapForDecode(bytes, offset, blockLength, mktdata::MessageHeader::sbeSchemaVersion(), bufferSize);
    std::cout << msg << std::endl;
    
    mktdata::MDIncrementalRefreshBook46::NoOrderIDEntries& mboEntries = msg.noOrderIDEntries();
    while(mboEntries.hasNext())
    {
        mktdata::MDIncrementalRefreshBook46::NoOrderIDEntries& mboEntry = mboEntries.next();
        const uint8_t referenceID = mboEntry.referenceID();
        std::cout << "referenceID: " << referenceID << std::endl;
        const mktdata::OrderUpdateAction::Value updateAction = mboEntry.orderUpdateAction();
        std::cout << "updateAction: " << updateAction << std::endl;
    }
    

    The parent message prints fine:

    {"Name": "MDIncrementalRefreshBook46", "sbeTemplateId": 46, "TransactTime": 1592154621498375773, "MatchEventIndicator": ["lastQuoteMsg"], "NoMDEntries": [{"MDEntryPx": {"mantissa": 302375000000000}, "MDEntrySize": 10, "SecurityID": 19074, "RptSeq": 1, "NumberOfOrders": 1, "MDPriceLevel": 1, "MDUpdateAction": "New", "MDEntryType": "Bid"}], "NoOrderIDEntries": [{"OrderID": 646981700454, "MDOrderPriority": 8786193103, "MDDisplayQty": 10, "ReferenceID": 1, "OrderUpdateAction": "New"}]}

    but if I try to retrieve the referenceID I get a null/0 (should be 1) and when reading the OrderUpdateAction I get an invalid value and an exception throw in OrderUpdateAction::get().

    This could be misuse on my part, but I can't see what I'm doing wrong?

    opened by cpp77 19
  • problem using cppbuild on Linux

    problem using cppbuild on Linux

    Hi,

    I am trying to build sbe project on Linux with gradle-5.6.3 by following README.md instructions. gradle itself was built successfully and the daemon is running.

    When I execute ./cppbuild/cppbuild I get the following:

    ./cppbuild/cppbuild Will make with "-j 4". Build directory (/home/vladt/encode/simple-binary-encoding-master/cppbuild/Release) exists, removing. -- The C compiler identification is GNU 4.4.7 -- The CXX compiler identification is GNU 4.4.7 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE
    -- Found Java: /usr/local/java/jdk1.8.0_45/bin/java (found version "1.8.0.45") -- Looking for clock_gettime in rt -- Looking for clock_gettime in rt - found Build type(Release): perf tests turning OFF bounds checks -- Configuring done -- Generating done -- Build files have been written to: /home/vladt/encode/simple-binary-encoding-master/cppbuild/Release Scanning dependencies of target gtest Scanning dependencies of target sbe-jar [ 2%] Creating directories for 'gtest' [ 4%] Generating SBE jar [ 6%] Performing download step (verify and extract) for 'gtest' -- verifying file... file='/home/vladt/encode/simple-binary-encoding-master/cppbuild/googletest-release-1.8.0.zip' -- verifying file... done -- extracting... src='/home/vladt/encode/simple-binary-encoding-master/cppbuild/googletest-release-1.8.0.zip' dst='/home/vladt/encode/simple-binary-encoding-master/cppbuild/Release/thirdparty/gtest/src/gtest' -- extracting... [tar xfz]

    FAILURE: Build failed with an exception.

    • What went wrong: Task 'test' not found in root project 'sbe'.

    • Try: Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

    • Get more help at https://help.gradle.org

    BUILD FAILED in 1s make[2]: *** [../../sbe-all/build/libs/sbe-all-1.15.0-SNAPSHOT.jar] Error 1 make[1]: *** [CMakeFiles/sbe-jar.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... -- extracting... [analysis] -- extracting... [rename] -- extracting... [clean up] -- extracting... done [ 10%] No patch step for 'gtest' [ 10%] No update step for 'gtest' [ 12%] Performing configure step for 'gtest' -- gtest configure command succeeded. See also /home/vladt/encode/simple-binary-encoding-master/cppbuild/Release/thirdparty/gtest/src/gtest-stamp/gtest-configure-.log [ 14%] Performing build step for 'gtest' -- gtest build command succeeded. See also /home/vladt/encode/simple-binary-encoding-master/cppbuild/Release/thirdparty/gtest/src/gtest-stamp/gtest-build-.log [ 16%] No install step for 'gtest' [ 18%] Completed 'gtest' [ 18%] Built target gtest make: *** [all] Error 2

    Is task 'test' missing from distribution?

    I must add that I am new to gradle so any relevant help will be appreciated.

    Thanks, Vlad

    opened by vladtolmachyov 19
  • [C#] Added more string read/write methods to the DirectBuffer #729

    [C#] Added more string read/write methods to the DirectBuffer #729

    Address #729

    • Added DirectBuffer methods for reading and writing string that need encoding.
    • Added convenience accessors for the various Encodings defined in the schema.
    • Changed the C# Code generator to create read/write methods using the new DirectBuffer methods.
    • Changed the sample to use the new methods.

    The suggestions that the ticket creator had require .net48+ as far as I can see - using the Encoding.GetString() methods. I kept with the current .net45 version and implmented the 'spirit' of of the request.

    opened by MFrpurdy 18
  • SBE 1.20.3 is generating C++ code with incorrect macro __cplucplus

    SBE 1.20.3 is generating C++ code with incorrect macro __cplucplus

    Below is a sample of generated code. The macro is missing a "s"

    Needs to be __cplusplus instead of _cplucplus

    #if __cpluplus >= 201703L length += std::apply(SymbolResponses::computeLength, e); #else length += SymbolResponses::computeLength(std::get<0>(e), std::get<1>(e)); #endif

    opened by samkhopkar 18
  • [C++] constexpr qualifier in static const field getters

    [C++] constexpr qualifier in static const field getters

    This is a follow up enhancement to the #195 issue.

    I am tailoring the incoming message reaction, and it would be great to use a switch statement in the vein of the structurechart's comment. Similarly to @structurechart I ran into the constexpr requirement.

    Would it be possible to discuss a solution similar to the sample commit vdidenko/simple-binary-encoding@fe16a8ce1f1e14916d3ebe363978549a26267d5c? It introduces public constexpr fields and uses them to maintain the getter function signatures.

    I have only changed the fields I think I care about, but should I make the pull request similarly modifying all static const declarations touched by the original solution at real-logic/simple-binary-encoding@a601c48409a093abda3ad6808407712fa677dfaf?

    opened by vdidenko 18
  • (C++) No version-checking code generated for group fields

    (C++) No version-checking code generated for group fields

    While there are xxxSinceVersion() and xxxInActingVersion() methods to check before decoding top-level fields, there are no such methods generated for groups in C++.

    opened by tamboril 17
  • 2 repos?

    2 repos?

    Sorry to ask again about that but I'm still not convinced about having a separate repo for the benchmarks, for a few reasons:

    • people who want to evaluate SBE have to clone 2 repos instead of one. I'm not sure everyone will go through the process of cloning the main one, discovering that the benchmarks are not there and clone the other repo (if they find it!) to then run the tests What they should be able to do is clone the repo, type something like "ant java:bench" and get everything build and running, benchmarks included.
    • when I change the code gen I'd like to be able to run the perf tests immediately to validate that perf has not regressed. At the moment it's a multi step process (generate new sbe.jar, generate new .NET dlls, copy over to other repo, build benchmark).
    • things can easily get out of sync: we are 3 devs updating sbe.jar and for instance on the .NET side I need the corresponding sbe.ddl and sbe.pdb to be up to date.
    • I'm working from another repository and other contributors will do the same. It's quite an overhead to have to manage 2 repos instead of one: for instance if I change the code generation I need to
      1. change the code in the main repo
      2. build
      3. copy sbe.jar and others to the bench repo
      4. build and run perf tests (if there is a problem back to step 1)
      5. commit changes in main repo to origin (my fork of sbe)
      6. commit changes in bench repo to origin (my fork)
      7. prepare and send 2 pull requests

    I understand that we want to keep the repo clean of third party dependencies like Google Proto Buff and others but we can download them on demand during the build instead of storing them in Git.

    I see more problems than advantages working that way and I find it quite painful at the moment, especially while things are still changing a lot.

    What do you guys think?

    enhancement question 
    opened by odeheurles 17
  • Runtime checks of variable-length calls

    Runtime checks of variable-length calls

    Accessing a variable length SBE property modifies the state of the auto-generated SBE classes by advancing an offset field. Client code has to access every variable length property exactly once, in order. It's common to pass around lists or strings whose length we can't know ahead of time; in such cases SBE client code is brittle and hard to debug.

    Could you add in optional runtime checks that validate that getters / setters of variable-length properties are called in the correct order? These would do the following:

    • If enabled:
      • Register calls that access variable length properties
      • If variable length properties are accessed out of order, throw an exception with a diagnostic error message
    • In disabled:
      • Do nothing. Ideally, don't even compile the code which enforces the check

    The checks would be either enabled / disabled at compile time. I would anticipate using them in debug builds (used in development work / some continuous integration plans) and disabling them in release builds (used in production), but other users might want to enable them in different contexts.

    enhancement 
    opened by grlrrchapman 0
  • DTOs to help with variable-length in non-latency-critical contexts

    DTOs to help with variable-length in non-latency-critical contexts

    Accessing a variable length SBE property modifies the state of the auto-generated SBE classes by advancing an offset field. Client code has to access every variable length property exactly once, in order. It's common to pass around lists or strings whose length we can't know ahead of time; in such cases SBE client code is brittle and hard to debug.

    Could you add auto-generated DTOs to the SBE code-gen to mitigate this problem? The requirements for these objects would be:

    • Safe to get or set any property in any order
    • Index-based access for array-like collections
    • Use separately allocated memory for the DTOs, with methods to convert the DTOs to / from the wire format

    The DTOs would be really useful in non-latency-critical scenarios such as testing, debugging tools and post-processing of persisted messages. It's understood that using the DTOs will incur a performance penalty.

    C++ / C# are the two languages used by the author of this request, but I can imagine other users wanting this feature in code-gen for other languages.

    enhancement 
    opened by grlrrchapman 0
  • [Rust] bugfix: nested composite name should not be based on field name

    [Rust] bugfix: nested composite name should not be based on field name

    I was refactoring the generated Rust codec to be more idiomatic, and discovered that the existing Rust codec generator uses the wrong type name for nested composite field:

    Minimal schema file to trigger error:

    <?xml version="1.0" encoding="UTF-8"?>
    <sbe:messageSchema xmlns:sbe="http://fixprotocol.io/2016/sbe"
                       xmlns:xi="http://www.w3.org/2001/XInclude"
                       package="nested-composite-name"
                       id="1"
                       version="0"
                       semanticVersion="5.2"
                       description="test case for nested composite field where the field name from the composite name"
                       byteOrder="littleEndian">
        <types>
            <composite name="MyComposite">
                <ref name="myFieldName" type="MyNestedComposite"/> <!-- The field name here is different from the nested composite name -->
            </composite>
            <composite name="MyNestedComposite">
                <type name="irrelevantField" primitiveType="uint16"/>
            </composite>
            <composite name="messageHeader">
                <type name="blockLength" primitiveType="uint16"/>
                <type name="templateId"  primitiveType="uint16"/>
                <type name="schemaId"    primitiveType="uint16"/>
                <type name="version"     primitiveType="uint16"/>
            </composite>
        </types>
        <sbe:message name="MyMessage" id="1" description="Nested composite name different from field name">
            <field name="irrelevantHeader" id="1" type="messageHeader"/>
            <field name="irrelevantField" id="2" type="MyComposite"/>
        </sbe:message>
    </sbe:messageSchema>
    

    Generated Rust output (current version):

    // my_composite_codec.rs
    // ...
            #[inline]
            pub fn my_field_name_encoder(self) -> MyFieldNameEncoder<Self> {
                                               // ^---- Incorrect name based on field name
                let offset = self.offset;
                MyFieldNameEncoder::default().wrap(self, offset)
             // ^---- Same here
            }
    
    

    Generated Rust output (fixed version):

    // my_composite_codec.rs
    // ...
            #[inline]
            pub fn my_field_name_encoder(self) -> MyNestedCompositeEncoder<Self> {
                                               // ^---- This is the correct name
                let offset = self.offset;
                MyNestedCompositeEncoder::default().wrap(self, offset)
             // ^---- Same here
            }
    
    opened by YongJieYongJie 0
  • Fixes for Rust Composites

    Fixes for Rust Composites

    Fixed several issues: 1: LibRsDef explicitly use crate:: to disambiguate from built in mods like bool 2: Added acting_version field to Composite structs to fix compilation errors when using Composite structs. This is an incomplete implementation because the parent doesn't pass the acting_version to the composite because you need to change the signature of wrap(parent, offset) to include the acting_version, so this version just ensures that if the acting_version isn't set on the composite, it disregards the version check. 3: fixed primitiveArrayDecoder to return an empty array of the right size if less than version required. 4: If a name of an element clashes with a rust type, prefix it with r#, which is the preferred way of dealing with clashes with keywords

    opened by adam-talos 1
  • Add CodeQL workflow

    Add CodeQL workflow

    Hi real-logic/simple-binary-encoding!

    This is not an automatic, 🤖-generated PR, as you can check in my GitHub profile, I work for GitHub and I am part of the GitHub Security Lab which is helping out with the migration of LGTM configurations to Code Scanning. You might have heard that we've integrated LGTM's underlying CodeQL analysis engine natively into GitHub. The result is GitHub code scanning!

    With LGTM fully integrated into code scanning, we are focused on improving CodeQL within the native GitHub code scanning experience. In order to take advantage of current and future improvements to our analysis capabilities, we suggest you enable code scanning on your repository. Please take a look at our blog post for more information.

    This pull request enables code scanning by adding an auto-generated codeql.yml workflow file for GitHub Actions to your repository — take a look! We tested it before opening this pull request, so all should be working :heavy_check_mark:. In fact, you might already have seen some alerts appear on this pull request!

    Where needed and if possible, we’ve adjusted the configuration to the needs of your particular repository. But of course, you should feel free to tweak it further! Check this page for detailed documentation.

    Questions? Check out the FAQ below!

    FAQ

    Click here to expand the FAQ section

    How often will the code scanning analysis run?

    By default, code scanning will trigger a scan with the CodeQL engine on the following events:

    • On every pull request — to flag up potential security problems for you to investigate before merging a PR.
    • On every push to your default branch and other protected branches — this keeps the analysis results on your repository’s Security tab up to date.
    • Once a week at a fixed time — to make sure you benefit from the latest updated security analysis even when no code was committed or PRs were opened.

    What will this cost?

    Nothing! The CodeQL engine will run inside GitHub Actions, making use of your unlimited free compute minutes for public repositories.

    What types of problems does CodeQL find?

    The CodeQL engine that powers GitHub code scanning is the exact same engine that powers LGTM.com. The exact set of rules has been tweaked slightly, but you should see almost exactly the same types of alerts as you were used to on LGTM.com: we’ve enabled the security-and-quality query suite for you.

    How do I upgrade my CodeQL engine?

    No need! New versions of the CodeQL analysis are constantly deployed on GitHub.com; your repository will automatically benefit from the most recently released version.

    The analysis doesn’t seem to be working

    If you get an error in GitHub Actions that indicates that CodeQL wasn’t able to analyze your code, please follow the instructions here to debug the analysis.

    How do I disable LGTM.com?

    If you have LGTM’s automatic pull request analysis enabled, then you can follow these steps to disable the LGTM pull request analysis. You don’t actually need to remove your repository from LGTM.com; it will automatically be removed in the next few months as part of the deprecation of LGTM.com (more info here).

    Which source code hosting platforms does code scanning support?

    GitHub code scanning is deeply integrated within GitHub itself. If you’d like to scan source code that is hosted elsewhere, we suggest that you create a mirror of that code on GitHub.

    How do I know this PR is legitimate?

    This PR is filed by the official LGTM.com GitHub App, in line with the deprecation timeline that was announced on the official GitHub Blog. The proposed GitHub Action workflow uses the official open source GitHub CodeQL Action. If you have any other questions or concerns, please join the discussion here in the official GitHub community!

    I have another question / how do I get in touch?

    Please join the discussion here to ask further questions and send us suggestions!

    opened by pwntester 0
  • Inconsistent handling of constant presence in groupSizeEncoding

    Inconsistent handling of constant presence in groupSizeEncoding

    I'm new here, but believe I observe the following:

    For the composite 'groupSizeEncoding' an encoder/decoder class will be generated, but not used. Code for handling the repeating group will be separately generated at point of usage based on the composite definition in the spec. This is, at least, confusing.

    If the blockLength type interior to groupSizeEncoding is specified like: <type name="blockLength" primitiveType="uint16" presence="constant">0</type>

    The unused generated decoder will correctly implement the constant presence, but the point of usage code will ignore constant presence. The two copies of the code will have incompatible size and offsets. This seems incorrect.

    Would it be accurate to say that constant presence for blockLength is not supported? If so, would it make sense for a specification like this to cause an error, rather than generating unexpected code?

    opened by lukepalmer 1
Releases(1.27.0)
  • 1.27.0(Oct 11, 2022)

    • [Java] Add preview support for package override on types from SBE 2.0. PR #904, PR #915.
    • Add support for transforming a schema to generate code for older versions when checking compatibility.
    • [Java] Support binary var data in JsonPrinter. PR #914.
    • [Java] Qualify methods with name yield to avoid warning in Java 17. PR #910.
    • Carry down the the sinceVersion value into composite types to make parsing OTF easier. Issue #908.
    • Set the size for constant encoded data type to be zero in IR.
    • Upgrade to Google Test 1.21.1.
    • Upgrade to Mockito 4.7.0.
    • Upgrade to JUnit 5.9.1.
    • Upgrade to Gradle 7.5.1.
    • Upgrade to Agrona 1.17.1.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.26.0(Jul 11, 2022)

    • [Java] Provide raw access to type for bitset. Issue #893.
    • [C++] Generate skip function with loop rather than lambda to avoid compilation errors on some compilers. Issue #903.
    • Check sinceVersion is valid on elements against schema version. Issue #902.
    • [Rust] Fix generated comparator when value is NaN. PR #896.
    • [Rust] Fix resource leak in Rust generator running in Java SBE compiler.
    • [Java] Generate @SuppressWarnings annotation on enums types. Issue #894.
    • Upgrade to Mockito 4.6.1.
    • Upgrade to Agrona 1.16.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.25.3(Apr 14, 2022)

  • 1.25.2(Mar 18, 2022)

    • [Java] Calling toString() on decoder should not disturb repeating group. PR #892.
    • [All] Generate NULL_VALUE for enums based on correct type. Issue #889.
    • [C++] Check for version before outputting to stream for composites and bitsets. PR #888.
    • [Java] Charset handling improvements for more efficient string decoding. PR #887.
    • Add descriptions to schema for IR.
    • Set warnings as errors in build and clean up existing warnings.
    • Add ability to set the error stream in ParserOptions.
    • Remove some unnecessary checks in XmlSchemaParser to improve generation performance.
    • Upgrade to Versions 0.42.0.
    • Upgrade to Checkstyle 9.3.
    • Upgrade to JMH 1.34.
    • Upgrade to Mockito 4.4.0.
    • Upgrade to Shadow 7.1.2.
    • Upgrade to JUnit 5.8.2.
    • Upgrade to Gradle 7.4.1.
    • Upgrade to Agrona 1.15.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.25.1(Nov 24, 2021)

  • 1.25.0(Nov 17, 2021)

    • [C#] Add support for wrapping ArraySegment in DirectBuffer. PR #878.
    • [Java] Add @Deprecated annotation to message codecs that have been marked as deprecated in SBE schema. PR #876.
    • [C++] Only generate necessary macros and includes for enum types.
    • [Rust] Reinstate improved Rust codec generation.
    • [Java] Add sbeSkip and sbeDecodedLength methods to codecs. PR #865.
    • Upgrade to JUnit 5.8.1.
    • Upgrade to Shadow 7.1.0.
    • Upgrade to Mockito 4.0.0.
    • Upgrade to Checkstyle 9.1.
    • Upgrade to JMH 1.33.
    • Upgrade to Gradle 7.2.
    • Upgrade to Agrona 1.13.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.24.0(Aug 6, 2021)

    • [Java] Improve efficiency of encoding CharSequence when ASCII character encoding is used.
    • [Java] Check that IR codecs are regenerated on codec change.
    • [Java/C++/C] Add sbeRewind to decoders so they can rewind to the beginning of a message and begin decoding again.
    • Upgrade to Google Test 1.11.0
    • Upgrade to ByteBuddy 1.11.3
    • Upgrade to Mockito 3.11.2.
    • Upgrade to Gradle 7.1.1.
    • Upgrade to Agrona 1.12.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.23.0(Jun 13, 2021)

    • [C#] Add ToString() method to generated codecs. PR #853.
    • [C#] Add more string read and write methods to codecs. PR #845.
    • [Go] Fix tokens passed when generating composites. PR #850.
    • [C/C++] Add Clang 12 to the matrix build.
    • Upgrade to Versions 0.39.0.
    • Upgrade to JUnit 5.7.2.
    • Upgrade to Mockito 3.11.1.
    • Upgrade to ByteBuddy 1.11.2.
    • Upgrade to JMH 1.32.
    • Upgrade to Shadow 7.0.0.
    • Upgrade to Gradle 7.0.2.
    • Upgrade to Agrona 1.11.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.22.0(Apr 16, 2021)

    • [C#] Convert benchmarks to use DotNetBenchmark. PR #841.
    • [C#] Help compiler to inline DirectBuffer.CheckLimit for better performance. PR #840.
    • Add Java 17-ea to the build matrix.
    • Test with CME FixBinary version 9.
    • [C++] Capture actingBlockLength and use it for prettying printing output in version aware codecs. Issue #839.
    • [C++] Remove invalid use of noexcept on resetCountToIndex. Issue #838.
    • [Java] Generate a wrapAndApplyHeader method for decoders to verify template then apply acting version and offset.
    • [C++] Clean up and improve samples.
    • [C++] Pass elements by reference rather than value in forEach for more efficient skip operations.
    • Upgrade to Version 0.38.0.
    • Upgrade to JMH 1.29.
    • Upgrade to JUnit 5.7.1.
    • Upgrade to Mockito 3.9.0.
    • Upgrade to Gradle 6.8.3.
    • Upgrade to Agrona 1.10.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.21.0(Jan 15, 2021)

    • [Java] Generate metadata methods to groups for min and max count values. PR #834.
    • [Java] Declare generated classes as final to allow for better optimisation and increased performance.
    • [C++] Null terminate constant char arrays to avoid warnings. Issue #832.
    • [C++] Add support for decoding unknown enum values from later versions by enabling sbe.decode.unknown.enum.values and provide access to raw values.
    • Clean up warnings in bash scripts.
    • Upgrade to Checkstyle 8.39.
    • Upgrade to Mockito 3.7.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.20.4(Dec 22, 2020)

    • Add parsing of CME iLink order protocol to tests.
    • [C++] Generate literal marked up for correct type when shifting for bitsets using uint64. PR #829.
    • [Java] Generate @return in javadoc for bitset values. Issue #828.
    • [Java] Escape basic HTML characters for javadoc. Issue #826.
    • [Go] Fix decoding into struct with pre-allocated slices. PR #825.
    • [C++] Fix spelling of _cplusplus macro. Issue #821.
    • [C#] Fix repeating group encode overflow. PR #823.
    • [C] Fix argument order for wrap_and_apply_header methods. PR #821.
    • [Java] Generate bulk methods for fixed length uint8 arrays. PR #819.
    • Add Java 16 EA to the build matrix.
    • Upgrade to JMH 1.27.
    • Upgrade to Checkstyle 8.38.
    • Upgrade to Versions 0.36.0.
    • Upgrade to Mockito 3.6.28.
    • Upgrade to Gradle 6.7.1.
    • Upgrade to Agrona 1.9.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.20.3(Oct 13, 2020)

    • Fix generation of GroupOrder annotations on Java codecs.
    • Java codecs have a new method MessageDecoderFlyweight.appendTo(StringBuilder) for the interface. PR #220.
    • Reduce scope of some fields in Java codecs which were protected and should have been package or private.
    • Add GCC 10 and Clang 10 to build mix.
    • Add Java 15 to build mix.
    • Upgrade to JMH 1.26.
    • Upgrade to Checkstyle 8.36.2.
    • Upgrade to Mockito 3.5.13.
    • Upgrade to Agrona 1.8.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.20.2(Sep 18, 2020)

    • Indicate files are auto-generated for C# so they can be ignored by VS. PR #814.
    • Fix CMake install on MacOS.
    • Support CMake 3.6.1 as a minimum version.
    • Upgrade to Versions 0.33.0.
    • Upgrade to JUnit 5.7.0.
    • Upgrade to Agrona 1.7.2.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.20.1(Sep 12, 2020)

    • Validate enum values are in the range for type considering min and max value, excluding null value.
    • Upgrade to Versions 0.31.0.
    • Upgrade to Checkstyle 8.36.
    • Upgrade to Mockito 3.5.10.
    • Upgrade to JMH 1.25.2.
    • Upgrade to Agrona 1.7.1.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.20.0(Sep 12, 2020)

    • Improve validation and error messages when parsing invalid SBE schemas.
    • Generate SBE header constants in C++ codecs so they can be used more easily in switch statements.
    • Generate C++ style casts rather than C style casts in C++ codecs.
    • Generate enum literals beyond int range correctly for C and C++.
    • Enum get methods for C++ codecs should not be noexcept. PR #806.
    • Improve javadoc.
    • Update Gradle build script so IDEA can more easily get the dependencies correct for generated code.
    • Upgrade to Gradle 6.6.1.
    • Upgrade to Mockito 3.5.7.
    • Upgrade to JMH 1.25.1.
    • Upgrade to Checkstyle 8.35.
    • Upgrade to Version 0.29.0.
    • Upgrade to Agrona 1.7.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.19.0(Jul 14, 2020)

    • Improve edge case generation for JsonPrinter when groups have num in group of zero. Issue #800.
    • Improve edge case generation for JsonPrinter when messages are empty. Issue #799.
    • Fix case of generated variables in C++ computeLength method which could cause compilation errors. Issue #796.
    • Enable Java 14 build in CI.
    • Improve Javadoc.
    • Remove use of constexpr from computeLength generated methods in C++ codecs to address compilation errors.
    • Validate usage of group and var data types within composite elements when XSDs are not used.
    • Improve C# samples.
    • Improve C++ samples.
    • Reduce warnings in C++ build on Windows.
    • Upgrade to .net core 3.1 LTS for for CI.
    • Upgrade to Mockito 3.4.0.
    • Upgrade to Checkstyle 8.34.
    • Upgrade to Shadow 6.0.0.
    • Upgrade to Gradle 6.5.1.
    • Upgrade to Agrona 1.6.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.18.2(May 27, 2020)

  • 1.18.1(May 23, 2020)

  • 1.18.0(May 23, 2020)

    • Only generate imports for Java codecs when required to address warnings.
    • Access fixed length arrays as Spans in C# codecs. PR #780.
    • Add SbeSchemaId and SbeSchemaVersion as constants in fixed flyweights for C# codecs.
    • Generate source docs from schema descriptions for C# codecs. PR #778.
    • Add offset and wrap methods to C# codecs for ease of use. PR #777.
    • Support non-standard message headers form the C++ codecs. PR #775.
    • Fix version support for enums in C codecs. Issue #773.
    • Improve formatting of generated C codecs.
    • Require a strict dependency on Agrona.
    • Upgrade to Agrona 1.5.0.
    • Upgrade to javadoc-links 5.1.0.
    • Upgrade to JUnit 5.6.2.
    • Upgrade to Gradle 6.4.1.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.17.0(Mar 28, 2020)

    • Remove redundant generated casts from Java codecs. Issue #766.
    • Add ability to compute the length of an encoded message before encoding for C++ codecs. PR #761.
    • Remove Rust support due to lack of community support.
    • Use sinceVersion from field which overrides type in C# codecs. Issue #763.
    • Upgrade to JUnit 5.6.1.
    • Upgrade to Gradle 6.3.
    • Upgrade to Agrona 1.4.1.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.16.3(Feb 28, 2020)

    • Re-generate Java IR codecs.
    • Make Java codec toString() methods version aware so the wrapped actingVersion and actingBlockLength are accounted for.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.16.2(Feb 27, 2020)

    • Variable data string improvements in C++ codecs. PR #758.
    • Short circuit toString and appendTo in Java codes when buffer is null to avoid debugging issues.
    • CLang Tidy fixes for C++ codecs. PR #757.
    • Eliminate allocation when calling appendTo on set based fields in Java codecs. PR #755.
    • Support optional groups which is a non-standard feature. Issue #754.
    • Fix issue with populating optional arrays in Go codecs. PR #750.
    • Migrate to maven-publish plugin for Gradle.
    • C++ codec sbePosition() method should be SBE_NOEXCEPT.
    • Upgrade to Versions 0.28.0.
    • Upgrade to Mockito 3.3.0.
    • Upgrade to Agrona 1.4.0.
    • Upgrade to Gradle 6.2.1.
    • Upgrade to JMH 1.23.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.16.1(Jan 21, 2020)

    • Simplify the generation of MetaAttribute lookup on fields in C++ codecs to reduce code size.
    • Allow setting a count for groups and then later setting it to the value of index. This allows for streaming operations when the count is not known but can be limited and then later set to index for what is encoded. Applied to Java, C++, and C# codecs. PR #746.
    • Fix generation of Java Decoder names when message names start with lowercase. Issue #745.
    • Upgrade to Agrona 1.3.0.
    • Upgrade to JUnit 5.6.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.16.0(Jan 10, 2020)

    • Generate simpler MetaAttribute lookup in Java Codecs which avoids the generation of a synthetic class.
    • Don't default timeunit and epoch for var data fields when parsing. This results in more compact IR.
    • Move CI build to GitHub Actions.
    • Track initialOffset on Java codecs so toString() can be called on the decoder at any time during decoding. This allows SBE decoding to be debugged.
    • Use skip methods to avoid unused warning in C++ codecs.
    • Upgrade to Agrona 1.2.0.
    • Upgrade to Checkstyle 8.28.
    • Upgrade to JUnit 5.6.0-RC1.
    • Upgrade to javadoc-links 4.1.6.
    • Upgrade to Mockito 3.2.4.
    • Upgrade to Gradle 6.0.1.
    • Upgrade to gtest 1.10.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.15.0(Nov 21, 2019)

    • Add skip methods for var data fields in C++ and Java codecs.
    • Handle encoding of Nan, Infinity, and -Infinity for floating point values in Java JsonPrinter.
    • Fix handling of sinceVersion in IR, Java codecs, and C++ codecs so that the max value is taken from type in field.
    • Set javadoc encoding to UTF-8.
    • Upgrade to javadoc-links 4.1.4.
    • Upgrade to Hamcrest 2.2.
    • Upgrade to Checkstyle 8.26.
    • Upgrade to Shadow 5.2.0.
    • Upgrade to Agrona 1.1.0.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.14.1(Nov 6, 2019)

    • Regenerate the Java IR Codecs to benefit for less allocation when pretty printing.
    • Add null enum value lookup into the switch for Java codecs. PR #732.
    • Upgrade to JMH 1.22.
    • Upgrade to Agrona 1.0.11.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.14.0(Oct 27, 2019)

    • Provide consistent API for constant char arrays to be compatible with mutable char arrays.
    • Cast length type for memcpy to address warning. PR #730.
    • Reduce string copying to improve performance of C++ codec generators.
    • Provide property accessor methods for getting string values Json escaped in C++ codecs.
    • Escape Json string in C++ codec pretty printing. Issue #718.
    • Escape Json strings in Java JsonTokenListener.
    • Upgrade to javadoc-links 4.1.3.
    • Upgrade to Checkstyle 8.25.
    • Upgrade to Gradle 5.6.3.
    • Upgrade to Agrona 1.0.9.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.13.3(Oct 9, 2019)

    • Fix type conversion warnings in C++ codecs. PR #725.
    • Allow setting of raw value for bit sets in C++ codecs. PR #724.
    • Fix initialisation of C++ codecs samples for printing as JSON.
    • Print non-printable char in C++ codecs as int. PR #716.
    • Fix unsigned conversation warning and use better default value for group index initialisation in C++ codecs. Issue #714.
    • Declare primitive accessors as noexcept for C++ codecs. PR #709.
    • Upgrade to Checkstyle 8.24.
    • Upgrade to javadoc-links 4.1.2.
    • Upgrade to Mockito 3.1.0.
    • Upgrade to Gradle 5.6.2.
    • Upgrade to Agrona 1.0.8.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
  • 1.13.2(Aug 16, 2019)

  • 1.13.1(Aug 14, 2019)

    • Reduce allocation and copying in Java codec generator to make stub generation more efficient.
    • Update C# README.
    • Generate package-info.java for codec Javadoc. Issue #703.
    • Upgrade to javadoc-links 3.8.4.
    • Upgrade to Agrona 1.0.7.

    Java binaries can be found here...

    Source code(tar.gz)
    Source code(zip)
Owner
Real Logic
Commercial support and development for Aeron & SBE
Real Logic
Simple Binary Encoding (SBE) - High Performance Message Codec

Simple Binary Encoding (SBE) SBE is an OSI layer 6 presentation for encoding and decoding binary application messages for low-latency financial applic

Real Logic 2.8k Dec 28, 2022
Classpy is a GUI tool for investigating Java class file, Lua binary chunk, Wasm binary code, and other binary file formats.

Classpy Classpy is a GUI tool for investigating Java class file, Lua binary chunk, Wasm binary code, and other binary file formats. Inspiration This t

null 1k Dec 17, 2022
📬Zola (Extremely) Simple Message Queue for spring, It is the simplest Message Queue you've ever experienced.

It is the simplest Message Queue you've ever experienced. home (engish version) korean version docs ?? Getting Started || Overview || Docs ZSMQ ZSMQ (

onigiri 42 Nov 23, 2022
Fast integer compression in C using the StreamVByte codec

streamvbyte StreamVByte is a new integer compression technique that applies SIMD instructions (vectorization) to Google's Group Varint approach. The n

Daniel Lemire 281 Dec 27, 2022
Java library for representing, parsing and encoding URNs as in RFC2141 and RFC8141

urnlib Java library for representing, parsing and encoding URNs as specified in RFC 2141 and RFC 8141. The initial URN RFC 2141 of May 1997 was supers

SLUB 24 May 10, 2022
Comparison between Java and Common Lisp solutions to a phone-encoding problem described by Prechelt

Prechelt Phone Number Encoding This project implements the phone number encoding described by Lutz Prechelt in his article for the COMMUNICATIONS OF T

Renato Athaydes 27 Nov 30, 2021
A high available,high performance distributed messaging system.

#新闻 MetaQ 1.4.6.2发布。更新日志 MetaQ 1.4.6.1发布。更新日志 MetaQ 1.4.5.1发布。更新日志 MetaQ 1.4.5发布。更新日志 Meta-ruby 0.1 released: a ruby client for metaq. SOURCE #介绍 Meta

dennis zhuang 1.3k Dec 12, 2022
This app is simple and awesome notepad. It is a quick notepad editing experience when writing notes,emails,message,shoppings and to do list.

This app is simple and awesome notepad. It is a quick notepad editing experience when writing notes,emails,message,shoppings and to do list.It is easy to use and enjoy hassle free with pen and paper.

Md Arif Hossain 1 Jan 18, 2022
ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.

ANTLR v4 Build status ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating

Antlr Project 13.6k Dec 28, 2022
Binary Artifact Management Tool

Artipie is an experimental binary artifact management tool, similar to Artifactory, Nexus, Archiva, ProGet, and many others. The following set of feat

Artipie 327 Dec 24, 2022
Java binary serialization and cloning: fast, efficient, automatic

Kryo is a fast and efficient binary object graph serialization framework for Java. The goals of the project are high speed, low size, and an easy to u

Esoteric Software 5.7k Jan 5, 2023
binary serialization format

Colfer Colfer is a binary serialization format optimized for speed and size. The project's compiler colf(1) generates source code from schema definiti

Pascal S. de Kloe 680 Dec 25, 2022
ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.

ANTLR v4 Build status ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating

Antlr Project 13.6k Jan 3, 2023
Apache POI - A Java library for reading and writing Microsoft Office binary and OOXML file formats.

Apache POI A Java library for reading and writing Microsoft Office binary and OOXML file formats. The Apache POI Project's mission is to create and ma

The Apache Software Foundation 1.5k Jan 1, 2023
RSocket is a binary protocol for use on byte stream transports such as TCP, WebSockets, and Aeron

RSocket RSocket is a binary protocol for use on byte stream transports such as TCP, WebSockets, and Aeron. It enables the following symmetric interact

RSocket 2.2k Dec 30, 2022
log4j2-scan is a single binary command-line tool for CVE-2021-44228 vulnerability scanning and mitigation patch

log4j2-scan is a single binary command-line tool for CVE-2021-44228 vulnerability scanning and mitigation patch. It also supports nested JAR file scan

Logpresso GitHub 839 Dec 29, 2022
GalaxyCDC is a core component of PolarDB-X which is responsible for global binary log generation, publication and subscription.

中文文档 What is ApsaraDB GalaxyCDC ? GalaxyCDC is a core component of PolarDB-X which is responsible for global binary log generation, publication and su

null 56 Dec 19, 2022
Spring-Boot-Plus is a easy-to-use, high-speed, high-efficient,feature-rich, open source spring boot scaffolding

Everyone can develop projects independently, quickly and efficiently! What is spring-boot-plus? A easy-to-use, high-speed, high-efficient, feature-ric

geekidea 2.3k Dec 31, 2022
Elegance, high performance and robustness all in one java bean mapper

JMapper Framework Fast as hand-written code with zero compromise. Artifact information Status Write the configuration using what you prefer: Annotatio

null 200 Dec 29, 2022
A high performance caching library for Java

Caffeine is a high performance, near optimal caching library. For more details, see our user's guide and browse the API docs for the latest release. C

Ben Manes 13k Jan 3, 2023