Nifty is an implementation of Thrift clients and servers on Netty

Related tags

Networking nifty
Overview

Project Status: 🚨 Unmaintained 🚨

This project is archived and no longer maintained. At the time of archiving, open issues and pull requests were closed and tagged with 2018-05-archive. For pre-existing users who need an open source alternative, we recommend taking a look at airlift/drift.

Nifty

Nifty is an implementation of Thrift clients and servers on Netty.

It is also the implementation used by Swift.

Examples

To create a basic Thrift server using Nifty, use the Thrift 0.9.0 code generator to generate Java stub code, write a Handler for your service interface, and pass it to Nifty like this:

public void startServer() {
    // Create the handler
    MyService.Iface serviceInterface = new MyServiceHandler();

    // Create the processor
    TProcessor processor = new MyService.Processor<>(serviceInterface);

    // Build the server definition
    ThriftServerDef serverDef = new ThriftServerDefBuilder().withProcessor(processor)
                                                            .build();

    // Create the server transport
    final NettyServerTransport server = new NettyServerTransport(serverDef,
                                                                 new NettyServerConfigBuilder(),
                                                                 new DefaultChannelGroup(),
                                                                 new HashedWheelTimer());

    // Create netty boss and executor thread pools
    ExecutorService bossExecutor = Executors.newCachedThreadPool();
    ExecutorService workerExecutor = Executors.newCachedThreadPool();

    // Start the server
    server.start(bossExecutor, workerExecutor);

    // Arrange to stop the server at shutdown
    Runtime.getRuntime().addShutdownHook(new Thread() {
        @Override
        public void run() {
            try {
                server.stop();
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        }
    });
}

Or the same thing using guice:

public void startGuiceServer() {
    final NiftyBootstrap bootstrap = Guice.createInjector(
        Stage.PRODUCTION,
        new NiftyModule() {
            @Override
            protected void configureNifty() {
                // Create the handler
                MyService.Iface serviceInterface = new MyServiceHandler();

                // Create the processor
                TProcessor processor = new MyService.Processor<>(serviceInterface);

                // Build the server definition
                ThriftServerDef serverDef = new ThriftServerDefBuilder().withProcessor(processor)
                                                                        .build();

                // Bind the definition
                bind().toInstance(serverDef);
            }
        }).getInstance(NiftyBootstrap.class);

    // Start the server
    bootstrap.start();

    // Arrange to stop the server at shutdown
    Runtime.getRuntime().addShutdownHook(new Thread() {
        @Override
        public void run() {
            bootstrap.stop();
        }
    });
}
Comments
  • Add in queueTimeout support to nifty

    Add in queueTimeout support to nifty

    Tuning load shedding via a fixed queue size is difficult, especially in heterogeneous workloads. Adding a queue timeout of let's say 100ms will allow a server to shed work if it is backing up while still being able to perform meaningful work that clients are actually waiting on and haven't timed out already.

    Tasks are expired off the queue in the minimum of queueTimeout or taskTimeout. By default queueTimeout is configured to be 0 which makes this diff a noop.

    CLA Signed 
    opened by tageorgiou 8
  • Add NiftyIODispatcher

    Add NiftyIODispatcher

    Adds a NiftyIODispatcher which serializes downstream messages that come from worker threads to be back on the IO threads.

    This has some benefits for some classes like SSL which otherwise have to take locks on the IO thread to deal with being called from various threads.

    CLA Signed 
    opened by siyengar 6
  • buffer overflow? `Maximum frame size of -2147483648 exceeded `

    buffer overflow? `Maximum frame size of -2147483648 exceeded `

    I'm sending a thrift struct of < 300MB.

    Versions:

    libraryDependencies ++= Seq(
      "com.facebook.nifty" % "nifty-core" % "0.18.0",
      "com.facebook.nifty" % "nifty-client" % "0.18.0",
      "com.facebook.swift" % "swift-service" % "0.18.0",
      "com.facebook.swift" % "swift-codec" % "0.18.0",
      "com.facebook.swift" % "swift-annotations" % "0.18.0"
    )
    

    Using java8 VM, oracle JDK.

    Thrift Server Setup:

        import io.airlift.units.DataSize.Unit.GIGABYTE;
        import io.airlift.units.DataSize
        val serverConfig = new ThriftServerConfig()
          .setBindAddress(config.schedulerEndpoint.ip)
          .setPort(config.schedulerEndpoint.port)
          .setMaxFrameSize(new DataSize(2, GIGABYTE)) // the default is 64MB
    
        val server = new ThriftServer(processor, serverConfig)
    
        Runtime.getRuntime.addShutdownHook(new Thread() {
          override def run() = {
            Try(server.close) match {
              case Success(p) => logger.info(s"Successfully stopped server")
              case Failure(exn) => logger.error(s"Error stoping service $exn")
            }
          }
        });
        server.start
    

    The overflow?

    org.jboss.netty.handler.codec.frame.TooLongFrameException: Maximum frame size of -2147483648 exceeded
            at com.facebook.nifty.codec.DefaultThriftFrameDecoder.tryDecodeFramedMessage(DefaultThriftFrameDecoder.java:102)
            at com.facebook.nifty.codec.DefaultThriftFrameDecoder.decode(DefaultThriftFrameDecoder.java:68)
            at com.facebook.nifty.codec.DefaultThriftFrameDecoder.decode(DefaultThriftFrameDecoder.java:33)
            at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
            at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
            at com.facebook.nifty.codec.DefaultThriftFrameCodec.handleUpstream(DefaultThriftFrameCodec.java:42)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
            at com.facebook.nifty.core.ChannelStatistics.handleUpstream(ChannelStatistics.java:79)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124)
            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124)
            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    
    

    Things I've tried:

    • Upgraded everything from 0.15.1 to 0.18.0 nifty & swfit.
    • Regenerated the python structs (client) & swift (with the facebook swift generator)
    • Problem only visible for big structs > 90MB
    • I also tried overriding my own thrift frme codec factory like this:
    
        import com.facebook.nifty.codec.{
          ThriftFrameCodecFactory,
          DefaultThriftFrameCodec}
        import org.apache.thrift.protocol.TProtocolFactory
        import org.jboss.netty.channel.ChannelHandler
    
        val frameFactory = new ThriftFrameCodecFactory(){
          override def create(maxFrameSize: Int,
            defaultProtocolFactory: TProtocolFactory): ChannelHandler = {
            import com.google.common.base.Verify.verify
            verify(maxFrameSize > 0, s"Frame size ($maxFrameSize) is negative!")
            new DefaultThriftFrameCodec(size, defaultProtocolFactory)
          }
        }
    
        import com.facebook.nifty.core.NiftyTimer
        import com.google.common.collect.ImmutableMap
        import org.apache.thrift.protocol.TBinaryProtocol
        import com.facebook.nifty.codec.ThriftFrameCodecFactory
        import com.facebook.nifty.duplex.TDuplexProtocolFactory
    
        val server = new ThriftServer(processor,
          serverConfig,
          new NiftyTimer("thrift"),
          ImmutableMap.of("framed",
            frameFactory.asInstanceOf[ThriftFrameCodecFactory]),
          ImmutableMap.of("binary",
            TDuplexProtocolFactory.fromSingleFactory(new TBinaryProtocol.Factory())),
          ThriftServer.DEFAULT_WORKER_EXECUTORS,
          ThriftServer.DEFAULT_SECURITY_FACTORY)
    

    However, this fails trying to create teh ChannelHandler

            verify(maxFrameSize > 0, s"Frame size ($maxFrameSize) is negative!")
    

    I'm using the thrift 0.9.2 compiler for python, the swift 0.18.0 compiler for java.

    Any tips, suggestions would be greatly appreciated. Note that this doesn't happen between a C++ service and the python client.

    Thanks in advance!

    • Alex
    2018-05-archive 
    opened by emaxerrno 6
  • NettyServerConfigBuilder instead of NettyConfigBuilder

    NettyServerConfigBuilder instead of NettyConfigBuilder

    There is a small mistake on the README.md

        // Create the server transport
        final NettyServerTransport server = new NettyServerTransport(serverDef,
                                                                     new NettyConfigBuilder(),
                                                                     new DefaultChannelGroup(),
                                                                     new HashedWheelTimer());
    

    replace by

        // Create the server transport
        final NettyServerTransport server = new NettyServerTransport(serverDef,
                                                                     new NettyServerConfigBuilder(),
                                                                     new DefaultChannelGroup(),
                                                                     new HashedWheelTimer());
    
    opened by reddragon 5
  • Implemented an SSL config file watcher

    Implemented an SSL config file watcher

    The watcher periodically polls a ticket seed file, private key file, and certificate file for changes, and updates the server configuration when any of the watched files change.

    Also cleaned up some issues in TicketSeedFileParser, and made sure that SslServerConfiguration copies arrays / Iterables instead of storing the provided reference, to make sure they can't be modified by caller later.

    CLA Signed 
    opened by ivmaykov 4
  • Code quality fix - Dead stores should be removed.

    Code quality fix - Dead stores should be removed.

    This pull request is focused on resolving occurrences of Sonar rule squid:S1854 - Dead stores should be removed. You can find more information about the issue here: https://dev.eclipse.org/sonar/rules/show/squid:S1854

    Please let me know if you have any questions.

    Faisal Hameed

    CLA Signed 2018-05-archive 
    opened by faisal-hameed 4
  • Use unresolved address for SOCKS connections

    Use unresolved address for SOCKS connections

    This fixes hostname resolution for SOCKS, which requires that the resolution is done on the remote end by using the hostname and not the IP address when connecting through the proxy.

    CLA Signed 
    opened by electrum 4
  • NettyServerTransport can take a range of ports to start.

    NettyServerTransport can take a range of ports to start.

    Add listen(minPort, maxPort) to ThriftServerDefBuilderBase, and let NettyServerTransport to start from a range of ports. Add a unit test, which try to start from a range of ports.

    opened by qujun 4
  • Maven 3.1 can not mvn install?

    Maven 3.1 can not mvn install?

    Ubuntu 12 clean install nifty using maven 3.1,errors occur: Caused by: java.lang.ClassNotFoundException: org.sonatype.aether.version.VersionConstraint at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50) at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:244) at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:230) ... 64 more

    opened by vongosling 4
  • Split read based timeouts.

    Split read based timeouts.

    A Nifty client actually has to deal with two separate timeouts:

    • request timeout - the maximum amount of time that the caller is willing to wait for an I/O request to complete
    • receive timeout - the maximum amount of time that the client is willing for the server to make any progress

    These times can be very different. e.g. a server might trickle a request sending a packet every three seconds for a long time and the caller is willing to wait minutes for a request to complete as long as it makes progress.

    This patch fixes the name confusion around read, request and receive timeouts in the nifty client code, uses consistent naming everywhere and adds a new timeout (requestTimeout) while the existing receiveTimeout now does what the name suggests.

    Existing code should still work fine as both timeouts are set to the same value by default.

    opened by hgschmie 4
  • Channel Statistics

    Channel Statistics

    What is the best way to expose Channel Statistics for gauging metrics (to ultimately log to Graphite)? Ideally, this would be implemented without modifying or duplicating existing Nifty classes.

    2018-05-archive 
    opened by ann 4
Owner
Meta Archive
These projects have been archived and are generally unsupported, but are still available to view and use
Meta Archive
Apache Thrift is a lightweight, language-independent software stack for point-to-point RPC implementation

Apache Thrift Introduction Thrift is a lightweight, language-independent software stack for point-to-point RPC implementation. Thrift provides clean a

The Apache Software Foundation 9.5k Jan 4, 2023
Akka gRPC - Support for building streaming gRPC servers and clients on top of Akka Streams.

akka-grpc Support for building streaming gRPC servers and clients on top of Akka Streams. This library is meant to be used as a building block in proj

Akka Project 420 Dec 29, 2022
An annotation-based Java library for creating Thrift serializable types and services.

Drift Drift is an easy-to-use, annotation-based Java library for creating Thrift clients and serializable types. The client library is similar to JAX-

null 225 Dec 24, 2022
IoT Platform, Device management, data collection, processing and visualization, multi protocol, rule engine, netty mqtt client

GIoT GIoT: GIoT是一个开源的IoT平台,支持设备管理、物模型,产品、设备管理、规则引擎、多种存储、多sink、多协议(http、mqtt、tcp,自定义协议)、多租户管理等等,提供插件化开发 Documentation Quick Start Module -> giot-starte

gerry 34 Sep 13, 2022
Netty project - an event-driven asynchronous network application framework

Netty Project Netty is an asynchronous event-driven network application framework for rapid development of maintainable high performance protocol serv

The Netty Project 30.5k Jan 3, 2023
Experimental Netty-based Java 16 application/web framework

Experimental Netty-based application/web framework. An example application can be seen here. Should I use this? Probably not! It's still incredibly ea

amy null 8 Feb 17, 2022
Simple & Lightweight Netty packet library + event system

Minimalistic Netty-Packet library Create packets with ease Bind events to packets Example Packet: public class TestPacket extends Packet { privat

Pierre Maurice Schwang 17 Dec 7, 2022
An netty based asynchronous socket library for benchion java applications

Benchion Sockets Library An netty based asynchronous socket library for benchion java applications ?? Documents ?? Report Bug · Request Feature Conten

Fitchle 3 Dec 25, 2022
JNetcat : a tool to debug network issues or simulate servers

JNetcat A tool to easily debug or monitor traffic on TCP/UDP and simulate a server or client No need of telnet anymore to test for a remote connection

io-panic 3 Jul 26, 2022
Unconventional Java code for building web servers / services without a framework.

Unconventional Java code for building web servers / services without a framework. Think dropwizard but as a seed project instead of a framework. If this project had a theme it would be break the rules but be mindful of your decisions.

StubbornJava 227 Nov 15, 2022
A network core plugin for the Spigot which best Experience for Minecraft Servers.

tCore The core plugin for Spigot. (Supports 1.8.8<=) 大規模サーバー、ネットワーク等の中核となるプラグインです。プロトコルバージョン 1.8 未満での動作は確認していません。かなりの量のソースになりますが、様々な機能が実装されています。中身自体は過

null 6 Oct 13, 2022
A public bot for Discord servers 🥶

Public sample bot for Discord servers ?? Bot is written in Java 16, currently there is only verification in the bot, but if someone knows at least the

Kacper 8 Jul 7, 2022
A barebones WebSocket client and server implementation written in 100% Java.

Java WebSockets This repository contains a barebones WebSocket server and client implementation written in 100% Java. The underlying classes are imple

Nathan Rajlich 9.5k Dec 30, 2022
The Java gRPC implementation. HTTP/2 based RPC

gRPC-Java - An RPC library and framework gRPC-Java works with JDK 7. gRPC-Java clients are supported on Android API levels 16 and up (Jelly Bean and l

grpc 10.2k Jan 1, 2023
Socket.IO Client Implementation in Java

Socket.IO-Client for Java socket.io-java-client is an easy to use implementation of socket.io for Java. It uses Weberknecht as transport backend, but

Enno Boland 946 Dec 21, 2022
Java library for representing, parsing and encoding URNs as in RFC2141 and RFC8141

urnlib Java library for representing, parsing and encoding URNs as specified in RFC 2141 and RFC 8141. The initial URN RFC 2141 of May 1997 was supers

SLUB 24 May 10, 2022
Pcap editing and replay tools for *NIX and Windows - Users please download source from

Tcpreplay Tcpreplay is a suite of GPLv3 licensed utilities for UNIX (and Win32 under Cygwin) operating systems for editing and replaying network traff

AppNeta, Inc. 956 Dec 30, 2022
Android application allowing to sniff and inject Zigbee, Mosart and Enhanced ShockBurst packets on a Samsung Galaxy S20

RadioSploit 1.0 This Android application allows to sniff and inject Zigbee, Mosart and Enhanced ShockBurst packets from a Samsung Galaxy S20 smartphon

Romain Cayre 52 Nov 1, 2022