TCP/IP packet demultiplexer. Download from:

Overview

TCPFLOW 1.5.0

Downloads directory: http://digitalcorpora.org/downloads/tcpflow/

Installation

Most common GNU/Linux distributions ship tcpflow in their repositories. So on Debian/Ubuntu/etc you can say

sudo apt-get install tcpflow

and on Fedora/RedHat/CentOS/etc you can say

sudo dnf install tcpflow

And that's it. If this isn't good-enough for whatever reason, you can build from source:

Building from source

To compile for Linux

Be sure you have the necessary precursors. There are files in the root directory that will do this for you, depending on your host operating system:

CONFIGURE_ARCH_17_8.sh CONFIGURE_FEDORA_18.sh CONFIGURE_FEDORA_26.sh CONFIGURE_UBUNTU_16_04.sh

Depending on your OS, just:

# sudo bash CONFIGURE_<YOUROS>.sh

Once you have configured your OS, compile and install with:

./configure
make
sudo make install

If you want do download the development tree with git, be sure to do a complete checkout with --recursive and then run bootstrap.sh, configure and make:

git clone --recursive https://github.com/simsong/tcpflow.git
cd tcpflow
bash bootstrap.sh
./configure
make
sudo make install  

To download and compile for Amazon AMI:

ssh ec2-user@<your ec2 instance>
sudo bash yum -y install git make gcc-c++ automake autoconf boost-devel cairo-devel libpcap-devel openssl-devel zlib-devel
git clone --recursive https://github.com/simsong/tcpflow.git
sh bootstrap.sh

To Compile for Windows with mingw on Fedora Core:

yum -y install mingw64-gcc mingw64-gcc-c++ mingw64-boost mingw64-cairo mingw64-zlib
mingw64-configure
make

To use CMake, see detailed instructions: cmake/README.md

Build RPM

From a clean repository as normal user (not root):

./bootstrap.sh     # Generates the file ./configure
./configure        # Generates the file tcpflow.spec
rpmbuild -bb tcpflow.spec --build-in-place

Check the specfile and resulted RPM:

rpmlint tcpflow.spec
rpmlint ~/rpmbuild/RPMS/x86_64/tcpflow-....rpm

Install:

sudo dnf install ~/rpmbuild/RPMS/x86_64/tcpflow-....rpm

Introduction To tcpflow

tcpflow is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis and debugging. Each TCP flow is stored in its own file. Thus, the typical TCP flow will be stored in two files, one for each direction. tcpflow can also process stored 'tcpdump' packet flows.

tcpflow stores all captured data in files that have names of the form:

   [timestampT]sourceip.sourceport-destip.destport[--VLAN][cNNNN]

where: timestamp is an optional timestamp of the time that the first packet was seen T is a delimiter that indicates a timestamp was provided sourceip is the source IP address sourceport is the source port destip is the destination ip address destport is the destination port VLAN is the VLAN port c is a delimiter indicating that multiple connections are present NNNN is a connection counter, when there are multiple connections with the same [time]/sourceip/sourceport/destip/destport combination.
Note that connection counting rarely happens when timestamp prefixing is performed.

HERE are some examples:

   128.129.130.131.02345-010.011.012.013.45103

The contents of the above file would be data transmitted from host 128.129.131.131 port 2345, to host 10.11.12.13 port 45103.

   128.129.130.131.02345-010.011.012.013.45103c0005

The sixth connection from 128.129.131.131 port 2345, to host 10.11.12.13 port 45103.

   1325542703T128.129.130.131.02345-010.011.012.013.45103

A connection from 128.129.131.131 port 2345, to host 10.11.12.13 port 45103, that started on at 5:19pm (-0500) on January 2, 2012

   128.129.130.131.02345-010.011.012.013.45103--3

A connection from 128.129.131.131 port 2345, to host 10.11.12.13 port 45103 that was seen on VLAN port 3.

You can change the template that is used to create filenames with the -F and -T options. If a directory appears in the template the directory will be automatically created.

If you use the -a option, tcpflow will automatically interpret HTTP responses.

   If the output file is
      208.111.153.175.00080-192.168.001.064.37314,

   Then the post-processing will create the files:
      208.111.153.175.00080-192.168.001.064.37314-HTTP
      208.111.153.175.00080-192.168.001.064.37314-HTTPBODY

   If the HTTPBODY was compressed with GZIP, you may get a 
   third file as well:

      208.111.153.175.00080-192.168.001.064.37314-HTTPBODY-GZIP

   Additional information about these streams, such as their MD5
   hash value, is also written to the DFXML file

tcpflow is similar to 'tcpdump', in that both process packets from the wire or from a stored file. But it's different in that it reconstructs the actual data streams and stores each flow in a separate file for later analysis.

tcpflow understands sequence numbers and will correctly reconstruct data streams regardless of retransmissions or out-of-order delivery. However, tcpflow currently does not understand IP fragments; flows containing IP fragments will not be recorded properly.

tcpflow can output a summary report file in DFXML format. This file includes information about the system on which the tcpflow program was compiled, where it was run, and every TCP flow, including source and destination IP addresses and ports, number of bytes, number of packets, and (optionally) the MD5 hash of every bytestream.

tcpflow uses the LBL Packet Capture Library (available at ftp://ftp.ee.lbl.gov/libpcap.tar.Z) and therefore supports the same rich filtering expressions that programs like 'tcpdump' support. It should compile under most popular versions of UNIX; see the INSTALL file for details.

What use is it?

tcpflow is a useful tool for understanding network packet flows and performing network forensics. Unlike programs such as WireShark, which show lots of packets or a single TCP connection, tcpflow can show hundreds, thousands, or hundreds of thousands of TCP connections in context.

A common use of tcpflow is to reveal the contents of HTTP sessions. Using tcpflow you can reconstruct web pages downloaded over HTTP. You can even extract malware delivered as 'drive-by downloads.'

Jeremy Elson originally wrote this program to capture the data being sent by various programs that use undocumented network protocols in an attempt to reverse engineer those protocols. RealPlayer (and most other streaming media players), ICQ, and AOL IM are good examples of this type of application. It was later used for HTTP protocol analysis.

Simson Garfinkel founded Sandstorm Enterprises in 1998. Sandstorm created a program similar to tcpflow called TCPDEMUX and another version of the program called NetIntercept. Those programs are commercial. After Simson left Sandstorm he had need for a tcp flow reassembling program. He found tcpflow and took over its maintenance.

Bugs

Please enter bugs on the github issue tracker

tcpflow currently does not understand IP fragments. Flows containing IP fragments will not be recorded correctly. IP fragmentation is increasingly a rare event, so this does not seem to be a significant problem.

RECOMMENDED CITATION

If you are writing an article about tcpflow, please cite our technical report:

  • Passive TCP Reconstruction and Forensic Analysis with tcpflow, Simson Garfinkel and Michael Shick, Naval Postgraduate School Technical Report NPS-CS-13-003, September 2013. https://calhoun.nps.edu/handle/10945/36026

MAINTAINER

Simson L. Garfinkel [email protected]

TCPFLOW 1.6 STATUS REPORT

I continue to port bulk_extractor, tcpflow, be13_api and dfxml to modern C++. After surveying the standards I’ve decided to go with C++17 and not C++14, as support for 17 is now widespread. (I probably don’t need 20). I am sticking with autotools, although there seems a strong reason to move to CMake. I am keeping be13_api and dfxml as a modules that are included, python-style, rather than making them stand-alone libraries that are linked against. I’m not 100% sure that’s the correct decision, though.

The project is taking longer than anticipated because I am also doing a general code refactoring. The main thing that is taking time is figuring out how to detangle all of the C++ objects having to do with parser options and configuration.

Given that tcpflow and bulk_extractor both use be13_api, my attention has shifted to using tcpflow to get be13_api operational, as it is a simpler program. I’m about three quarters of the way through now. I anticipate having something finished before the end of 2020.

--- Simson Garfinkel, October 18, 2020

ACKNOWLEDGEMENTS

Thanks to:

  • Jeffrey Pang, for the radiotap implementation
  • Doug Madory, for the Wifi parser
  • Jeremy Elson, for the original idea and initial tcp/ip implementation
Comments
  • tcpflow build error with openssl 1.1

    tcpflow build error with openssl 1.1

    The build problem is in src/dfxml/src/hash_t.h, here is my patch.

    diff --git a/src/hash_t.h b/src/hash_t.h index 0f2331e..0da8e31 100644 --- a/src/hash_t.h +++ b/src/hash_t.h @@ -31,6 +31,7 @@ #ifndef HASH_T_H #define HASH_T_H

    +#include #include #include

    @@ -189,7 +190,7 @@ inline std::string digest_name<sha512_t>() {

    template<const EVP_MD md(),size_t SIZE> class hash_generator__ { / generates the hash */

    • EVP_MD_CTX mdctx; /* the context for computing the value */
    • EVP_MD_CTX* mdctx; /* the context for computing the value / bool initialized; / has the context been initialized? / bool finalized; / Static function to determine if something is zero / @@ -202,21 +203,22 @@ class hash_generator__ { / generates the hash / public: int64_t hashed_bytes; / This function takes advantage of the fact that different hash functions produce residues with different sizes */
    • hash_generator__():mdctx(),initialized(false),finalized(false),hashed_bytes(0){ }
    • hash_generator__():mdctx(nullptr),initialized(false),finalized(false),hashed_bytes(0){ } ~hash_generator__(){ release(); } void release(){ /* free allocated memory */ if(initialized){
    •   EVP_MD_CTX_cleanup(&mdctx);
      
    •   EVP_MD_CTX_free(mdctx);
        initialized = false;
        hashed_bytes = 0;
      
      } } void init(){ if(initialized==false){
    •   EVP_MD_CTX_init(&mdctx);
      
    •   EVP_DigestInit_ex(&mdctx, md(), NULL);
      
    •   mdctx = EVP_MD_CTX_new();
      
    •   if (!mdctx) throw std::bad_alloc();
      
    •   EVP_DigestInit_ex(mdctx, md(), NULL);
        initialized = true;
        finalized = false;
        hashed_bytes = 0;
      

    @@ -228,7 +230,7 @@ public: std::cerr << "hashgen_t::update called after finalized\n"; exit(1); }

    • EVP_DigestUpdate(&mdctx,buf,bufsize);
    • EVP_DigestUpdate(mdctx,buf,bufsize); hashed_bytes += bufsize; } hash__<md,SIZE> final() { @@ -242,7 +244,7 @@ public: } hash__<md,SIZE> val; unsigned int len = sizeof(val.digest);
    • EVP_DigestFinal(&mdctx,val.digest,&len);
    • EVP_DigestFinal(mdctx,val.digest,&len); finalized = true; return val; }
    opened by fefe17 26
  • Unable to see UDP traffic

    Unable to see UDP traffic

    Hi,

    Using tcpflow version 1.2.6, with libpcap 1.2.1, in a NAS which is running Linux 2.6.12, I'm unable to see any UDP traffic.

    I'm guessing is a problem in my environment, but I just tested with tcpdump and that works.

    Any ideas?

    opened by rb07 23
  • Stack-based buffer overflow in setbit()

    Stack-based buffer overflow in setbit()

    There exists a stack-based buffer overflow in setbit() function at iptree.h, invoked by get_histogram() in src/netviz/address_histogram.cpp. The issue gets triggered when the value of depth in get_histogram() is set greater than 127 & being passed to i in setbit().

    The setbit() function sets the ith bit to 1, in case where the i value exceeds more than 127, the computation goes wrong & a stack overflow is triggered.

    Tested version -

    1.5.0 (Master)

    Command -

    tcpflow -a -D -b -m -o -Fk –r $POC

    Debugging:

     /* set the ith bit to 1 */
        static void setbit(uint8_t *addr,size_t i){
            addr[i / 8] |= (1<<((7-i)&7));   //Buffer overflow
        }
    
    gef➤  p/d i
    $6 = 127
    
    
    gef➤  p/d (1<<((7-i)&7))
    $8 = 1
    
    gef➤  p/d addr[127 / 8]
    $14 = 0
    gef➤  p/d addr[126 / 8]
    $15 = 0
    gef➤  p/d addr[128 / 8]
    $16 = 96
    
    gef➤  p/d (1<<((7-127)&7))
    $18 = 1
    gef➤  p/d (1<<((7-128)&7))
    $19 = 128
    
    

    Backtrace:

    .
    .
    .
    .
    .
    #3 0x438649 in iptreet<unsigned long, 16ul>::get_histogram(std::vector<iptreet<unsigned long, 16ul>::addr_elem, std::allocator<iptreet<unsigned long, 16ul>::addr_elem> >&) const iptree.h:402
    #4 0x437d92 in address_histogram::address_histogram(iptreet<unsigned long, 16ul> const&) netviz/address_histogram.cpp:30
    #5 0x4489eb in one_page_report::render(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) netviz/one_page_report.cpp:268
    #6 0x4f7307 in scan_netviz /home/ace/Desktop/tcpflow/src/scan_netviz.cpp:74
    #7 0x493ebd in be13::plugin::phase_shutdown(feature_recorder_set&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*) be13_api/plugin.cpp:403
    #8 0x4c75ec in main /home/ace/Desktop/tcpflow/src/tcpflow.cpp:909
    #9 0x7ffff4ec082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #10 0x408f58 in _start (/usr/bin/tcpflow+0x408f58)
    

    ASAN Report

    ==24797==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffffffb840 at pc 0x000000439d28 bp 0x7fffffffb710 sp 0x7fffffffb700
    READ of size 1 at 0x7fffffffb840 thread T0
        #0 0x439d27 in iptreet<unsigned long, 16ul>::setbit(unsigned char*, unsigned long) iptree.h:245
        #1 0x438fda in iptreet<unsigned long, 16ul>::get_histogram(int, unsigned char const*, iptreet<unsigned long, 16ul>::node const*, std::vector<iptreet<unsigned long, 16ul>::addr_elem, std::allocator<iptreet<unsigned long, 16ul>::addr_elem> >&) const iptree.h:393
        #2 0x438649 in iptreet<unsigned long, 16ul>::get_histogram(std::vector<iptreet<unsigned long, 16ul>::addr_elem, std::allocator<iptreet<unsigned long, 16ul>::addr_elem> >&) const iptree.h:402
        #3 0x437d92 in address_histogram::address_histogram(iptreet<unsigned long, 16ul> const&) netviz/address_histogram.cpp:30
        #4 0x4489eb in one_page_report::render(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) netviz/one_page_report.cpp:268
        #5 0x4f7307 in scan_netviz /home/ace/Desktop/tcpflow/src/scan_netviz.cpp:74
        #6 0x493ebd in be13::plugin::phase_shutdown(feature_recorder_set&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*) be13_api/plugin.cpp:403
        #7 0x4c75ec in main /home/ace/Desktop/tcpflow/src/tcpflow.cpp:909
        #8 0x7ffff4ec082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
        #9 0x408f58 in _start (/usr/bin/tcpflow+0x408f58)
    
    Address 0x7fffffffb840 is located in stack of thread T0 at offset 208 in frame
        #0 0x438ded in iptreet<unsigned long, 16ul>::get_histogram(int, unsigned char const*, iptreet<unsigned long, 16ul>::node const*, std::vector<iptreet<unsigned long, 16ul>::addr_elem, std::allocator<iptreet<unsigned long, 16ul>::addr_elem> >&) const iptree.h:379
    
      This frame has 3 object(s):
        [32, 72) '<unknown>'
        [128, 144) 'addr0'
        [192, 208) 'addr1' <== Memory access at offset 208 overflows this variable
    HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
          (longjmp and C++ exceptions *are* supported)
    SUMMARY: AddressSanitizer: stack-buffer-overflow iptree.h:245 iptreet<unsigned long, 16ul>::setbit(unsigned char*, unsigned long)
    Shadow bytes around the buggy address:
      0x10007fff76b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x10007fff76c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x10007fff76d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x10007fff76e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1
      0x10007fff76f0: f1 f1 00 00 00 00 00 f4 f4 f4 f2 f2 f2 f2 00 00
    =>0x10007fff7700: f4 f4 f2 f2 f2 f2 00 00[f4]f4 f3 f3 f3 f3 00 00
      0x10007fff7710: 00 00 00 00 00 00 f1 f1 f1 f1 00 00 f4 f4 f3 f3
      0x10007fff7720: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x10007fff7730: f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2 00 00 00 f4
      0x10007fff7740: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
      0x10007fff7750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07 
      Heap left redzone:       fa
      Heap right redzone:      fb
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack partial redzone:   f4
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
    ==24797==ABORTING
    [Inferior 1 (process 24797) exited with code 01]
    
    

    Note: Issue reproducible when compiled with AddressSanitizer

    Please check if you are able to reproduce the issue via the Reproducer file

    opened by SegfaultMasters 16
  • tcpflow prints

    tcpflow prints "write error to stdout" after every packet

    Hi. This is forwarded from debian bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=736417

    This applies when tcpflow is invoked with -c.

    From: Frank Gevaerts [email protected] To: Debian Bug Tracking System [email protected] Subject: tcpflow prints "write error to stdout" after every packet Date: Thu, 23 Jan 2014 14:20:18 +0100 Package: tcpflow Version: 1.4.4+repack1-2 Severity: normal

    tcpflow prints "write error to stdout" after every packet. This is printed when the expected length and the actually written length are different, in tcpip.c line 272. The root cause is that the written length is not calculated correctly.

    On line 265, written is incremented by the output of fputc(), which seems to be assumed to be 1 in case a character is successfully written. However, fputc() is defined as returning the character actually written. One solution would be:

    --- tcpflow-1.4.4+repack1.orig/src/tcpip.cpp
    +++ tcpflow-1.4.4+repack1/src/tcpip.cpp
    @@ -262,7 +262,7 @@ void tcpip::print_packet(const u_char *d
         else if(demux.opt.output_strip_nonprint){
            for(const u_char *cc = data;cc<data+length;cc++){
                if(isprint(*cc) || (*cc=='\n') || (*cc=='\r')){
    -               written += fputc(*cc,stdout);
    +               written += (fputc(*cc,stdout)!=EOF);
                }
            }
         }
    

    Frank

    opened by dkogan 14
  • What is the output format of capture packet?

    What is the output format of capture packet?

    I tested tcpflow in FreeBSD and Fedora Linux to capture packets from my network interface.

    But the output binary file can't be read by wireshark. I tried auto format detection and also tcpdump pcap. But wireshark report it doesn't understand the format.

    What is the specification of tcpflow output format?

    I'm using 1.5.0 tcpflow and 3.0.3 wireshark.

    TIA

    opened by rickyzhang82 13
  • tcpflow does not properly reassemble TCP streams containing retransmissions and out of order packets

    tcpflow does not properly reassemble TCP streams containing retransmissions and out of order packets

    I am seeing an issue with tcpflow version 1.4.5 where TCP packets are not reassembled properly when they appear out of sequence in the recording.

    The sequence basically looks like this when viewed in wireshark:

    1. Seqno 1
    2. Seqno 3 [TCP previous segment not captured]
    3. Seqno 2 [TCP retransmission]
    4. Seqno 4

    When processed with tcpflow -cCs -r <pcap_file>, the bytes within the packets are printed out like so: Seqno 1 Seqno 3 Seqno 2 Seqno 4

    My assumption is that tcpflow should be reordering these based on the TCP sequence numbers. Is this one of the goals of the utility? I would be happy to dig through the code and try to fix the bug if the tool had previously functioned this way, but since has broken.

    I've got a small pcap demonstrating this which I can share via email if necessary.

    opened by mwallace582 13
  • HTTP body extraction does not handle persistent connections

    HTTP body extraction does not handle persistent connections

    When tcpflow -AH sees an HTTP response, it productes a -HTTPBODY file containing the body of the response. However, when this connection is re-used for a subsequent HTTP request and response, the -HTTPBODY file contains the first response body concatenated with subsequent HTTP response headers and bodies. -AH really only works properly for HTTP/1.1 with Connection: close.

    The scanner should extract the Content-Length header and emit multiple HTTP bodies to separate files as appropriate.

    I note that the scan_http implementation is rather simple, e.g. also not supporting HTTP/1.1 chunked transfer encoding (which is a MUST for RFC compliance). In case full HTTP support is a long time coming, a mention in the documentation with current limitations would be helpful too. tcpflow is useful for me even as-is, but knowing about possible issues ahead of time would have saved me troubleshooting.

    opened by willglynn 12
  • tcpflow 1.2.7 exits immediately without output

    tcpflow 1.2.7 exits immediately without output

    I'm using tcpflow on gentoo. Version 1.2.6 works as expected, but version 1.2.7 just exits without error message. The most information I can get is with the -v flag:

    tcpflow -v -C -i eth0

    tcpflow[16368]: printing packets to console only tcpflow[16368]: packet header dump suppressed tcpflow[16368]: converting non-printable characters to '.' tcpflow[16368]: tcpflow version 1.2.7 tcpflow[16368]: Open FDs at end of processing: 0 tcpflow[16368]: Flow map size at end of processing: 0 tcpflow[16368]: Total flows processed: 0 tcpflow[16368]: Total packets processed: 0

    I also checked the latest git checkout, here the problem is the same with just a little bit different debugging output:

    tcpflow -v -C -i wlan0

    tcpflow[14985]: printing packets to console only tcpflow[14985]: packet header dump suppressed tcpflow[14985]: converting non-printable characters to '.' tcpflow[14985]: tcpflow version 1.2.7 tcpflow[14985]: Open FDs at end of processing: 0 tcpflow[14985]: Flow map size at end of processing: 0 pcap_fake.cpp DEBUG: Total flows processed = 0 pcap_fake.cpp DEBUG: Total packets processed = 0

    Thank you for maintaining this package! Best regards.

    opened by PiQuer 12
  • tcpflow doesn't handle MPLS data

    tcpflow doesn't handle MPLS data

    Subj.

    Here's a patch to fix this:

    diff --git a/src/datalink.cpp b/src/datalink.cpp
    index 934f4c6..4b52fd5 100644
    --- a/src/datalink.cpp
    +++ b/src/datalink.cpp
    @@ -183,6 +183,16 @@ void dl_ppp(u_char *user, const struct pcap_pkthdr *h, const u_char *p)
    
     #ifdef DLT_LINUX_SLL
     #define SLL_HDR_LEN       16
    +
    +#define SLL_ADDRLEN 8
    +
    +#ifndef ETHERTYPE_MPLS
    +#define ETHERTYPE_MPLS      0x8847
    +#endif
    +#ifndef ETHERTYPE_MPLS_MULTI
    +#define ETHERTYPE_MPLS_MULTI    0x8848
    +#endif
    +
     void dl_linux_sll(u_char *user, const struct pcap_pkthdr *h, const u_char *p){
         u_int caplen = h->caplen;
         u_int length = h->len;
    @@ -196,8 +206,25 @@ void dl_linux_sll(u_char *user, const struct pcap_pkthdr *h, const u_char *p){
        DEBUG(6) ("warning: received incomplete Linux cooked frame");
        return;
         }
    +
    +    struct _sll_header {
    +        u_int16_t   sll_pkttype;    /* packet type */
    +        u_int16_t   sll_hatype; /* link-layer address type */
    +        u_int16_t   sll_halen;  /* link-layer address length */
    +        u_int8_t    sll_addr[SLL_ADDRLEN];  /* link-layer address */
    +        u_int16_t   sll_protocol;   /* protocol */
    +    };
    +
    +    _sll_header *sllp = (_sll_header*)p;
    +    int mpls_sz = 0;
    +    if (ntohs(sllp->sll_protocol) == ETHERTYPE_MPLS) {
    +        // unwind MPLS stack
    +        do {
    +            mpls_sz += 4;
    +        } while ( ((*(p+SLL_HDR_LEN + mpls_sz - 2)) & 1) == 0 );
    +    }
    
    -    packet_info pi(DLT_LINUX_SLL,h,p,tvshift(h->ts),p + SLL_HDR_LEN, caplen - SLL_HDR_LEN);
    +    packet_info pi(DLT_LINUX_SLL,h,p,tvshift(h->ts),p + SLL_HDR_LEN + mpls_sz, caplen - SLL_HDR_LEN - mpls_sz);
         process_packet_info(pi);
     }
     #endif
    

    Sorry for not providing the most common solution ;) I just needed this only case.

    Just sharing back my own patch.

    // Yury Ershov

    opened by ershov 11
  • per flow pcap structure output

    per flow pcap structure output

    Shall provide an alternative processing pkt path for extruding isolated pcap flows for processing stacked upon tcpflow lower level logic.

    Signed-off-by: Matthias Tafelmeier [email protected]

    opened by cherusk 10
  • coredump in scan_http.cpp

    coredump in scan_http.cpp

    I am trying to catch large gzip-encoded "application/amf" traffic and hit coredumps at

    scan_http_write_data_zlib(), after inflate() returned Z_STREAM_ERROR

    /usr/local/bin/tcpflow[29754]: 094.236.007.012.00080-192.168.000.100.49766: closing file /usr/local/bin/tcpflow[29754]: ::open(094.236.007.012.00080-192.168.000.100.49766,0,0)=13 /usr/local/bin/tcpflow[29754]: 094.236.007.012.00080-192.168.000.100.49766-HTTPBODY: detected zlib content, decompressing /usr/local/bin/tcpflow[29754]: ::open(094.236.007.012.00080-192.168.000.100.49766-HTTPBODY,1089,420)=15 /usr/local/bin/tcpflow[29754]: 094.236.007.012.00080-192.168.000.100.49766-HTTPBODY-2: detected zlib content, decompressing /usr/local/bin/tcpflow[29754]: ::open(094.236.007.012.00080-192.168.000.100.49766-HTTPBODY-2,1089,420)=15 29754 Segmentation Fault (core dumped) /usr/local/bin/tcpflow -a -b 500000 -d10 -AH -E http 'port 80' > /dev/null

    Ubuntu 12.04 on i383, default zlib, etc

    opened by gigantspider 10
  • Introduced change creates problem with spurious data

    Introduced change creates problem with spurious data

    Hello

    https://github.com/simsong/tcpflow/blob/6fd956d27f2293d7c05ff593a61eb03a0134ae46/src/tcpdemux.cpp#L748

    the old code here would read : uint16_t ip_payload_len = ip_len - ip_header_len;

    it was changed to: uint16_t ip_payload_len = pi.ip_datalen - ip_header_len;

    but the new codes just ignores the header information. In our use case, there are 2 to 6 paddings bytes at the end of the IP data packet, introduce by some hardware device.

    Please revert the code. Regards

    opened by aasselin 3
  • Is there a way to find media file's HTTP link of non web browser app?

    Is there a way to find media file's HTTP link of non web browser app?

    Is there a way and how to find media source file's HTTP link of non web browser application, esp. Telegram? Tried in Json format but only get so many decimal numbers after source & destination ip - port. Is there definitive steps to acquire that link, although if it will require extra utilitiy & work itd be fine. Thanks much

    opened by abdulbadii 3
  • Lack update of read_end_offset in shift_file (tcpip.cpp)

    Lack update of read_end_offset in shift_file (tcpip.cpp)

    at https://github.com/simsong/tcpflow/blob/6fd956d27f2293d7c05ff593a61eb03a0134ae46/src/tcpip.cpp#L448

    there should be an additional line: read_end_offset -= bytes_this_time;

    the code won't work if the existing data in the file is longer than buffer size

    (the SO post referenced near the shift_file function was updated with a comment) Regards

    opened by aasselin 0
  • Provide option to perform reverse DNS lookup of IP addresses

    Provide option to perform reverse DNS lookup of IP addresses

    I love tcpflow, and generally reach for it more often than tcpdump. One small issue I have with tcpflow is it cannot automatically perform reverse DNS on the ip addresses it prints out. Particularly, I'd find this feature useful when running tcpflow with the console print -c option.

    For example, tcpdump does this by default. It can be disabled in tcpdump via the -n option:

    tcpdump:
    ...
           -n     Don't convert host addresses to names.  This can be used to avoid DNS lookups.
    
    opened by dasl- 7
  • .findx file not written in real-time

    .findx file not written in real-time

    When I do a live capture on an interface I see that tcpflow writes the output/data file every second, but the .findx file (command line option -I) just stays at 0 bytes:

    vagrant@vagrant:~$ ls -lh --full-time /tmp/
    -rw-r--r-- 1 root root    6 2021-04-22 15:54:10.565581023 +0000 010.000.002.002.54150-010.000.002.015.00022
    -rw-r--r-- 1 root root    0 2021-04-22 15:54:03.633581023 +0000 010.000.002.002.54150-010.000.002.015.00022.findx
    -rw-r--r-- 1 root root 1.7K 2021-04-22 15:54:27.377581023 +0000 010.000.002.002.56520-010.000.002.015.00022
    -rw-r--r-- 1 root root    0 2021-04-22 15:54:10.553581023 +0000 010.000.002.002.56520-010.000.002.015.00022.findx
    -rw-r--r-- 1 root root  264 2021-04-22 15:54:10.565581023 +0000 010.000.002.015.00022-010.000.002.002.54150
    -rw-r--r-- 1 root root    0 2021-04-22 15:54:03.633581023 +0000 010.000.002.015.00022-010.000.002.002.54150.findx
    -rw-r--r-- 1 root root 1.7K 2021-04-22 15:54:27.377581023 +0000 010.000.002.015.00022-010.000.002.002.56520
    -rw-r--r-- 1 root root    0 2021-04-22 15:54:10.553581023 +0000 010.000.002.015.00022-010.000.002.002.56520.findx
    

    It seems it gets only written once I stop the program. This makes it impossible for me to get timestamps for the data I'm interested in while I run the capture.

    I observe this behaviour both with version 1.4.5 and 1.6.1 Is this a bug or done on purpose?

    opened by sandrocom 4
Releases(tcpflow-1.6.1)
Owner
Simson L. Garfinkel
I'm an author, programmer, university instructor, and US Government employee.
Simson L. Garfinkel
Simple & Lightweight Netty packet library + event system

Minimalistic Netty-Packet library Create packets with ease Bind events to packets Example Packet: public class TestPacket extends Packet { privat

Pierre Maurice Schwang 17 Dec 7, 2022
A packet sniffer for Realm of the Mad God in Java

A packet sniffer for Realm of the Mad God in Java. Most backend code is writen by Cortex, huge tanks to him for all the help. Inspired by work done by

Xcom 22 Dec 23, 2022
TCP/UDP client/server library for Java, based on Kryo

KryoNet can be downloaded on the releases page. Please use the KryoNet discussion group for support. Overview KryoNet is a Java library that provides

Esoteric Software 1.7k Jan 2, 2023
A High Performance Network ( TCP/IP ) Library

Chronicle-Network About A High Performance Network library Purpose This library is designed to be lower latency and support higher throughputs by empl

Chronicle Software : Open Source 231 Dec 31, 2022
Magician is an asynchronous non-blocking network protocol analysis package, supports TCP, UDP protocol, built-in Http, WebSocket decoder

An asynchronous non-blocking network protocol analysis package Project Description Magician is an asynchronous non-blocking network protocol analysis

贝克街的天才 103 Nov 30, 2022
RSocket is a binary protocol for use on byte stream transports such as TCP, WebSockets, and Aeron

RSocket RSocket is a binary protocol for use on byte stream transports such as TCP, WebSockets, and Aeron. It enables the following symmetric interact

RSocket 2.2k Dec 30, 2022
A small java project consisting of Client and Server, that communicate via TCP/UDP protocols.

Ninja Battle A small java project consisting of Client and Server, that communicate via TCP/UDP protocols. Client The client is equipped with a menu i

Steliyan Dobrev 2 Jan 14, 2022
FileServer - A multithreaded client-server program that uses Java Sockets to establish TCP/IP connection

A multithreaded client-server program that uses Java Sockets to establish TCP/IP connection. The server allows multiple clients to upload, retrieve and delete files on/from the server.

Lokesh Bisht 3 Nov 13, 2022
Pcap editing and replay tools for *NIX and Windows - Users please download source from

Tcpreplay Tcpreplay is a suite of GPLv3 licensed utilities for UNIX (and Win32 under Cygwin) operating systems for editing and replaying network traff

AppNeta, Inc. 956 Dec 30, 2022
A Linux packet crafting tool.

Pig Pig (which can be understood as Packet intruder generator) is a Linux packet crafting tool. You can use Pig to test your IDS/IPS among other stuff

Rafael Santiago 431 Dec 24, 2022
Simple & Lightweight Netty packet library + event system

Minimalistic Netty-Packet library Create packets with ease Bind events to packets Example Packet: public class TestPacket extends Packet { privat

Pierre Maurice Schwang 17 Dec 7, 2022
Bukkit transaction API for predicting when a server packet arrives at a client. Mainly intended for use in Minecraft anticheats.

Pledge A high performance and lightweight Bukkit packet tracking API for predicting when a server packet arrives at a client using transactions. Mainl

Thomazz 32 Dec 1, 2022
NoChatLag fixes lagspikes caused by chat by removing the sender UUID from the chat packet. Please note that this breaks the vanilla client side block list.

NoChatLagServer Fabric Version | Forge Version Downloads: SpigotMC | GitHub Releases NoChatLagServer fixes WEB-5587 by setting the sender-uuid to that

Noah van der Aa 3 Jan 10, 2022
Stops double clicks from impacting a player's knockback by only allowing the attack packet to be processed only once per tick per player

Stops double clicks from impacting a player's knockback by only allowing the attack packet to be processed only once per tick per player. It also moves the damage tick check to execute as soon as possible.

Jaiden 2 Oct 28, 2022
A packet sniffer for Realm of the Mad God in Java

A packet sniffer for Realm of the Mad God in Java. Most backend code is writen by Cortex, huge tanks to him for all the help. Inspired by work done by

Xcom 22 Dec 23, 2022
Java - Packet Analyzer Application based on Java, Networking and Swing UI

Network-Packet-Tracer-using-Java Java - Packet Analyzer / Sniffing System Application based on Java, Networking and Swing UI Java - Packet Analyzer Ap

Muhammad Asad 6 Feb 3, 2022
Titanium is a plugin meant to block harmful packets before they're received by the Minecraft packet handler.

Titanium Titanium is a plugin meant to block harmful packets before they're received by the Minecraft packet handler. Report Bug · Request Feature Tab

Jaden 35 Dec 7, 2022
TCP/UDP client/server library for Java, based on Kryo

KryoNet can be downloaded on the releases page. Please use the KryoNet discussion group for support. Overview KryoNet is a Java library that provides

Esoteric Software 1.7k Jan 2, 2023
A High Performance Network ( TCP/IP ) Library

Chronicle-Network About A High Performance Network library Purpose This library is designed to be lower latency and support higher throughputs by empl

Chronicle Software : Open Source 231 Dec 31, 2022
Jetserver is a high speed nio socket based multiplayer java game server written using Netty and Mike Rettig's Jetlang.It is specifically tuned for network based multiplayer games and supports TCP and UDP network protocols.

Note New version of Jetserver is called Nadron and is in a new netty 4 branch of this same repo. JetServer is a java nio based server specifically des

Abraham Menacherry 1.2k Dec 14, 2022