GoReplay is an open-source tool for capturing and replaying live HTTP traffic into a test environment in order to continuously test your system with real data. It can be used to increase confidence in code deployments, configuration changes and infrastructure changes.

Last update: Jan 4, 2023

Overview

https://goreplay.org/

GoReplay is an open-source network monitoring tool which can record your live traffic, and use it for shadowing, load testing, monitoring and detailed analysis.

About

As your application grows, the effort required to test it also grows exponentially. GoReplay offers you the simple idea of reusing your existing traffic for testing, which makes it incredibly powerful. Our state of art technique allows you to analyze and record your application traffic without affecting it. This eliminates the risks that come with putting a third party component in the critical path.

GoReplay increases your confidence in code deployments, configuration and infrastructure changes.

GoReplay offers unique approach for shadowing. Instead of being a proxy, GoReplay in background listen for traffic on your network interface, requiring no changes in your production infrastructure, rather then running GoReplay daemon on the same machine as your service.

Check latest documentation.

Installation

Download latest binary from https://github.com/buger/goreplay/releases or compile by yourself.

Getting started

The most basic setup will be sudo ./gor --input-raw :8000 --output-stdout which acts like tcpdump. If you already have test environment you can start replaying: sudo ./gor --input-raw :8000 --output-http http://staging.env.

See the our documentation and Getting started page for more info.

Subscribe to our newsletter to stay informed about the latest features and changes to Gor project.

Want to Upgrade?

We have created a GoReplay PRO extension which provides additional features such as support for binary protocols like Thrift or ProtocolBuffers, saving and replaying from cloud storage, TCP sessions replication, etc. The PRO version also includes a commercial-friendly license, dedicated support, and it also allows you to support high-quality open source development.

Problems?

If you have a problem, please review the FAQ and Troubleshooting wiki pages. Searching the issues for your problem is also a good idea.

All bug-reports and suggestions should go through Github Issues or our Google Group (you can just send email to [email protected]). If you have a private question feel free to send email to [email protected].

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Added some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request

Companies using Gor

GOV.UK - UK Government Digital Service
theguardian.com - Most popular online newspaper in the UK
TomTom - Global leader in navigation, traffic and map products, GPS Sport Watches and fleet management solutions.
3SCALE - API infrastructure to manage your APIs for internal or external users
Optionlab - Optimize customer experience and drive engagement across multiple channels
TubeMogul - Software for Brand Advertising
Videology - Video advertising platform
ForeksMobile - One of the leading financial application development company in Turkey
Granify - AI backed SaaS solution that enables online retailers to maximise their sales
And many more!

If you are using Gor, we are happy to add you to the list and share your story, just write to: [email protected]

Author

Leonid Bugaev, @buger, https://leonsbox.com

Comments

How to merge/split traffic from multiple endpoints

We have multiple endpoints for HA/load balancing reasons. To reconstruct production traffic recorded on all those nodes we would need to merge all recorded files in to one.

I noticed in recording file timestamps are not in exact order but seems to be played in order. Is that right ?

Will concatenating and sorting all records by timestamp do the job ?

Also in the future we would like to split merged file to run on N clients to emulate real clients.

We plan to split by sessions/source IP. Ideas ?

I believe we are not first who tries to do this. I know google groups would be more appropriate , just do not see any activity there.

opened by nikolai-derzhak-distillery 34
Middleware for custom request rewrite logic
Middleware

Middleware is a program that accepts request and response payload at STDIN and emits modified requests at STDOUT. You can implement any custom logic like stripping private data, advanced rewriting, support for oAuth and etc.

Original request +--------------+ +-------------+----------STDIN---------->+ | | Gor input | | Middleware | +-------------+----------STDIN---------->+ | Original response +------+---+---+ | ^ +-------------+ Modified request v | | Gor output +<---------STDOUT-----------------+ | +-----+-------+ | | | | Replayed response | +------------------STDIN----------------->----+

Middleware can be written in any language, see examples/middleware folder for examples. Middleware program should accept the fact that all communication with Gor is asynchronous, there is no guarantee that original request and response messages will come one after each other. Your app should take care of the state if logic depends on original or replayed response, see examples/middleware/token_modifier.go as example.

Simple bash echo middleware (returns same request) will look like this:

while read line; do echo $line end

Middleware can be enabled using --middleware option, by specifying path to executable file:

gor --input-raw :80 --middleware "/opt/middleware_executable" --output-http "http://staging.server"

Communication protocol

All messages should be hex encoded, new line character specifieds the end of the message, eg. new message per line.

Decoded payload consist of 2 parts: header and HTTP payload, separated by new line character.

Example request payload:

1 932079936fa4306fc308d67588178d17d823647c 1439818823587396305 GET /a HTTP/1.1 Host: 127.0.0.1

Example response payload:

2 8e091765ae902fef8a2b7d9dd960e9d52222bd8c 2782013 HTTP/1.1 200 OK Date: Mon, 17 Aug 2015 13:40:23 GMT Content-Length: 0 Content-Type: text/plain; charset=utf-8

Header contains request meta information separated by spaces. First value is payload type, possible values: 1 - request, 2 - original response, 3 - replayed response. Next goes request id: unique among all requests (sha1 of time and Ack), but remain same for original and replayed response, so you can create associations between request and responses. Third argument varies depending on payload type: for request - start time, for responses - round-trip time.

HTTP payload is unmodified HTTP requests/responses intercepted from network. You can read more about request format here, here and here. You can operate with payload as you want, add headers, change path, and etc. Basically you just editing a string, just ensure that it is RCF compliant.

At the end modified (or untouched) request should be emitted back to STDOUT, keeping original header, and hex-encoded. If you want to filter request, just not send it. Emitting responses back is optional, and does not affect anything at the moment.

Advanced example

Imagine that you have auth system that randomly generate access tokens, which used later for accessing secure content. Since there is no pre-defined token value, naive approach without middleware (or if middleware use only request payloads) will fail, because replayed server have own tokens, not synced with origin. To fix this, our middleware should take in account responses of replayed and origin server, store originalToken -> replayedToken aliases and rewrite all requests using this token to use replayed alias. See examples/middleware/token_modifier.go and middleware_test.go#TestTokenMiddleware as example of described scheme.

Latest precompiled binaries: https://www.dropbox.com/s/27chmbsqxrolvz4/gor_0.9.9_middleware_x64.tar.gz?dl=1 https://www.dropbox.com/s/vmeg5sexcleoo2e/gor_0.9.9_middleware_x86.tar.gz?dl=1
opened by buger 28
Using same input and output port causes request duplication

Hello! Amazing work you did here. However, we found a small issue at our company.

Using gor with this commandline: gor --output-http-timeout '1800s' --input-raw 'intf-ip-addr:8080' --output-http 'http://output-http-addr:8080|100%' --http-allow-url '/some/url' causes requests to be replayed multiple times on target machine (as checked on tomcat access-logs and tcpdump). Replaying request to another port (eg. 8081) fixes the issue. We confirmed the issue on gor versions 0.12.1 and 0.13.0, however, issue is not present in version 0.10.1. Machine runs binary release of gor on CentOS 6.6 2.6.32-504.12.2.el6.x86_64.

opened by walczakp 26
--output-http-track-response not returning response for replayed requests.

Hi here,

I am trying to capture responses of the replayed requests , but not able to get back responses all time . On replaying a request, the program code is getting stuck on the net.conn.Read() call. https://github.com/buger/goreplay/blob/master/http_client.go#L199 .

I see the response in tcpdump output, but not able to obtain the same from within the code using net.conn.Read()

opened by jacoblukose 22
Errors: dial tcp: cannot assign requested address
Hi, I'm seeing many errors from gor:

2014/09/04 16:20:20 Request error: Get http://....: dial tcp x.x.x.x:8080: cannot assign requested address

(x.x.x.x is an IP address and the actual http url removed).

I use gor as:

sudo ./gor --input-raw :80 --output-tcp y.y.y.y:28020

and on y.y.y.y:

./gor --input-tcp :28020 --output-http http://x.x.x.x:8080

The error seem to stem from high load in some sense. When request rate is low they are gone and everything works as expected. It's possible that the accepting http server (x.x.x.x:8080) is slow on high load. Could that be the reason for the errors on gor? Anyway, it's more of a hunch. What do these errors actually mean? And how to mitigate them? (even assuming that the accepting server is indeed slow)

I tried, out of the blue to increase ulimit and set GOMAXPROCS to various numbers but these are just shots in the dark and as expected, they didn't help.

What's your advise?

(btw, sorry if there's a mailing list I didn't find it, so I'm posting this as a bug here, it doesn't mean I think it's an actual bug with gor of course).
opened by rantav 21
Is the request id unique globally?

As the document, I expect it is.

But after I captured traffic and analyze the .gor file, I saw that request id is not seemly unique. The following picture is my traffic file and I searched one request Id but 46 matched.

And I found that tcp_message.go#UUID() function calculates the request id by src address and dest address, which seems not to be unique.

Could you please answer this question? Thanks!

opened by ZmfCn 20
Gor performance issues

Don't know which would be the best way to debug GOR for performance testing. We've been working with it for almost a month now, and I've seen an increase on the load avg on a canary 300+ req/s aprox. It reach to a point where not only gor as a listener but the replay begin to grow instance loads.

I am talking about m1.large instances on AWS 8GB ram, 2 cores Instance Store, yesterday someone mentioned that maybe it could be related to encoding characters on our requests (because we have japanese characters) but I think it doesn't make any sense.

What you think @buger? any test that I can run on my systems to detect the bottleneck? and we are running 0.7.5.

opened by rhoml 20
Limit listening side

Hi,

I have a volume problem where we get up to 20k qps on the server I need to sniff traffic from but i only need to forward a very small subset of them onto our test servers, ideally around 5-10 qps. I have tried limiting on the replay side but each time I start up I currently see the following error: "Error while Accept() accept tcp [::]:28020: too many open files"

I believe if there is a way to limit as well on the sending side then this would not be an issue.

Thanks

Mark

opened by pixie79 20
elasticsearch can't parse URI

hi there,

Based on your doc via https://github.com/buger/goreplay/wiki/Exporting-to-ElasticSearch

The es7 and gor 1.2 have been installed in my centos7 OS and, I ran the command as below: (Is there any version limitation of elasticsearch?)

/usr/local/bin/gor --input-raw :8080 --output-http http://localhost:8081 --output-http-elasticsearch http://localhost:9200/test123

The test123 index has not been created. Also, the pattern of es URI must be scheme://host/index_name Otherwise, the exception will be thrown out: 2020/11/13 15:49:38 Can't initialize ElasticSearch plugin.Wrong ElasticSearch URL format. Expected to be: scheme://host/index_name

It is appreciated that you could guide me on how to resolve this issue.

And, another issue is about kafka with version 1.2, which https://github.com/buger/goreplay/issues/848 has been submitted. I am concerning whether this bug related with kafka will be fixed in next version (v1.3?) and when will the next version be released?

Sorry for your inconvenience. BR Kimi
needs info

opened by quzhixue-Kimi 18
Replaying requests for large files
I wanted to use gor for testing a HTTP server while mirroring real traffic. Unfortunately the traffic was for relatively big media files (from 100MB up to few GBs). In that case the buffer for reading the response body was too small and gor replayer was failing to replicate the load which real users are generating on the mirrored server.

An other limitation was the fixed deadline for reading the response body. With very large files a timeout of 5s, 30s or whatever is not practical as very big files can take some time to be downloaded. On the other hand a relatively small timeout is required to stop stalled connections which do not move any data around anymore. Maybe because of a faulty HTTP server under test, network problems or something else.

This pull request aims to remedy both of the problems. It does it by

Reading the "whole" body of the response. Or at least trying to do so. The reading is done in small chunks which are discarded instantly in order not to consume too much memory for large responses. An upper limit of the read data is still present to make sure a faulty server will not create an never ending chunked response or some other problem of the kind.

Instead of a fixed deadline, a rolling timeout is introduced while reading the response. The connection will timeout only if there is no network activity at all for the set amount of time.

The behaviour regarding small files is not changed in any way.

Fixes few crashes which were happening from time to time during prolonged tests.
opened by ironsmile 18

Can't rewrite Host header

This is on 0.7.5

~#  ./gor --input-raw :80  --output-http "http://my-staging-host"  --output-http-header "Host: my-staging-host"

But.. sniffing the traffic

~# tcpdump -c 20 -s 1023 -A host my-staging-host and tcp port http
...
Host: news-business.vlex.com
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
...
Host: borme.vlex.es
Connection: Keep-alive
Accept: */*

so it seems that Hostname is not being rewritten

opened by angelf 18

Bump actions/cache from 3.0.5 to 3.2.2
Bumps actions/cache from 3.0.5 to 3.2.2.

Release notes

Sourced from actions/cache's releases.

v3.2.2

What's Changed

Fix formatting error in restore/README.md by @me-and in actions/cache#1044

save/README.md: Fix typo in example by @mmuetzel in actions/cache#1040

README.md: remove outdated Windows cache tip link by @me-and in actions/cache#1042

Revert compression changes related to windows but keep version logging by @Phantsure in actions/cache#1049

New Contributors

@me-and made their first contribution in actions/cache#1044

@mmuetzel made their first contribution in actions/cache#1040

Full Changelog: https://github.com/actions/cache/compare/v3.2.1...v3.2.2

v3.2.1

What's Changed

Release compression related changes for windows by @Phantsure in actions/cache#1039

Upgrade codeql to v2 by @Phantsure in actions/cache#1023

Full Changelog: https://github.com/actions/cache/compare/v3.2.0...v3.2.1

v3.2.0

What's Changed

fix wrong timeout env var key in README.md by @walterddr in actions/cache#959

Updated release doc with correct env variable by @kotewar in actions/cache#960

Create pull_request_template.md by @pdotl in actions/cache#963

Update README with clearer info about cache-hit and its value by @kotewar in actions/cache#961

Change datadog/squid to Ubuntu/squid in CI check by @bishal-pdMSFT in actions/cache#976

Add more details to version section in readme by @bishal-pdMSFT in actions/cache#971

Update hashFiles documentation reference by @asaf400 in actions/cache#979

Updated link for cache segment download info by @kotewar in actions/cache#986

Readme update for deleting caches by @t-dedah in actions/cache#981

Add oncall logic to assign issues and PRs by @vsvipul in actions/cache#997

Bump minimatch from 3.0.4 to 3.1.2 by @dependabot in actions/cache#998

Revert "Bump minimatch from 3.0.4 to 3.1.2" by @vsvipul in actions/cache#1005

Fix npm vulnerability by @Phantsure in actions/cache#1007

refactor: Use early return pattern to avoid nested conditions by @jongwooo in actions/cache#1013

Use cache in check-dist.yml by @jongwooo in actions/cache#1004

chore: Use built-in cache action to cache dependencies by @jongwooo in actions/cache#1014

Updated node example by @t-dedah in actions/cache#1008

Fix: Node npm doc example by @apascualm in actions/cache#1026

docs: fix an invalid link in workarounds.md by @teatimeguest in actions/cache#929

General Availability release for granular cache by @kotewar in actions/cache#1035 More details here on beta release.

New Contributors

@walterddr made their first contribution in actions/cache#959

@asaf400 made their first contribution in actions/cache#979

@jongwooo made their first contribution in actions/cache#1013

@apascualm made their first contribution in actions/cache#1026

@teatimeguest made their first contribution in actions/cache#929

... (truncated)

Changelog

Sourced from actions/cache's changelog.

3.0.5

Removed error handling by consuming actions/cache 3.0 toolkit, Now cache server error handling will be done by toolkit. (PR)

3.0.6

Fixed #809 - zstd -d: no such file or directory error

Fixed #833 - cache doesn't work with github workspace directory

3.0.7

Fixed #810 - download stuck issue. A new timeout is introduced in the download process to abort the download if it gets stuck and doesn't finish within an hour.

3.0.8

Fix zstd not working for windows on gnu tar in issues #888 and #891.

Allowing users to provide a custom timeout as input for aborting download of a cache segment using an environment variable SEGMENT_DOWNLOAD_TIMEOUT_MINS. Default is 60 minutes.

3.0.9

Enhanced the warning message for cache unavailablity in case of GHES.

3.0.10

Fix a bug with sorting inputs.

Update definition for restore-keys in README.md

3.0.11

Update toolkit version to 3.0.5 to include @actions/core@^1.10.0

Update @actions/cache to use updated saveState and setOutput functions from @actions/core@^1.10.0

3.1.0-beta.1

Update @actions/cache on windows to use gnu tar and zstd by default and fallback to bsdtar and zstd if gnu tar is not available. (issue)

3.1.0-beta.2

Added support for fallback to gzip to restore old caches on windows.

3.1.0-beta.3

Bug fixes for bsdtar fallback if gnutar not available and gzip fallback if cache saved using old cache action on windows.

3.2.0-beta.1

Added two new actions - restore and save for granular control on cache.

3.2.0

Released the two new actions - restore and save for granular control on cache

3.2.1

Update @actions/cache on windows to use gnu tar and zstd by default and fallback to bsdtar and zstd if gnu tar is not available. (issue)

Added support for fallback to gzip to restore old caches on windows.

Added logs for cache version in case of a cache miss.

3.2.2

Reverted the changes made in 3.2.1 to use gnu tar and zstd by default on windows.

Commits

4723a57 Revert compression changes related to windows but keep version logging (#1049)

d1507cc Merge pull request #1042 from me-and/correct-readme-re-windows

3337563 Merge branch 'main' into correct-readme-re-windows

60c7666 save/README.md: Fix typo in example (#1040)

b053f2b Fix formatting error in restore/README.md (#1044)

501277c README.md: remove outdated Windows cache tip link

c1a5de8 Upgrade codeql to v2 (#1023)

9b0be58 Release compression related changes for windows (#1039)

c17f4bf GA for granular cache (#1035)

ac25611 docs: fix an invalid link in workarounds.md (#929)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 2

Go module should be github.com/buger/goreplay/v2 for Golang servers to pick it up

For an example, please see here.

The version 1.3.3 is also not picked up.

I use this script to download all package versions that Golang server sees:

#!/bin/sh

SINCE=$1 # like 2022-12-01T00:00:00Z

while [ -n "${SINCE}" ]; do

        # download
        echo "downloading SINCE=${SINCE}"
        wget "https://index.golang.org/index?limit=2000&since=${SINCE}" -q -O output-${SINCE}.json

        # get last since
        SINCE_NEW=`tail -1 output-${SINCE}.json | sed -e 's|.*"Timestamp":"||; s|".*||'`

        # EOF?
        if [ "${SINCE_NEW}" = "${SINCE}" ] || [ -z "${SINCE_NEW}" ]; then
                echo "=== End of project list reached @ ${SINCE} ==="
                exit 0
        else
                SINCE=${SINCE_NEW}
        fi
done

opened by yurivict 0

Why is the gor file output after my local compilation 50MB, while the officially released file is only 10MB?

My environment: OS: CentOS Linux release 7.9.2009 (Core) libpcap: libpcap-1.7.4 goreplay: goreplay-1.3.3

Could anyone be so kind as to help me on the following questions? Thanks a lot first! What're the reasons? What should I do to output my gor with the very size of around 10 MB?

opened by fengnex 1
Use the token bucket algorithm to optimize Limiter

The current limiting algorithm is a fixed window, which may lead to high burst traffic at the millisecond level, such as the following usage: gor --input-file "requests.gor|10000%" --output-http="http://test.com|500 Usage Scenario: QPS Stress Tests

The token bucket algorithm can avoid this situation and make the output traffic smoother

opened by code4wt 1
Not 100% requests are written to file

Only about 1% of requests are recorded under heavy traffic, Is there any configuration to record all ./gor --input-raw :8085 --input-raw-buffer-size 10MB -input-raw-expire 30s --http-allow-url /updateinfo -output-file ./updateinfo_prod_20221121_15 --output-file-append

opened by hanyongqi1 0

Releases(v2.0.0-rc2)

v2.0.0-rc2(Feb 7, 2022)

TBD
Source code(tar.gz)
Source code(zip)
v2.0.0-rc1(Feb 7, 2022)

TBD
Source code(tar.gz)
Source code(zip)
1.3.3(Oct 6, 2021)
What's Changed

Various improvements for packet capture, especially for POST requests https://github.com/buger/goreplay/pull/1007

Fixed replay of pcap files https://github.com/buger/goreplay/pull/1007

Fixed replay of files with malformed records https://github.com/buger/goreplay/pull/1015

Full Changelog: https://github.com/buger/goreplay/compare/v1.3.2...1.3.3
Source code(tar.gz)
Source code(zip)
gor-1.3.3_windows.zip(7.57 MB)
gor-1.3.3.pkg(7.92 MB)
gor_1.3.3_mac.tar.gz(7.92 MB)
gor-1.3.3-1.x86_64.rpm(10.51 MB)
gor_1.3.3_amd64.deb(10.55 MB)
gor_1.3.3_x64.tar.gz(10.55 MB)
v1.3.2(Aug 19, 2021)
Fix promiscuous mode - if enabled no need to filter on host as we want to capture traffic not related to this host #994

Fix --input-raw-realip-header https://github.com/buger/goreplay/pull/996

Fixed false positive network interface lookups https://github.com/buger/goreplay/pull/1000

Thanks @DimaGolomozy!
Source code(tar.gz)
Source code(zip)
gor-1.3.2_windows.zip(7.57 MB)
gor-1.3.2.pkg(10.93 MB)
gor_1.3.2_mac.tar.gz(10.93 MB)
gor-1.3.2-1.x86_64.rpm(10.51 MB)
gor_1.3.2_amd64.deb(10.55 MB)
gor_1.3.2_x64.tar.gz(10.55 MB)
v1.3.1(Aug 12, 2021)
Fix duplicated requests on Linux #989

Improve packet capture, to make packet processing multithreaded 11d61dcb4d518f7ba33ae6f3817a279cee36398d

Source code(tar.gz)
Source code(zip)
gor-1.3.1_windows.zip(7.57 MB)
gor-1.3.1.pkg(10.93 MB)
gor_1.3.1_mac.tar.gz(10.93 MB)
gor-1.3.1-1.x86_64.rpm(10.51 MB)
gor_1.3.1_amd64.deb(10.55 MB)
gor_1.3.1_x64.tar.gz(10.55 MB)
v1.3.0(Aug 6, 2021)
Highlights

Added official Windows support https://github.com/buger/goreplay/wiki/Running-on-Windows

Traffic capture engine fully re-constructed to be more scalable, efficient and bulletproof. You should see way lower packet loss, capture quality, and CPU/Memory usage.

Deprecated own HTTP client in favour of native Go HTTP library https://github.com/buger/goreplay/pull/833

Increased stability, and numerous bug fixes.

Features

Added way to capture traffic from multipe ports with single input --input-raw :80,8080,3000 https://github.com/buger/goreplay/pull/935

Added --output-tcp-skip-verify flag to skip TCP output TLS verification https://github.com/buger/goreplay/pull/814 @tomerf

Added --output-tcp-workers to configure number of TCP output connections https://github.com/buger/goreplay/pull/819

Added new debugging endpoint when --http-pprof enabled: /debug/vars

Added new %i file name variable to inject unique instance ID to file name. Can be useful if you have logs from multiple machines and you need unique names.

Added --input-file-dry-run option to previewing how long it will take and how many requests in given file input https://github.com/buger/goreplay/commit/8e76559b492402ef1d2aee99e0106d085db7f935

Added --input-raw-max-wait option (in seconds), which allows to skip long pauses in recorded files https://github.com/buger/goreplay/commit/19ad90aa4bb0320c8ba623d5509a0690e8c3ec25

Added --input-file-read-depth option to pre-read and buffer requests (and sort them). By default, is 100 https://github.com/buger/goreplay/commit/625ed54f1ebcc006e7d04a27544807fbaff08508

Added --input-raw-timestamp-type go to enable application level timestamps, when network timestamps unreliable https://github.com/buger/goreplay/commit/8edb74e57276c712444ccfb89ed40dd0ca5c39d6

Thank you, @urbanishimwe, @slimus, @arijitAD for jumping with me to this adventure!

Huge kudos to all GoReplay community, especially to our contributors @DimaGolomozy @tomerf @swills @davidFR @two @jl2005 @rmasclef @othererik @ankitdobhal @eko @code4wt @wangfeng22 @lins05 @betty1126 @YuriYin @StanleyP
Source code(tar.gz)
Source code(zip)
gor-1.3.0_windows.zip(7.57 MB)
gor-1.3.0.pkg(10.93 MB)
gor_1.3.0_mac.tar.gz(10.93 MB)
gor-1.3.0-1.x86_64.rpm(10.51 MB)
gor_1.3.0_amd64.deb(10.55 MB)
gor_1.3.0_x64.tar.gz(10.55 MB)
v1.3.0_RC11(Aug 3, 2021)
Fix loopback interface issue on windows https://github.com/buger/goreplay/commit/c9274ac92a6f021240d82682002240cfceaecd5e

Fix buffer index number for sticky tcp output connection https://github.com/buger/goreplay/pull/981 @DimaGolomozy

Fix output file size limits https://github.com/buger/goreplay/pull/974 @lins05

Fixed usage of --input-raw-bpf-filter https://github.com/buger/goreplay/commit/2b993eda49b2a9e9d54ae6cbd616dbb6f6f28388

Fix BPF filter when listen on all interfaces https://github.com/buger/goreplay/commit/a727ade00608b526f0388e9d60c21705136be75c

Optimize packet capture thread to reduce context switching https://github.com/buger/goreplay/commit/214edb45f63ac38dc63c9f21017ddfab557f9d06

Added new debugging endpoint when --http-pprof enabled: /debug/vars

Added --input-file-dry-run option to previewing how long it will take and how many requests in given file input https://github.com/buger/goreplay/commit/8e76559b492402ef1d2aee99e0106d085db7f935

Added --input-raw-max-wait option (in seconds), which allows to skip long pauses in recorded files https://github.com/buger/goreplay/commit/19ad90aa4bb0320c8ba623d5509a0690e8c3ec25

Added --input-file-read-depth option to pre-read and buffer requests (and sort them). By default, is 100 https://github.com/buger/goreplay/commit/625ed54f1ebcc006e7d04a27544807fbaff08508

Added --input-raw-timestamp-type go to enable application level timestamps, when network timestamps unreliable https://github.com/buger/goreplay/commit/8edb74e57276c712444ccfb89ed40dd0ca5c39d6

Source code(tar.gz)
Source code(zip)
gor-1.3.0_RC11_windows.zip(7.57 MB)
gor-1.3.0_RC11.pkg(10.93 MB)
gor_1.3.0_RC11_mac.tar.gz(10.93 MB)
gor-1.3.0_RC11-1.x86_64.rpm(10.51 MB)
gor_1.3.0_RC11_amd64.deb(10.55 MB)
gor_1.3.0_RC11_x64.tar.gz(10.55 MB)
v1.3.0_RC6(Jun 30, 2021)
Significantly improved CPU and Memory usage.

Added af_packet packet capture engine for Linux https://github.com/buger/goreplay/pull/950

Skip incomplete or malformed messages from output (can be enabled back by setting --input-raw-allow-incomplete) https://github.com/buger/goreplay/pull/953

Source code(tar.gz)
Source code(zip)
gor-v1.3.0_RC6_windows.zip(7.55 MB)
gor-v1.3.0_RC6.pkg(10.89 MB)
gor_v1.3.0_RC6_mac.tar.gz(10.89 MB)
gor-v1.3.0_RC6-1.x86_64.rpm(10.48 MB)
gor_v1.3.0_RC6_amd64.deb(10.53 MB)
gor_v1.3.0_RC6_x64.tar.gz(10.53 MB)
v1.3.0_RC5(Jun 10, 2021)
Added a way to capture multiple ports using one input like this: --input-raw :80,8080,3000 #935

Fixed usage of "localhost" hostname when trying to limit scope of packet capture (e.g. --input-raw localhost:80), now it properly binds to loopback interface #943

Fixed detection of windows network interfaces #943

Fixed Makefile job to generate windows binaries

Fixed big memory allocation issue happening during HTTP message end check. Was also causing issues in chunked message detection.

Source code(tar.gz)
Source code(zip)
gor-v1.3.0_RC5_windows.zip(7.53 MB)
gor-v1.3.0_RC5.pkg(10.86 MB)
gor_v1.3.0_RC5_mac.tar.gz(10.86 MB)
gor-v1.3.0_RC5-1.x86_64.rpm(10.40 MB)
gor_v1.3.0_RC5_amd64.deb(10.45 MB)
gor_v1.3.0_RC5_x64.tar.gz(10.44 MB)
v1.3.0_RC4(Jun 9, 2021)
Changes from previous RC

Big changes in capture engine to give more performance and stability #916

Fixed a bug when GoRepay were including outgoing HTTP request initialized within the app, and having the same port #941

Added (again) windows support #940

Fix Output HTTP connection leak #927

Source code(tar.gz)
Source code(zip)
gor-1.3.0_RC4_windows.zip(7.52 MB)
gor-1.3.0_RC4.pkg(10.85 MB)
gor_1.3.0_RC4_mac.tar.gz(10.85 MB)
gor-1.3.0_RC4-1.x86_64.rpm(10.40 MB)
gor_1.3.0_RC4_amd64.deb(10.44 MB)
gor_1.3.0_RC4_x64.tar.gz(10.44 MB)
v1.3.0_RC1(Jan 15, 2021)

Source code(tar.gz)
Source code(zip)
gor_1.3_RC1_mac.tar.gz(10.91 MB)
gor-1.3_RC1-1.x86_64.rpm(10.85 MB)
gor_1.3_RC1_amd64.deb(10.90 MB)
gor_1.3_RC1_x64.tar.gz(10.90 MB)
v1.2.0(Sep 17, 2020)
Engine CPU consumption reduced by 60%

Implemented M'mapped AF_PACKET for Linux. To enable set --input-raw-engine af_packet. #807

Added Kafka TLS configuration flag #800

Fix wrong param value updated while --http-set-param #791

The engine can now capture more requests with large payload #797

Created TCP package #97

Fixed minor bugs in HTTP proto parser

Source code(tar.gz)
Source code(zip)
gor-v1.2.0.pkg(11.29 MB)
gor_v1.2.0_mac.tar.gz(11.29 MB)
gor-v1.2.0-1.x86_64.rpm(10.88 MB)
gor_v1.2.0_amd64.deb(10.92 MB)
gor_v1.2.0_x64.tar.gz(10.92 MB)
v1.1.0(Jul 4, 2020)
Fix usage of multiple --input-raw plugins, e.g. --input-raw :80 --input-raw :81 --input-raw :82 https://github.com/buger/goreplay/pull/700

Allow to set byte size limits in human friedly format: --output-file-size-limit, --output-file-max-size-limit, --copy-buffer-size and --input-raw-buffer-size can now parse inputs from differents bases and data units like: 10mb, 10kb, 100gb, 18t https://github.com/buger/goreplay/pull/754

Fix calculation of the output file size when using gzip compression https://github.com/buger/goreplay/pull/777

Added --output-tcp-sticky so request/response with same ID sent to the same connection.

Better handling of malformed middleware output https://github.com/buger/goreplay/pull/737

Fixed race when reading or writing using files https://github.com/buger/goreplay/pull/756 https://github.com/buger/goreplay/pull/764

Improved errors verbosity https://github.com/buger/goreplay/issues/703

Updated dependencies, including libpcap which should bring more stability and performance

Move to Go modules

PRO code moved to the main repository under separate license

Source code(tar.gz)
Source code(zip)
goreplay-1.1.0.pkg(11.27 MB)
gor_1.1.0_mac.tar.gz(11.27 MB)
goreplay-1.1.0-1.x86_64.rpm(10.81 MB)
goreplay_1.1.0_amd64.deb(10.86 MB)
gor_1.1.0_x64.tar.gz(10.86 MB)
v1.0.0(Mar 30, 2019)
Long awaited major release of GoReplay!

Added option to specify custom BPF filter (e.g. tcpdump syntax). Can be useful in case of non standard network interfaces like tunneling or SPAN port. Example: --input-raw-bpf-filter 'dst port 80’ https://github.com/buger/goreplay/pull/478

Support for reading directly from pcap file, using --input-raw-engine pcap_file --input-raw ./recording.pcap_

Added official Docker image docker pull buger/goreplay. Should be run with --network host argument.

Added RPM and DEB packages

Added support for HTTP proxies. Just set HTTP_PROXY env variable.

Added HTTP basic auth filter: --http-basic-auth-filter "^customer[0-9].*. You specify regexp which match userid:passwd string. [https://github.com/buger/goreplay/pull/475]

Added option to limit size of output file: when limit is reached, it will exit. Can be used for safety reasons. Example: --output-file-max-size-limit 5gb.

Added way to configure output HTTO request queue, which used to hold requests, if all workers are busy. Example: --output-http-queue-len 5000. Default is 1000.

Control frequency of --output-http-stats reports using --output-http-stats-msoption. Example: --output-http-stats-ms 5000 (every 5 seconds).

Configurable way to set minimum number of HTTP workers, by setting --output-http-workers-min. Can be used in conjunction with --output-http-workers which will act as a max worker count.

Added way to dynamically profile GoReplay performance using Golang pprof tools. Example: --http-pprof :8181. It starts web server on given address, and expose special /debug/pprof endpoint with list of reports.

Added —input-raw-buffer-size which controls size of the OS buffer (in bytes) which holds packets until they dispatched. Default value depends by system: in Linux around 2MB. If you see big package drop, increase this value.

In addition to buffer size, added option to turn pcap immediate mode when packets delivered without buffering. Can help reduce packet drop. Example: --input-raw-immediate-mode.

Added way to use standard Golang HTTP client, by adding --output-http-compatibility-mode

Snaplen (max number of bytes being read for each packet) now dynamically set based on interface MTU + max header size. In most situations it should reduce package drop, because each packet will consume less space in the buffer. However in some virtualized environments like OpenShift, packet size can be significantly bigger then MTU, so you can disable optimization by setting --input-raw-override-snaplen.

Added support for ElasticSearch basic HTTP auth

Numerous fixes to improve quality of HTTP packets parsing

Source code(tar.gz)
Source code(zip)
gor_1.0.0_mac.tar.gz(7.08 MB)
gor_1.0.0_x64.tar.gz(5.45 MB)
gor-1.0.0-1.x86_64.rpm(7.05 MB)
gor_1.0.0_amd64.deb(7.08 MB)
v1.0-rc2(Feb 23, 2019)
Added support for HTTP proxies. Just set HTTP_PROXY env variable.

Source code(tar.gz)
Source code(zip)
gor_1.0-RC2_x64.tar.gz(5.45 MB)
gor_1.0-RC2_mac.tar.gz(7.08 MB)
v0.16.1(Jun 30, 2017)
Add support for TLS connections between --input-tcp and --output-tcp. --input-tcp-secure --input-tcp-certificate ./cert.pem --input-tcp-certificate-key ./key.pem --output-tcp-secure #457

Add basic support for old server, like HTTP 0.9 #468 #463 #467

GoReplay now exit once finished replaying with --input-file #456

Fix prettifier issue, when there is problems with gzip encoding payload #470

Source code(tar.gz)
Source code(zip)
gor_0.16.1_x64.tar.gz(4.61 MB)
gor_0.16.1_mac.tar.gz(4.34 MB)
v0.16.0.2(May 1, 2017)
Finally a new big release, a lot of changes on all fronts. New features, usability stability, fixes.

Thank you, everyone, who made it happen!

It would not be possible without this people @smostovoy @exklamationmark @manjeshnilange @kudos @oivoodoo @sattvik @ylegat @nrwiersma @SophisticaSean

CHANGELOG

Major

[PRO] Added S3 input and output https://github.com/buger/goreplay/wiki/%5BPRO%5D-Using-S3-for-storing-and-replaying-traffic

Added new NodeJS middleware framework https://github.com/buger/goreplay/tree/master/middleware

Add Kafka input and output https://github.com/buger/goreplay/wiki/Streaming-from-and-to-Apache-Kafka

Improve accuracy of replay by using timestamp of when TCP packet was received by network interface (previously used time of package capture by GoReplay)

Using —prettify-http option, you can automatically decode Gzip encoded responses, and de-construct chunked bodies.

Now you can enable HTTP response tracking by providing —output-http-track-respose. Previously was available only if middleware is turned on.

Minor

Add new option --input-raw-expire to configure TCP message expiration

Filter response if its request was filtered

Allow space inside filters syntax: `--http-disallow-header 'Host: www.vertaa.fi' now valid syntax

File name pattern now support request id %r, will log each request to separate file, and %t for payload type (0 - request, 1 - response, 2 - replayed response).

Add basic SNI support to support HTTP replay to hosts that require SNI, such as Amazon API Gateway. API Gateway)

Fix --output-file-size-limit option

Do not add port to Host header #383

Improve malformed TCP packet handling to avoid panics

Fix HTTP timeout for 204 (No content) responses

Fix handling of HTTP error codes like 400 or 304.

Fixed replay of HTTP PATCH requests

Source code(tar.gz)
Source code(zip)
gor_0.16.0_mac.tar.gz(4.34 MB)
gor_0.16.0_x64.tar.gz(4.60 MB)
v0.15.1(Aug 31, 2016)
Add support for basic http auth for --output-http 1674b58

Fix handling of "Connection: close" for POST requests fc0361d

Fix inconsistency in replayed response meta when used in middleware a2f5e43

Source code(tar.gz)
Source code(zip)
gor_v0.15.1_mac.tar.gz(3.80 MB)
gor_v0.15.1_x64.tar.gz(4.19 MB)
v0.15.1-beta1(Aug 23, 2016)
Gopacket package now vendored same as the rest of 3-rd party dependencies

Source code(tar.gz)
Source code(zip)
v0.15.0(Aug 10, 2016)
[PRO] Added support for working with binary protocols (thrift/protocol buffers)

[PRO] Recording and replaying keep alive TCP sessions

Replaying request in proper order when using multiple files #300

Validate bodies when Content-Length or Transfer-Encoding found #317

Return of ElasticSearch support! #331 #333

Added --exit-after option to specify duration after which Gor will exit #336

Updated response output file format, now third value is timestamp and forth is latency

Properly read body when no Content-Length but there is Connection: close d34c27c

Remove --input-http option (too confusing and never really worked) f0acd31

Force Go DNS resolver (C one caused crashes for multiple people) 07fa6d9

Fix interception on loopback interface when non-local IP used 1ed8691

Fix intercepting traffic from virtual interfaces 51860e1

Fix 100-continue header when it places not in the end #314

Fix relative file names when using --output-file #304

Properly cleanup used resources on exit #305

Fix connection timeout when using --output-http-timeout c4271ff

Source code(tar.gz)
Source code(zip)
gor_v0.15.0_x64.tar.gz(4.16 MB)
gor_v0.15.0_mac.tar.gz(3.76 MB)
gor.exe(7.50 MB)
v0.14.1(Jun 10, 2016)
Fixed critical issues with crashing when used --output-file

Source code(tar.gz)
Source code(zip)
gor_v0.14.1_x64.tar.gz(4.03 MB)
gor_v0.14.1_mac.tar.gz(3.61 MB)
gor.exe(7.13 MB)
v0.14.0(Jun 9, 2016)
Windows support!

Allow date variables in --output-file names: --output-file %Y-%m-%d-%H.log will create new file each hour #290

Support for gzip compressed files, both for input and output #290

Replaying from multiple files: --input-file now support file name patterns #290

Looping files for replaying indefinitely #290

New default --output-file behaviour for writing files in multiple chunks #293

Faster buffered file output #290

Support for lower-case HTTP headers

Added built-in file server for the tutorial: gor file-server :8000

Fix bug when gor catch its own --output-http traffic if it replayed to the same port as origin #295

--input-raw-realip-header option to injecting header with user real IP #296

Source code(tar.gz)
Source code(zip)
gor.exe(7.13 MB)
gor_v0.14.0_mac.tar.gz(3.61 MB)
gor_v0.14.0_x64.tar.gz(4.03 MB)
v0.13.0(May 25, 2016)
--input-raw now do not track responses by default, you can enable them using --input-raw-track-response #279

--output-dummy now renamed to --output-stdout and does not require arguments #282

HTTP client response buffer now configurable via --output-http-response-buffer #249

Fixed recovery when --output-tcp endpoint goes down https://github.com/buger/gor/commit/0ccace8cad988dfb15ceb725ff3085b5f7c386fd

Improved HTTP capture and replaying for requests with multi-packet headers and POST requests #277 #281

Fixed libpcap crashing when listening on multiple interfaces https://github.com/buger/gor/commit/98868e81c500579428049944ce47ee50d0ec107b

Fixed --input-tcp crashing for payloads > 64kb https://github.com/buger/gor/commit/d40d6c6116c93f5f0f8da768296738cf915821ad

Fixed raw socket engine IPv6 support https://github.com/buger/gor/commit/98fe7f19b21eaaffcd0d733a57668f63de5921d7

Source code(tar.gz)
Source code(zip)
gor_0.13.0_mac.tar.gz(3.53 MB)
gor_0.13.0_x64.tar.gz(3.95 MB)
v0.12.1(May 13, 2016)
Full Mac OS support including pre-compiled binaries (go for mac does not support static linking, you need to have libpcap installed)

Update to use latest libpcap: was 0.8 now 1.7.4

Ensure to clean up sockets after exit

Fixed crashing on system with complex networking

Fixed few data-races

Source code(tar.gz)
Source code(zip)
gor_0.12.1_mac.tar.gz(3.53 MB)
gor_0.12.1_x64.tar.gz(3.95 MB)
v0.12.0(May 11, 2016)
Should be faster, consume less memory, and catch more traffic

Support for IPv6

Fixed traffic interception without specifying port (e.g. on all interfaces: ":80")

Fixed memory leaks

Better handling of fragmented messages

Source code(tar.gz)
Source code(zip)
gor_v0.12.0_x64.tar.gz(3.77 MB)
v0.11.2(May 3, 2016)
Bugfix release

libpcap now linked statically and does not require host system to have it

Fixed panic when gor meet malformed TCPpacket

Source code(tar.gz)
Source code(zip)
gor_0.11.2_x64.tar.gz(3.77 MB)
v0.11.1(Apr 29, 2016)
Fixed critical bug preventing using --input-raw

https://gortool.com - test your system with real data
Source code(tar.gz)
Source code(zip)
gor_0.11.1_x64.tar.gz(2.93 MB)
v0.11.0(Apr 29, 2016)
New site and announce of Pro version https://gortool.com

Release details:

License switched to LGPL, and added option for commercial friendly option

Default traffic interception engine now libpcap, which significantly increased capture quality and speed

Multiple fixes improving stability, speed and quality

Go 1.6.1

Source code(tar.gz)
Source code(zip)
gor_0.11.1_x64.tar.gz(2.93 MB)
v0.10.1(Sep 3, 2015)
Fixed memory leaks #194 #201

Improved speed by dramatically reducing number of goroutines #194

Fixed input-file #202

Source code(tar.gz)
Source code(zip)
gor_0.10.1_x86.tar.gz(1.93 MB)
gor_0.10.1_x64.tar.gz(2.06 MB)
0.10.0(Aug 24, 2015)
Middleware support! See README and examples folder. #162

Raw input now track responses (they are recorded to file input as well) #162

Changed file format, now it human readable #162

Changed internal TCP communication format (same as file format) #162

Add protection for malformed messages

Significantly improved test speed #162

Fix: do not add 80 port to HOST header #190

Source code(tar.gz)
Source code(zip)
gor_0.10.0_x86.tar.gz(1.92 MB)
gor_0.10.0_x64.tar.gz(2.05 MB)