Randomized Testing (Core JUnit Runner, ANT, Maven)

Related tags

Testing randomizedtesting
Overview
Comments
  • Question: Enhance ThreadLeakControl to print additional metadata

    Question: Enhance ThreadLeakControl to print additional metadata

    In Solr we have MDCAwareThreadPool that tracks a submitterStackTrace for the threads that it manages. This information would be really useful to print on leaked threads when it is available, since it would help pinpoint where this thread came from that was leaked (useful when it looks like a generic http client thread, for example).

    I don't think it makes sense for the runner to know about Solr's convention, but maybe we can make some kind of extensible API cut point that would allow us to customize how we gather stack trace information?

    opened by madrob 11
  • Convenience System property setting on SystemPropertiesRestoreRule?

    Convenience System property setting on SystemPropertiesRestoreRule?

    SystemPropertiesRestoreRule is pretty nice. It'd be even nicer if I could conveniently set particular system properties at the site of the rule definition, to make it clear why the rule is there in the first place. This would work by adding a convenient (String key, String value) constructor, and another (Map<String, String> properties) when there is more than one. My proposal here is inspired by https://gist.github.com/mike10004/7f6865b46c23005e9496a53589194788 which I found with some googling.

    Of course this isn't necessary. I can define this rule, and then somewhere in some @Before or wherever, I can call System.setProperty(name, value. But my proposal keeps these linked together. One might want to define System.setProperty as a "forbidden API", in tests, and instead rely purely on your JUnit Rule to manipulate them, which communicates to readers that it's important to clean up after test execution.

    opened by dsmiley 1
  • Provide means to use RandomizedTesting without the test runner.

    Provide means to use RandomizedTesting without the test runner.

    I work on a project that depends on lucene-test-framework for its wonderful test utilities to validate use of text analysis components. The project does not use randomizedtesting's test runners. I want to use some test utilities that I cannot invoke now because they directly/indirectly invoke randomizedtesting which fails because there is no context:

    java.lang.IllegalStateException: No context information for thread: Thread[id=1, name=main, state=RUNNABLE, group=main]. Is this thread running under a class com.carrotsearch.randomizedtesting.RandomizedRunner runner context? Add @RunWith(class com.carrotsearch.randomizedtesting.RandomizedRunner.class) to your test class. Make sure your code accesses random contexts within @BeforeClass and @AfterClass boundary (for example, static test class initializers are not permitted to access random contexts).

    Asking users/projects to a specific test runner can be an awkward constraint. For some of my tests, I can use your runner, for some, I cannot because another is used.

    When I look at RandomizedContext, there isn't sufficient access for me to manually manage it's lifecycle outside of your runner. I would like this added and/or a JUnit Rule as an option. Such a Rule wouldn't bring all the baggage that RandomizedRunner has -- no thread leak detection. Just provide what this project's namesake is -- randomness. That's all.

    opened by dsmiley 3
  • Move lucene's mockfs into junit or randomizedtesting?

    Move lucene's mockfs into junit or randomizedtesting?

    Lucene mockfs (https://github.com/apache/lucene-solr/tree/master/lucene/test-framework/src/java/org/apache/lucene/mockfile) basically tries to wrap the default nio.2 filesystem with a provider that changes things up:

    • ShuffleFS: Directory listings in an unpredictable but deterministic order regardless of underlying filesystem order (e.g. OS X returns in sorted order, Linux does not)
    • LeakFS: Fails tests if they leave open filehandles. (Only files with nio.2 api, not sockets, but useful).
    • DisableFsyncFS: Makes actual fsync() calls a no-op. If you are e.g. a database or similar and using the filesystem seriously, this speeds up tests with no downside...
    • VerboseFS: Prints destructive filesystem operations to infostream (Logging writes/deletes/renames/etc). For debugging.
    • ExtrasFS: Adds 'bonus' files to directories. Kind of annoying and lucene-specific. But you could tame it by only adding 'realistic bonus files' like '.DS_Store'.
    • WindowsFS: Acts like windows (imperfect, just prevents deleting open files at the moment)

    The idea of the thing is that you can build a chain, of several of these, wrapping the default filesystem. None of the stuff is really dependent on lucene though, and some of these filesystem issues are real traps (e.g. system-specific stuff like order of results from listing directories)?

    However, there are some problems/downsides/questions as it stands out:

    • We wrap at test runner level (we pass to getTempDir() and all lucene tests use that). Ideally it would be at a higher level and pass system properties to the jvm to wrap the default filesystem so that it can't be backdoored by calls to e.g. FileSystem.getDefault()/Paths.get()/.... I have hacked it to work this way for debugging production code (VerboseFS) with https://github.com/rmuir/verbosefs .
    • NIO.2 api is incompletely wrapped for some stuff. We don't yet wrap the FileLock API (because lucene has a bad history with fs locking code, but we should do it now), some implementations are incomplete because Lucene only uses a subset of the filesystem operations.
    • There are unit tests, but they could be improved...
    • There are some dragons in the current code, while it works for lucene's case, if you add to an app more extensively using the java NIO apis, you will find: https://github.com/rmuir/verbosefs/blob/master/src/rmuir/FilterPath.java#L57 These are basically VFS layer violation bugs with things being wrapped twice or something like that.
    • Is NIO.2 really being adopted? Not many projects are using it yet. But if you fully cut over, you can take advantage of stuff like this.

    Anyway, I'd be interested in helping out if its a better home or fixes to the issues or whatever. Or may be we could only take some of ideas and try to rewrite it to work better, etc.

    opened by rmuir 1
  • Implement true heartbeat from the forked JVM

    Implement true heartbeat from the forked JVM

    A "true" heartbeat would provide several answers to problems we're currently facing:

    • it's not known where the "deadlocked" JVM actually is (stack traces),
    • it's not known whether it's a resource (no cpu usage) or active (spinloop) deadlock, sending back the stack traces a few time could answer this (as well as maybe generic CPU load),
    • it's not known whether the forked process is still alive at all (should heartbeat messages stop arriving, we can try to terminate the forked process and end with a build error).
    feature refactoring ant-junit4 maven-junit4 backwards_incompatible 
    opened by dweiss 1
Releases(release/2.8.0)

  • release/2.8.0(Jun 28, 2022)

    This release includes two minor changes and should be fully backwards-compatible, unless deep internals of the code were used (such as event subscribers).

    • GH-303: Messages from ThreadLeakControl.checkThreadLeaks can be inconsistent due to mismatched use of thread data.

    • GH-304: Replace master/slave with main/forkedjvm terminology. This change may result in backward incompatibilities if you used low-level event bus subscription methods or JSON reports. Grep and replace 'slave' with 'forkedJvm' and 'master' with 'main' seed.

    Source code(tar.gz)
    Source code(zip)

  • release/2.7.9(Oct 22, 2021)

  • release/2.7.8(Jul 7, 2020)

  • release/2.7.7(Mar 2, 2020)

    Bug Fixes

    • GH-283: RandomizedRunner throws NoTestsRemainException from the constructor when method filter is supplied and no methods match (Mark Vieira).
    Source code(tar.gz)
    Source code(zip)

  • release/2.7.5(Mar 2, 2020)

  • release/2.7.6(Mar 2, 2020)

  • release/2.7.4(Oct 23, 2019)

  • release/2.7.3(May 7, 2019)

  • release/2.7.2(Dec 14, 2018)

    Bug Fixes

    • GH-275: Timeouts can deadlock on trying to interrupt leaked threads.

    Improvements

    • GH-274: PrintEventListener should emit display name if method name is not available in testIgnored callback.
    Source code(tar.gz)
    Source code(zip)

  • release/2.7.1(Nov 20, 2018)

  • release/2.7.0(Sep 10, 2018)

    Bug Fixes

    • GH-269: 2.6.x runners and junit 4.12 show much noise for a single test method in IntellJ Idea (thanks Vladimir Dolzhenko).

    Changes in Backwards Compatibility

    • GH-270: Upgrade to support JUnit 4.12. This also means that Hamcrest is required (as JUnit 4.12 uses it).
    Source code(tar.gz)
    Source code(zip)

  • release/2.6.4(Sep 5, 2018)

    Bug Fixes

    • GH-247: Allow java.io.tmpdir to be placed under cwd with isolated directories and nonEmptyWorkDirAction=wipe.

    • GH-255: Invalid event nesting on security logging crashes the runner.

    • GH-266: QUIT event may not be read from the event log on slow machines.

    • GH-267: Project does not build/ test on Java 11.

    Improvements

    • An additional property 'junit4.tempDir' is passed to forked JVMs to allow easier configuration of security policy files. This folder must have read/ write file policy.

    • Added a test case that runs with a security manager enabled.

    • GH-253: IntelliJ only recognizes proper test methods for re-runs when descriptions have " [...]" instead of " {...}. This fix uses square brackets on IntelliJ runner.

    Changes in Backwards Compatibility

    • GH-265: Deprecate RandomizedTest.randomInt(v) and RandomizedTest.randomLong(v).
    Source code(tar.gz)
    Source code(zip)

  • release/2.6.3(Jun 14, 2018)

  • release/2.6.2(Jun 11, 2018)

  • release/2.6.1(Jun 11, 2018)

  • release/2.6.0(Jun 11, 2018)

  • release/2.5.3(Aug 31, 2017)

    JavaDoc | ANT task

    Bug fixes

    • GH-251: Fixes test case filtering in IDEs (IntelliJ, Eclipse). The tests are now filtered out eagerly from the returned suite's Description and don't show in the user interface at all. This also applies to -Dtests.method=... patterns.
    Source code(tar.gz)
    Source code(zip)

  • release/2.5.2(Jul 4, 2017)

  • release/2.5.1(Apr 13, 2017)

    JavaDoc | ANT task

    Changes in Backwards Compatibility

    • GH-248: Cleaning up some confusion on RandomizedTest.randomAscii* and RandomStrings.randomAscii* methods. These methods generated random strings based on letters from the Unicode basic latin block (C0). All these methods are now deprecated and will be removed in favor of more explicit methods that tell exactly (?) what kind of letters are involved (Ryan Ernst, Dawid Weiss). For details, see:
    RandomStrings.randomAsciiLettersOfLength(..)
RandomStrings.randomAsciiLettersOfLengthBetween(..)
RandomStrings.randomAsciiAlphanumOfLength(..)
RandomStrings.randomAsciiAlphanumOfLengthBetween(..)
    Source code(tar.gz)
    Source code(zip)

  • release/2.5.0(Jan 23, 2017)

    JavaDoc | ANT task

    Changes in Backwards Compatibility

    • GH-243: Switches from File to NIO (Path) APIs in a few places may result in slightly different runner's behavior (on exceptional conditions).
    • GH-244: the new onNonEmptyWorkDirectory attribute's default value is 'fail' which may break existing builds that have garbage in work directories of forked JVMs. Change explicitly to any of the alternatives ('ignore' or 'wipe') or fix your build to wipe the temporary folders yourself.
    • GH-246: RequireAssertionsRule enforcement optional (consistent with "-Dtests.asserts"); explicit target class is also required. (Dawid Weiss)

    New Features

    • GH-244: A new attribute "onNonEmptyWorkDirectory" controls the action to take on non-empty initial work directory (for isolated JVMs). (Dawid Weiss)

    Other Changes

    • GH-246: RequireAssertionsRule enforcement optional (consistent with "-Dtests.asserts"); explicit target class is also required. (Dawid Weiss)
    • GH-243: Switches from File to NIO (Path) APIs in a few places in the ANT runner. Improved messages concerning non-empty temporary JVM folder (both initially and at the end of the tests' run). (Dawid Weiss)
    • GH-245: Integrate Forbidden-APIs check and fix follow-up issues. (Dawid Weiss)
    Source code(tar.gz)
    Source code(zip)

  • release/2.4.1(Jan 18, 2017)

  • release/2.4.0(Oct 20, 2016)

    JavaDoc | ANT task

    Changes in Backwards Compatibility

    • GH-237: Methods previously found in RandomInts have been moved to RandomNumbers. (Dawid Weiss)

    • GH-234: Run strict validation (thread-ownership, lifecycle) in AssertingRandom only with assertions enabled. This change means that running without -ea will from now on ignore the fact that random context's Random instance is shared where it shouldn't be. On the other hand, any additional memory barriers can be now avoided, in particular when used in conjunction with @TestContextRandomSupplier (see GH-235 below).

      To enable consistent checking, enable assertions selectively (for com.carrotsearch.randomizedtesting package only). (Dawid Weiss)

    • GH-229: The default implementation of the Random instance returned from the randomization context has been changed to Xoroshiro128P, details here: http://xoroshiro.di.unimi.it/

      The returned Random isn't synchronized and should yield much better pseudo-random sequences than the default Java's Random. (Dawid Weiss)

    New Features

    • GH-238: Add BiasedNumbers (random selection with bias towards "evil" numbers, in particular floats) as an alternative to RandomNumbers. (Dawid Weiss)
    • GH-237: Rename RandomInts to RandomNumbers and add methods operating on longs. (Dawid Weiss).
    • GH-235: A new class annotation was added: @TestContextRandomSupplier allows declaring a custom class (RandomSupplier) that supplies a Random for the given seed. This supplier can return any Random instance for the given initial seed (secure random, faster-than-jdk random, non-random random...). (Dawid Weiss)

    Bug Fixes

    • GH-233: some adjustments and refactorings in an attempt to avoid deadlocks on suite timeouts. Very hard to reproduce (i.e., impossible) and some very odd stack traces too. (Dawid Weiss)

    Other Changes

    • GH-232: Rework reporting of ignored, assumption-ignored, filtering-expression ignored tests. (Dawid Weiss)
    • GH-218: There are some internal changes in handling @Nightly annotation. The support for @Nightly is now provided via GroupEvaluator (and this class is now public, with methods allowing inspection of whether a test would be filtered out or not). (Dawid Weiss)
    Source code(tar.gz)
    Source code(zip)

  • release/2.3.4(Apr 4, 2016)

  • release/2.3.3(Jan 14, 2016)

  • release/2.3.2(Dec 7, 2015)

    JavaDoc | ANT task

    Changes in Backwards Compatibility

    • GH-222: Make unexpected JVM output available to listeners. This issue makes JVM output available to ANT JUnit4's listeners so that they can log it properly. Previously the JVM output was copied to System.out which could be hard to work with. See TextReport.onJvmOutput for an example. The default value of 'jvmOutputAction' has been changed from 'pipe, warn' to 'listeners, warn' so that listeners get notified about JVM output instead of it being copied directly to sysout. (Dawid Weiss, Ryan Ernst).

    Other Changes

    • GH-220: Add an indication of the number of failed suites so far to text report. (Dawid Weiss, Chris Hostetter).
    • GH-224: Remove excess use of Guava's collections. (Dawid Weiss)
    • GH-221: Upgrade to Guava 19.0-rc3. (Dawid Weiss, Ryan Ernst)

    Bug Fixes

    • PR #223: Access private fields within doPrivileged (Robert Muir)
    Source code(tar.gz)
    Source code(zip)

  • release/2.3.1(Nov 9, 2015)

  • release/2.3.0(Nov 6, 2015)

  • release/2.2.0(Oct 28, 2015)

    JavaDoc | ANT task

    Changes in Backwards Compatibility

    • PR #178: Move to Java 1.7 and use the java NIO 2 API for temp files. (Simon Willnauer)
    • GH-184: RandomizedTest should not extend Assert. This permits easier use of other frameworks (like assertj) via static imports without naming conflicts. (Olivier Bourgain, Dawid Weiss)

    New Features

    • GH-191: "Tests with failures" should repeat the master seed at the end of the report. (Dawid Weiss)

    Bug Fixes

    • GH-192: Reworked json streaming to get rid of gson and permit dumping JVM-related diagnostics (security manager policy dumps, etc.). (Dawid Weiss)
    • GH-206: Potential NPE in checking getDefaultUncaughtExceptionHandler. (Dawid Weiss)

    Other Changes

    • GH-210: Make RR compilable/ testable under Java 1.9. (Dawid Weiss)
    • GH-209: Remove dependency on commons-io. (Dawid Weiss)
    • GH-204: modernize and clean up project layout. (Dawid Weiss)
    Source code(tar.gz)
    Source code(zip)

  • release/2.1.17(Sep 22, 2015)

    JavaDoc | ANT task

    Bug Fixes

    • GH-201: Tests still running (and zombified) after suite timeout occurred. (Dawid Weiss)

    Other Changes

    • GH-205: Report test seed in halt on failure message. (Dawid Weiss)
    • GH-203: report the number of completely ignored suites in the summary. (Dawid Weiss)
    • PR #202: support running with security manager and limit the required permissions. (Robert Muir)
    Source code(tar.gz)
    Source code(zip)
Owner
null
GitHub http://labs.carrotsearch.com/randomizedtesting.html
Automation Testing | RESTful webservices | Java | RestAssured | OAuth 2.0 | Lombok | TestNG | Maven | ExtentReport | Allure Reports | Java mail API | Design Patterns (Singleton) | Jenkins | Data-Driven Testing using JSON file | Expected Data using XML file

MasterRestAssuredFramework With further updates - This project is designed and developed with the help of - Udemy course (OmPrakash Chavan - REST Assu

Rajat Verma 27 Nov 13, 2022
A simple yet powerful parameterized test runner for Java.

TestParameterInjector Introduction TestParameterInjector is a JUnit4 test runner that runs its test methods for different combinations of field/parame

Google 324 Dec 30, 2022
A BDD-style test runner for Java 8. Inspired by Jasmine, RSpec, and Cucumber.

Spectrum A colorful BDD-style test runner for Java Spectrum is inspired by the behavior-driven testing frameworks Jasmine and RSpec, bringing their ex

Greg Haskins 143 Nov 22, 2022
Java testing framework for testing pojo methods

Java testing framework for testing pojo methods. It tests equals, hashCode, toString, getters, setters, constructors and whatever you report in issues ;)

Piotr Joński 48 Aug 23, 2022
Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.

Testcontainers Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium we

null 6.7k Jan 9, 2023
CodeSheriff is a simple library that helps you in writing JUnit tests that check the quality of your code

CodeSheriff is a simple library that helps you in writing JUnit tests that check the quality of your code. For example, CodeSheriff may fail because you have methods in your code that have more than X lines of code, or that have complexity greater than Y.

Maurício Aniche 62 Feb 10, 2022
JUnit 5 Parameterized Test Yaml Test Data Source

Yamaledt — JUnit 5 Parameterized Tests Using Yaml and Jamal Introduction and usage Note This is the latest development documentation. This is a SNAPSH

Peter Verhas 4 Mar 23, 2022
Exercicio envolvendo criação de Classe, Junit 5 e exception

ClasseData Complemente a classe Data desenvolvida anteriormente para que lance uma ExcecaoDataInvalida. Com esta única exceção, você deverá ser capaz

Mateus Samartini 3 May 7, 2021
Exercico com Junit 5, UML, Interação de classes e Exception

AcessoClube Exercico com Junit 5, UML, Interação de classes e Exception Exercicio: Projete e desenvolva um sistema de controle de acesso a um clube co

Mateus Samartini 3 May 3, 2021
This is a repository to collect JUnit Tests for GAD exercises at TUM in SS21

TUM_GAD_Tests_SS21 This is a repository to collect JUnit Tests for GAD exercises at TUM in SS21. These tests have been written by students for student

null 41 Oct 31, 2022
A template for Spring Boot REST API tested with JUnit 5 and Cucumber 6

demo-bdd Un template Spring Boot pour lancer un BDD/ATDD avec Cucumber 6 et JUnit 5. Maven et le JDK 17 seront nécessaires. Exécuter les tests Le proj

Rui Lopes 4 Jul 19, 2022
IntelliJ IDEA and JUnit: Writing, Finding, and Running Tests

IntelliJ IDEA and JUnit: Writing, Finding, and Running Tests ?? Webinar https://blog.jetbrains.com/idea/2021/11/live-stream-recording-intellij-idea-an

Christian Stein 11 Jul 23, 2022
Testes unitários em Java utilizando JUnit 5

Testes unitários em Java utilizando JUnit Este curso tem como objetivo habilitar o(a) aluno(a) a testar soluções desenvolvidas na linguagem Java, torn

Camila Cavalcante 3 Jan 18, 2022
Testes unitários em Java utilizando JUnit

Testes unitários em Java utilizando JUnit Este curso tem como objetivo habilitar o(a) aluno(a) a testar soluções desenvolvidas na linguagem Java, torn

Camila Cavalcante 6 Nov 8, 2022
JUnit 5 extension for easy-random

Easy Random/Faker JUnit 5 extension The simple, stupid random Java™ beans generator for JUnit 5 The easy random extension provides a test with randoml

Libing Chen 8 Nov 9, 2022
Intercept network request by running Selenium tests with JUnit on LambdaTest cloud.

Run Selenium 4 Tests With JUnit On LambdaTest Blog ⋅ Docs ⋅ Learning Hub ⋅ Newsletter ⋅ Certifications ⋅ YouTube       Learn how to use JUnit framewor

null 11 Jul 11, 2022
Override device mode by running Selenium test with JUnit on LambdaTest cloud.

Run Selenium 4 Tests With JUnit On LambdaTest Blog ⋅ Docs ⋅ Learning Hub ⋅ Newsletter ⋅ Certifications ⋅ YouTube       Learn how to use JUnit framewor

null 11 Jul 11, 2022
Emulate geolocation by running Selenium tests with JUnit on LambdaTest cloud.

Run Selenium 4 Tests With JUnit On LambdaTest Blog ⋅ Docs ⋅ Learning Hub ⋅ Newsletter ⋅ Certifications ⋅ YouTube       Learn how to use JUnit framewor

null 11 Jul 11, 2022
Testes unitários em Java utilizando JUnit

Testes unitários em Java utilizando JUnit Este curso tem como objetivo habilitar o(a) aluno(a) a testar soluções desenvolvidas na linguagem Java, torn

Camila Cavalcante 11 Oct 13, 2022