State of the art mutation testing system for the JVM

Henry Coles

Last update: Dec 26, 2022

Overview

Pitest (aka PIT) is a state of the art mutation testing system for Java and the JVM.

Read all about it at http://pitest.org

Releases

1.7.3

#952 Mutate map return to emptyMap instead of null
#954 Allow mutators to be excluded
#957 Filter equivalent mutations to Boolean.TRUE and Boolean.FALSE

1.7.2

#943 Change default mutators - replace negate conditional with remove conditional
#946 Mutate stream returns to empty stream instead of null

1.7.1

#932 Improve switch mutation descriptions
#934 Configure console output verbosity
#940 Hide internal features
#942 Fix for 3rd party mutation operators not added to client classpath
#939 Improve caching of bytecode retrieval

1.7.0

#923 Internal interface changes
#930 Pluggable mutators

Due to internal changes some third party plugins maybe incompatible with this release.

All history files should be deleted before upgrading.

The names of the remove conditionals mutators have changed slightly as a result of #930 and may need to be updated in build scripts if explicitly activated.

1.6.9

#922 Filter equivalent stream.empty mutants in flatMap calls
#921 Guarantee order of mutation operators
#919 Filter junk mutations in java records

1.6.8

#917 - Add method to retrieve all mutator ids for pitclipse and other tooling
#913 - Bump ASM to 9.2

1.6.7

#897 Fix description when replacing value with empty set
#900 Support multiple test engines
#822 Fix regression of NPE when reading classpath manifest
#798 Fix regression of jvmArgs parameter support via maven
#797 Fix regression of line coverage on console

As a result of #900 the testPlugin parameter is now defunct, but pitest will continue to accept it without error for this release. Pitest will use any test-engines supplied on the classpath, falling back to the built in JUnit4 support when other test engines cannot handle the class.

1.6.6

#891 - Fix history performance
#889 - Do not scan classpath for history when option not supplied
#887 - Fix css for timeouts and memory errors
#888 - Remove defunct max mutants per class perameter
#890 - Remove defunct mutate static initalizers parameter

1.6.5

#882 Avoid running tests when no mutants possible

1.6.4

#862 Update ASM for Java 16

1.6.3

#853 Fix case insensitive feature check
#855 Make coverage data available to interceptors
#857 Pass stderr/out data by line (thanks @kgeilmann)
#858 Fix test not failing when maxSurviving is 0 (thanks @alexkoltz)
#860 Fix junk mutations in try with resources in java 11+

1.6.2

#770 and #746 Fix NPE during coverage stage (thanks @LaurentTho3)
#849 Make feature names case insensitive
#844 Extend feature system to work with listeners
#842 Make report options available to listeners

1.6.1

Automate release to maven central
#774 Test strength statistic (thanks @alex859)
#798 Enable jvm args to be passed from command line (thanks @yfrolov)
#797 Add line coverage to console (thanks @qxo)
#822 Mitigate NPE on accidental dependency (thanks @szpak)

1.5.2

#749 ANT support for the fullMutationMatrix option (thanks @ayaankazerouni)
#752, #755, #757 Version bumps and legacy code improvements (thanks @AlexElin)
#758 Correctly can for test packages (thanks @nicerloop)
#765 Fix incremental analysis for TestNG (thanks @StefanPenndorf)

1.5.1

#737 Add skipFailingTests flag to command line tool (thanks @szpak)
#739 Fix typo in error message (thanks @vmellgre)
#745 Improve error message when junit not on classpath (thanks @szpak)
#748 Bump asm to 8.0 for Java 14 support

1.5.0

#556 - Do not mutate enum constructors
#726 - Ensure static initializer coverage is recorded
#734/#735/#736 - Replace legacy interfaces with Java API (thanks @AlexElin)

Older versions

1.4.11

#683 - Filter try-with-resources before filtering inlined code (thanks @Vampire)
#686 - Do not print the class name twice for unsplittable test units (thanks @Vampire)
#672 - Do not include the current directory to the minion class path (thanks @Vampire)
#692 - Add property='skipPitest' to skip attribute inn maven plugin (thanks @cjgwhite)
#697 - TestNG 7.0.0 compatibility (thanks @kris-scheibe)
#666 - UOI4 reports mutated field name (thanks @LaurentTho3)
#716 - Bump asm to 7.3.1 (required for Java 14)
#710 - Use the new mutator set by default

1.4.10

#534 Smaller blocks for more precise test targeting (thanks @jon-bell)
#664 Fix A0D2 map key (thanks @Vampire)
#656 Escape characters in init methods for html report (thanks @Vampire)
#404 Filter junk mutations to compiler generated Objects.requireNonNull calls

1.4.9

#613 / #623 - Fix for powermock issues on (thanks @jon-bell)
#614 - Improved error message when no test plugin (thanks @szpak)
#620 - Support annotation processors such as Micronaut that do not set debug filename

1.4.8

#597 Fix for bug in coverage when large number of classes (thanks @jon-bell)
#601 Avoid stealing keyboard focus on macos (thanks @maxgabut)

1.4.7

#545 - Faster coverage calculation (thanks @jon-bell)

1.4.6

#580 - upgrade to ASM 7.1
#573 - Ant support for testPlugin and failWhenNoMutations (thanks @mduggan)
#555 - Allow empty elements in maven configuration (thanks @maxgabut)

1.4.5

#557 - Issues running on Java 8

1.4.4

#518 - Experimental BigInteger mutator (thanks @ripdajacker)
#513 - Sort mutators in html report (thanks @ThLeu)
#553 - Classic mutators from literature (thanls @LaurentTho3)
#528 - Added skipFailingTests option from maven plugin (thanks @nicerloop)

1.4.3

#510 - Compute mutation test matrix (thanks @nrainer)
#519 - Java 11 support

1.4.2

#500 - Support for large classpaths with new useClasspathJar option (thanks @jqhan)

1.4.1

#446 - Ignore empty directories when aggregating report (thanks @maxgabut)
#457 - Fix base dir in multi module projects (thanks @sarahBuisson and @giggluigg)
#471 - Basic Java 11 support
#477 - Null byte in xml (thanks @maxgabut)
#480 - Allow aggregation of timestamped reports (thanks @maxgabut)
#495 - Move summary to end of console output (thanks @pedrorijo91)
#499 - Fix for equivalent mutants in new return val mutators

1.4.0

#445 - Raise minimum supported java version to 8
Switch to ASM 6.1.1
#448 Aggregate report based on module (thanks @sarahBuisson)
#462 Escape test names in coverage xml (thanks @maxgabut)

1.3.2

#433 - Aggregate reports via maven (thanks @rchargel)
#438 - Filter tests at method level (thanks @ftrautsch)
#439 - Primitives return mutator doesn't handle bytes
#380 - Remove reliance on xstream for improved java 9 support
#414 - Replace xstream history store (thanks @kiftio)

Note when upgrading that history files generated by previous releases are not compatible with 1.3.2 an above.

1.3.1

#434 - XStream security warning

1.3.0

#196 - Raise minimum java version to 7
#425 - Broaden for-each detection
#428 - Add excludedTestClasses parameter
#379 - Remove support for mutating static initializers
#432 - Add export plugin functionality into pitest
#427 - Better return values mutators
#429 - Simplify test api

428 changes the existing behaviour of the excludedClasses filter. Previously this excluded classes from both being mutated and run as tests which may require changes for some people when upgrading.

As a result of 429 TestNG tests will no longer be automatically detected - the new testPlugin parameter must be explicitly set to "testng".

1.2.5

#407 - SCM goal cannot be run from within module (thanks @sbuisson)
#256 - Inline style violate Content Security Policy (thanks @kiftio)
#403 - No css in the html report viewed from jenkins (thanks @kiftio)
#409 - Mutate based on changes across branched (thanks @sbuisson)
#418 - Avoid for loop increments
#424 - Avoid for each code
#389 - Widen matching of implicit null checks

1.2.4

#397 - Report reasons maven modules skipped
#389 - Filter junk getClass null checks in lambdas
#400 - Update to ASM 6.0

1.2.3

Update to ASM 6 for Java 9 support
#388 - Mark maven plugin as threadsafe
#362 - Suppress common equivalent mutant in equals methods

1.2.2

#367 - Static analysis cannot find core classes in some projects

1.2.1

#347 - Avoid autogenerated code (e.g by lombok) anotated with @Generated
#285 - Avoid compiler generated conditionals in switch on string. (thanks @Kwaq)
#218 - New "naked receiver" / method chain mutator. (thanks @UrsMetz)
#354/#356 - New extension point plus changes to reduce cost of random access to bytecode for static analysis
#353 - Improved static initializer code detection
#364 - Filter infinite loops via static analysis
#365 - Configuration system for plugins

(Note #347 will not detect javax.annotaion.Generated as it has only source retention.)

1.2.0

#318 - Separate main process code from code sharing client classpath
#295 - Ignore abstract TestNG tests
#215 - Automatic selection of target classes for maven
#314 - Do not fail for maven modules without tests or code
#306 - Do not fail when only interfaces are in changeset
#325 - Anonymous class in Spock test causing error
#334 - Compatibility with recent TestNG releases
#335 - Incorrect coverage with powermock

Note - as of this release the maven plugin will automatically determine which classes to mutate instead of assuming that the package names match the group id.

1.1.11

#269 - Not possible to break build on 1 mutant
#271 - Include method descriptor in line coverage
#170 - Exclusion by JUnit runner
#291 - Handle empty excludes
#287 - Check class hierarchy for annotations
#289 - Option to supply classpath from file

1.1.10

#260 - Initial support for mutating Kotlin code
#264 - Support for yatspec runner (thanks @theangrydev)
Break build when more than maxSurviving mutants survive

1.1.9

#132 - Allow analysis of only files touched in last commit (thanks Tomasz Luch)

1.1.8

#239 - Provide a shortcut to set history files via maven
#240 - Support for regexes (thanks sebi-hgdata)
#243 - Use ephemeral ports to communicate with minions

1.1.7

#196 - Raise minimum java version to 1.6
#231 - Process hangs

1.1.6

#10 - Add maven report goal (thanks jasonmfehr)
#184 - Remove undocumented project file feature
#219 - Performance improvement for report generation (thanks tobiasbaum)
#190 - Allow custom properties for plugins

Note this release contains a known issue (#231). Please upgrade.

1.1.5

Fix for #148 - Stackoverflow with TestNG data providers when using JMockit
Fix for #56 - Not reporting junit incompatibilities
Fix for #174 - Invalid linecoverage.xml with static initializers
Fix for #183 - Can't run GWTMockito tests
Fix for #179 - Broken includeLaunchClasspath=false on Windows
#173 - Read exclusions and groups from maven surefire config

1.1.4

#157 - Support maven -DskipTests flag (thanks lkwg82)
Fix for #163 - Should not include test tree in coverage threshold
#166 - Allow classpath exclusions for maven plugin (thanks TomRK1089)
#155 - Restore Java 5 compatibility
Fix for #148 - Issue with JMockit + TestNG (thanks estekhin and KyleRogers)

1.1.3

Fix for #158 - Tests incorrectly excluded from mutants
Fix for #153 - SCM plugin broken for maven 2
Fix for #152 - Does not work with IBM jdk

1.1.2

Fix for #150 - line coverage under reported

1.1.1

Block based coverage (fixes 79/131)
End support for running on Java 5 (java 5 bytecode still supported)
Skip flag for maven modules (#106)
Stop declaring TestNG as a dependency
New parameter propagation mutator (thanks UrsMetz)

1.1.0

Change scheme for identifying mutants (see https://github.com/hcoles/pitest/issues/125)
Support alternate test apis via plugin system
Report error when supplied mutator name does not match (thanks artspb)
Report exit codes from coverage child process (thanks KyleRogers)
Treat JUnit tests with ClassRule annotation as one unit (thanks devmop)

Please note that any stored history files or sonar results are invalidated by this release.

1.0.0

Switch version numbering scheme
Upgrade to ASM 5.0.2
Fix for #114 - fails to run for java 8 when -parameters flag is set
#99 Support additionalClasspathElements property in maven plugin (thanks artspb)
#98 Do not mutate java 7 try with resources (thanks @artspb)
#109 extended remove conditional mutator (thanks @vrthra)

0.33

Move to Github
Upgrade of ASM to support Java 8 bytecode (thanks to "iirekm")
Partial support for JUnit categories (thanks to "chrisr")
New Remove Increments Mutator (thanks to Rahul Gopinath)
Minor logging improvements (thanks to Kyle Rogers aka Stephan Penndorf)
Fix for #92 - broken maven 2 support
Fix for #75 - incorrectly ignored tests in classes with both @Ignore and @BeforeClass / @AfterClass

0.32

restores java 7 compatibility
new remove conditionals mutator
support for mutating static initializers with TestNG
properly isolate classpaths when running via Ant
break builds on coverage threshold
allow JVM to be specified
support user defined test selection strategies
support user defined output format
support user defined test prioritisation
fix for issue blocking usage with Robolectric

Note, setup for Ant based projects changes in this release. See ant setup for details on usage.

0.31

Maven 2 compatibility restored
Much faster line coverage calculation
Fix for #78 - Error when PowerMockito test stores mock as member

This release also changes a number of internal implementation details, some of which may be of interest/importance to those maintaining tools that integrate with PIT.

Mutations are now scoped internally as described in https://groups.google.com/forum/#!topic/pitusers/E0-3QZuMYjE

A new class (org.pitest.mutationtest.tooling.EntryPoint) has been introduced that removes some of the duplication that existed in the various ways of launching mutation analysis.

0.30

Support for parametrized Spock tests
Support for JUnitParams tests
Fix for #73 - JUnit parameterised tests calling mutee during setup failing during mutation phase
Fix to #63 - ant task fails when empty options supplied
Ability to override maven options from command line
Ability to fail a build if it does not achieve a given mutation score
Performance improvement when tests use @BeforeClass or @AfterClass annotations
Slightly improved scheduling over multiple threads
Improved maven multi project support
Integration with source control for maven users

0.29

Incremental analysis (--historyInputLocation and --historyOutputLocation)
Inlined code detection turned on by default
Quieter loggging by default
Improved Java 7 support
Upgrade of ASM from 3.3 to 4
Fix for concurrency issues during coverage collection
Fix for #53 - problems with snapshot junit versions
Fix for #59 - duplicate dependencies set via maven

0.28

Inlined finally block detection (--detectInlinedCode)
New experimental switch statement mutator (contributed by Chris Rimmer)
Do not mutate Groovy classes
Fix for #33 - set user.dir to match surefire
Fix for #43 - optionally suppress timestamped folders (--timestampedReports=true/false)
Fix for #44 - concurrent modification exception when gathering coverage
Fix for #46 - incorrect setting of flags by ant task
Smaller memory footprint for main process
Faster coverage gathering for large codebases
Faster classpath scanning for large codebases
Support for JUnit 3 suite methods
Fixes for incorrect detection of JUnit 3 tests

Known issue - Fix for #33 may not resolve issue for maven 2 users.

Detection of Groovy code has not yet been tested with Groovy 2 which may generate substantially different byte code to earlier versions.

0.27

Much prettier reports
Now avoids mutating assert statements
Removed inScopeClasses option - use targetClasses and targetTests instead
Fix for 100% CPU usage when child JVM crashes
Fix for #35 #38 - experimental member variable mutator now corrects stack
Fix for #39 - order of classpath elements now maintained when running from maven

Upgrading users may need to modify their build due to removal of the inScopeClasses parameter

0.26

Ant support
New experimental mutator for member variables
Fix for #12 #27 - no longer hangs when code under test launches non daemon threads
Fix for #26 - now warns when no test library found on classpath
Fix for #30 - now errors if mutated classes have no line or source debug
Fix for #32 - now correctly handles of JUnit assumptions

Known issue - The new member variable mutator may cause errors in synchronized errors. The mutator is however disabled by default, and the generated errors are correctly handled by PIT.

0.25

TestNG support (experimental)
Fix for issue where mutations in nested classes not isolated from each other
Fix for broken classpath isolation for projects using xstream
Improved handling of JUnit parametrized tests
Ability to limit mutations to specific classpath roots (--mutableCodePaths)
Ability to add non launch classpath roots (--classPath) (experimental)
Read configuration values from XML (experimental)
Option to not throw error when no mutations found
Consistent ordering of classes in HTML report
Statistics written to console
Classes no longer loaded during initial classpath scanning
New syntax to easily enable all mutation operations

0.24

JMockit support
Option to output results in XML or CSV
Fix for #11
Improved INLINE_CONSTS mutator

0.23

Fix for issue 7 - source files not located

0.22

Upgrade of Xstream to 1.4.1 to enable OpenJDK 7 support
Fix for #5 - corruption of newline character in child processes
Ability to set child process launch arguments

0.21

Significant performance improvements
Support for powermock via both classloader (requires PowerMockIgnore annotation) and java agent
Minor error reporting and usability improvements
Fix for major defect around dependency analysis
PIT dependencies no longer placed on classpath when running via maven
Support for excluding certain classes or tests
Support for verbose logging

0.20

Limit number of mutations per class
Upgrade xstream to 1.3.1
Make available from maven central

0.19

Built in enum methods now excluded from mutation
Fixed bug around reporting of untested classes
Support for excluding tests greater than a certain distance from class
Support for excluding methods from mutation analysis
Performance improvements
Removed support for launching mutation reports from JUnit runner

0.18

First public release

Issues

Please consult our issue management rules before creating or working on issues.

Credits

Pitest is mainly the work of me but has benefited from contributions from many others.

Notable contributions not visible here as they were made before this code was migrated to github include

Nicolas Rusconi - Ant Task
Struan Kerr-Liddell - Improvements to html report
Stephan Pendorf - Multiple improvments including improved mutators

Although PIT does not incorporate any code from the Jumble project (http://jumble.sourceforge.net/), the Jumble codebase was used as a guide when developing some aspects of PIT.

Other stuff

The codebase is checked up on in a few places that give slower feedback than the github hooks.

maven2 on IBM JDK check

Comments

Junit5 support

Is that issue for you?

[22:57:22]E: Step 4/4: pitest (Command Line) (5s)
[22:57:22]i:     [Step 4/4] ##teamcity[buildStatisticValue key='buildStageDuration:buildStepRUNNER_5' value='0.0']
[22:57:22] :     [Step 4/4] Starting: E:\study\TeamCity\buildAgent\temp\agentTmp\custom_script4513790511060446848.cmd
[22:57:22] :     [Step 4/4] in directory: E:\study\TeamCity\buildAgent\work\b81bd2d58675567
[22:57:25] :     [Step 4/4] :compileJava UP-TO-DATE
[22:57:25] :     [Step 4/4] :processResources UP-TO-DATE
[22:57:25] :     [Step 4/4] :classes UP-TO-DATE
[22:57:25] :     [Step 4/4] :compileTestJava UP-TO-DATE
[22:57:25] :     [Step 4/4] :processTestResources UP-TO-DATE
[22:57:25] :     [Step 4/4] :testClasses UP-TO-DATE
[22:57:26]W:     [Step 4/4] 22:57:26 PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
[22:57:26]W:     [Step 4/4] 22:57:26 PIT >> INFO : Sending 107 test classes to minion
[22:57:26]W:     [Step 4/4] 22:57:26 PIT >> INFO : Sent tests to minion
[22:57:26]W:     [Step 4/4] 22:57:26 PIT >> INFO : MINION : 22:57:26 PIT >> INFO : Checking environment
[22:57:26]W:     [Step 4/4] 
[22:57:26]W:     [Step 4/4] 22:57:26 PIT >> INFO : MINION : lip 08, 2016 10:57:26 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:26]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:26]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.jupiter.engine.discovery.JavaElementsResolver resolveClass
[22:57:27]W:     [Step 4/4] WARNING: Class 'org.pojo.tester.assertion.AssertionErrorTest$MockOfAssertionError' could not be resolved
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
[22:57:27]W:     [Step 4/4] INFO: Discovered TestEngines with IDs: [junit-jupiter]
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : lip 08, 2016 10:57:27 PM org.junit.jupiter.engine.discovery.JavaElementsResolver resolveClass
[22:57:27]W:     [Step 4/4] WARNING: Class 'org.pojo.tester.assertion.AssertionErrorTest$MockOfAssertionError' could not be resolved
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : 22:57:27 PIT >> SEVERE : Error calculating coverage. Process will exit.
[22:57:27]W:     [Step 4/4] org.pitest.util.PitError: JUnit Jupiter
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] Please copy and paste the information and the complete stacktrace below when reporting an issue
[22:57:27]W:     [Step 4/4] VM : Java HotSpot(TM) 64-Bit Server VM
[22:57:27]W:     [Step 4/4] Vendor 
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : : Oracle Corporation
[22:57:27]W:     [Step 4/4] Version : 25.92-b14
[22:57:27]W:     [Step 4/4] Uptime : 775
[22:57:27]W:     [Step 4/4] Input -> 
[22:57:27]W:     [Step 4/4]  1 : -javaagent:E:\study\TeamCity\buildAgent\temp\buildTmp\1468011446165039291095250538736248591398183955901.jar
[22:57:27]W:     [Step 4/4]  2 : -Dfile.encoding=windows-1250
[22:57:27]W:     [Step 4/4]  3 : -Duser.country=PL
[22:57:27]W:     [Step 4/4]  4 : -Duser.language=p
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : l
[22:57:27]W:     [Step 4/4]  5 : -Duser.variant
[22:57:27]W:     [Step 4/4] BootClassPathSupported : true
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4]     at org.pitest.util.Unchecked.translateCheckedException(Unchecked.java:20)
[22:57:27]W:     [Step 4/4]     at org.pitest.util.IsolationUtils.convertForClassLoader(IsolationUtils.java:102)
[22:57:27]W:     [Step 4/4]     at org.pitest.junit.JUnitCustomRunnerTestU
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : nitFinder.descriptionToTest(JUnitCustomRunnerTestUnitFinder.java:154)
[22:57:27]W:     [Step 4/4]     at org.pitest.junit.JUnitCustomRunnerTestUnitFinder.access$000(JUnitCustomRunnerTestUnitFinder.java:42)
[22:57:27]W:     [Step 4/4]     at org.pitest.junit.JUnitCustomRunnerTestUnitFinder$1.apply(JUnitCustomRunnerT
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : estUnitFinder.java:133)
[22:57:27]W:     [Step 4/4]     at org.pitest.junit.JUnitCustomRunnerTestUnitFinder$1.apply(JUnitCustomRunnerTestUnitFinder.java:129)
[22:57:27]W:     [Step 4/4]     at org.pitest.functional.FCollection.mapTo(FCollection.java:40)
[22:57:27]W:     [Step 4/4]     at org.pitest.functional.FCollection.map(FCollection.java:48
[22:57:27] :     [Step 4/4] :pitest FAILED
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : )
[22:57:27]W:     [Step 4/4]     at org.pitest.functional.MutableList.map(MutableList.java:127)
[22:57:27]W:     [Step 4/4]     at org.pitest.functional.MutableList.map(MutableList.java:24)
[22:57:27]W:     [Step 4/4]     at org.pitest.junit.JUnitCustomRunnerTestUnitFinder.splitIntoFilteredUnits(JUnitCustomRunnerTestUnitFinder.java:123)
[22:57:27]W:     [Step 4/4]     at o
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : rg.pitest.junit.JUnitCustomRunnerTestUnitFinder.findTestUnits(JUnitCustomRunnerTestUnitFinder.java:58)
[22:57:27]W:     [Step 4/4]     at org.pitest.junit.CompoundTestUnitFinder.findTestUnits(CompoundTestUnitFinder.java:20)
[22:57:27]W:     [Step 4/4]     at org.pitest.junit.CompoundTestUnitFinder.findTestUnits(Com
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : poundTestUnitFinder.java:20)
[22:57:27]W:     [Step 4/4]     at org.pitest.testapi.execute.FindTestUnits.findTestUnits(FindTestUnits.java:57)
[22:57:27]W:     [Step 4/4]     at org.pitest.testapi.execute.FindTestUnits.getTestUnits(FindTestUnits.java:40)
[22:57:27]W:     [Step 4/4]     at org.pitest.testapi.execute.FindTestUnits.findTestUnitsFor
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> SEVERE : Coverage generator Minion exited abnormally due to UNKNOWN_ERROR
[22:57:27]W:     [Step 4/4] Exception in thread "main" org.pitest.util.PitError: Coverage generation minion exited abnormally!
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] Please copy and paste the information and the complete stacktrace below when reporting an issue
[22:57:27]W:     [Step 4/4] VM : Java HotSpot(TM) 64-Bit Server VM
[22:57:27]W:     [Step 4/4] Vendor : Oracle Corporation
[22:57:27]W:     [Step 4/4] Version : 25.92-b14
[22:57:27]W:     [Step 4/4] Uptime : 1160
[22:57:27]W:     [Step 4/4] Input -> 
[22:57:27]W:     [Step 4/4]  1 : -Dfile.encoding=windows-1250
[22:57:27]W:     [Step 4/4]  2 : -Duser.country=PL
[22:57:27] :     [Step 4/4] 
[22:57:27]W:     [Step 4/4]  3 : -Duser.language=pl
[22:57:27] :     [Step 4/4] BUILD FAILED
[22:57:27]W:     [Step 4/4]  4 : -Duser.variant
[22:57:27] :     [Step 4/4] 
[22:57:27]W:     [Step 4/4] BootClassPathSupported : true
[22:57:27] :     [Step 4/4] Total time: 4.57 secs
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] Please copy and paste the information and the complete stacktrace below when reporting an issue
[22:57:27]W:     [Step 4/4] VM : Java HotSpot(TM) 64-Bit Server VM
[22:57:27]W:     [Step 4/4] Vendor : Oracle Corporation
[22:57:27]W:     [Step 4/4] Version : 25.92-b14
[22:57:27]W:     [Step 4/4] Uptime : 1161
[22:57:27]W:     [Step 4/4] Input -> 
[22:57:27]W:     [Step 4/4]  1 : -Dfile.encoding=windows-1250
[22:57:27]W:     [Step 4/4]  2 : -Duser.country=PL
[22:57:27]W:     [Step 4/4]  3 : -Duser.language=pl
[22:57:27]W:     [Step 4/4]  4 : -Duser.variant
[22:57:27]W:     [Step 4/4] BootClassPathSupported : true
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4]     at org.pitest.util.Unchecked.translateCheckedException(Unchecked.java:20)
[22:57:27]W:     [Step 4/4]     at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:103)
[22:57:27]W:     [Step 4/4]     at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:49)
[22:57:27]W:     [Step 4/4]     at org.pitest.mutationtest.tooling.MutationCoverage.runReport(MutationCoverage.java:110)
[22:57:27]W:     [Step 4/4]     at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:103)
[22:57:27]W:     [Step 4/4]     at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:45)
[22:57:27]W:     [Step 4/4]     at org.pitest.mutationtest.commandline.MutationCoverageReport.runReport(MutationCoverageReport.java:76)
[22:57:27]W:     [Step 4/4]     at org.pitest.mutationtest.commandline.MutationCoverageReport.main(MutationCoverageReport.java:45)
[22:57:27]W:     [Step 4/4] Caused by: org.pitest.util.PitError: Coverage generation minion exited abnormally!
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] Please copy and paste the information and the complete stacktrace below when reporting an issue
[22:57:27]W:     [Step 4/4] VM : Java HotSpot(TM) 64-Bit Server VM
[22:57:27]W:     [Step 4/4] Vendor : Oracle Corporation
[22:57:27]W:     [Step 4/4] Version : 25.92-b14
[22:57:27]W:     [Step 4/4] Uptime : 1160
[22:57:27]W:     [Step 4/4] Input -> 
[22:57:27]W:     [Step 4/4]  1 : -Dfile.encoding=windows-1250
[22:57:27]W:     [Step 4/4]  2 : -Duser.country=PL
[22:57:27]W:     [Step 4/4]  3 : -Duser.language=pl
[22:57:27]W:     [Step 4/4]  4 : -Duser.variant
[22:57:27]W:     [Step 4/4] BootClassPathSupported : true
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4]     at org.pitest.coverage.execute.DefaultCoverageGenerator.gatherCoverageData(DefaultCoverageGenerator.java:142)
[22:57:27]W:     [Step 4/4]     at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:87)
[22:57:27]W:     [Step 4/4]     ... 6 more
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : AllSuppliedClasses(FindTestUnits.java:29)
[22:57:27]W:     [Step 4/4]     at org.pitest.coverage.execute.CoverageMinion.discoverTests(CoverageMinion.java:148)
[22:57:27]W:     [Step 4/4]     at org.pitest.coverage.execute.CoverageMinion.getTestsFromParent(CoverageMinion.java:128)
[22:57:27]W:     [Step 4/4]     at org.pitest.coverage.execute.Co
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : verageMinion.main(CoverageMinion.java:79)
[22:57:27]W:     [Step 4/4] Caused by: java.lang.ClassNotFoundException: JUnit Jupiter
[22:57:27]W:     [Step 4/4]     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
[22:57:27]W:     [Step 4/4]     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[22:57:27]W:     [Step 4/4]     at sun.misc.Launcher$AppClass
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : Loader.loadClass(Launcher.java:331)
[22:57:27]W:     [Step 4/4]     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[22:57:27]W:     [Step 4/4]     at java.lang.Class.forName0(Native Method)
[22:57:27]W:     [Step 4/4]     at java.lang.Class.forName(Class.java:348)
[22:57:27]W:     [Step 4/4]     at org.pitest.util.IsolationUtils.convertForClassLoader(IsolationUtil
[22:57:27]W:     [Step 4/4] 22:57:27 PIT >> INFO : MINION : s.java:100)
[22:57:27]W:     [Step 4/4]     ... 18 more
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] FAILURE: Build failed with an exception.
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] * What went wrong:
[22:57:27]W:     [Step 4/4] Execution failed for task ':pitest'.
[22:57:27]W:     [Step 4/4] > Process 'command 'C:\Program Files\Java\jdk1.8.0_92\bin\java.exe'' finished with non-zero exit value 1
[22:57:27]W:     [Step 4/4] 
[22:57:27]W:     [Step 4/4] * Try:
[22:57:27]W:     [Step 4/4] Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
[22:57:27]W:     [Step 4/4] Process exited with code 1
[22:57:27]i:     [Step 4/4] ##teamcity[buildStatisticValue key='buildStageDuration:buildStepRUNNER_5' value='5129.0']
[22:57:27]E:     [Step 4/4] Step pitest (Command Line) failed

opened by sta-szek 31

Aggregated report for multi-module projects

From [email protected] on April 26, 2012 14:34:14

I would be useful to have a better support for multi-module projects. Currently it is possible to run pitest on multi-module project (tested with Maven plugin), but there are missing for me aggregated statistics.

Because it could be time consuming to implement at least links from a parent report to the child reports would be useful (now it is required to find reports for every module separately).

Probably others will have some other ideas about multi-module builds.

Original issue: http://code.google.com/p/pitestrunner/issues/detail?id=41
enhancement imported

opened by hcoles 24
Naked receiver mutator

Mutator that replaces a method call with the object on which the method is called in case the return type of the method matches the type of the object.

It is named after @mbj mutation operator in mutant performing the same mutation.

opened by UrsMetz 23
Initial DSL mutator concept
Also on users lists: https://groups.google.com/forum/#!topic/pitusers/o4Vpb_KI9Fc

I'm looking into mutating DSLs, for instance camel DSL:

from("file:src/data?noop=true") .to("file:target/messages/uk") .to("file:target/messages/others");

So I wanna mutations to remove DSL steps to make sure all my configurations (or at least most) are meaningful and being tested somewhere. So that piece of code would be mutated into something like:

from("file:src/data?noop=true") .to("file:target/messages/others");

AND/OR

from("file:src/data?noop=true") .to("file:target/messages/uk");

I also wanna test @Builders and queries DSL.

So pitest would remove query conditions or builder parameters.

I started creating my own mutator for it (which I will submit as a pull request once properly tested)

Right now, it does remove any method that return the same type as the class that declares it. https://github.com/hcoles/pitest/pull/221/files#diff-abcacb866451ca350bef8b81ef634c9bR35

It is not merge ready, it is just to illustrate what I'm doing.

But when I do that, dsl.chain(1) become to have the same effect as dsl = null would be https://github.com/hcoles/pitest/pull/221/files#diff-0f0887a036c22255aa52f202e494fe58R90

And a NPE happen:

java.lang.NullPointerException at org.pitest.mutationtest.engine.gregor.mutators.DSLMethodCallMutatorTest$HasDslMethodCall.call(DSLMethodCallMutatorTest.java:90) at org.pitest.mutationtest.engine.gregor.mutators.DSLMethodCallMutatorTest$HasDslMethodCall.call(DSLMethodCallMutatorTest.java:1) at org.pitest.mutationtest.engine.gregor.MutatorTestBase.runInClassLoader(MutatorTestBase.java:151) at org.pitest.mutationtest.engine.gregor.MutatorTestBase.mutateAndCall(MutatorTestBase.java:117) at org.pitest.mutationtest.engine.gregor.MutatorTestBase.assertMutantCallableReturns(MutatorTestBase.java:106) at org.pitest.mutationtest.engine.gregor.mutators.DSLMethodCallMutatorTest.shouldRemoveDslMethods(DSLMethodCallMutatorTest.java:128)

I think I can find the problem by myself if is there any way to get ASMifier running on a class after mutated, is that possible? Or, if what I'm doing wrong is obvious, can someone point out?

Is there anything obviously wrong that you guys see on my code?

Keep in mind this is just my initial work, as I master what need to be done I will fix checkstyle and make the design as similar as possible to what is already there.

Also, is this functionality something that worth incorporating to pitest?

Marvin
opened by velo 22
Enable scmMutationCoverage goal to analyse files in the last commit

For use in Continuous Integration scenarios it would be helpful to run mutation analysis only on the files changed last. At the moment this is only possible for uncommitted changes, which is only useful for analysis on developers' machines before pushing.

I'm not sure how difficult this would be since the Maven SCM Plugin does not seem to have a generic way to get hold of the files modified in a certain revision.

Since most CI mechanisms now the revision under consideration at runtime it would be ok if the revision would be a parameter and the SCM would not have to figure out the head revision.
enhancement

opened by noroute 21
raise minimum jdk to 1.7

It is set to 1.5 in parent pom and I think this makes too much very complicate. (Compatibility is a burden)

Supporting only last and current version seems to be reasonable to me.

What is you opinion?

opened by lkwg82 20
False error: Mutation testing requires a green suite
I am running pitest against a green suite. Though specific test classes fail when run with pitest.

The failing test classes have the following annotations:

@RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(locations = "/domain-test-context.xml")

This setup is being used so that we test code against a real DB.

If I am not mistaken these annotations is the reason these classes fail when being run by pitest.

Is this solveable ?

If not then is it possible to somehow configure pitest to ignore such tests ?
opened by Kidlike 20

unable to use latest versions of pitest, tesng and jmockit together

Related to https://github.com/jmockit/jmockit1/issues/24 which describes problems when using gradle+testng+pitest+jmockit.

When using maven pitest plugin and a very simple project (a pom.xml, a "main" class and a test class) simply adding jmockit dependency breaks the build even if the jmockit is not used in tests.

The pom xml is:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>sandbox</groupId>
    <artifactId>sandbox-pitest-jmockit</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.jmockit</groupId>
            <artifactId>jmockit</artifactId>
            <version>1.13</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.8.8</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>1.1.1</version>
            </plugin>
        </plugins>
    </build>
</project>

mvn clean org.pitest:pitest-maven:mutationCoverage results in a slave failure which seems to be due to Caused by: java.lang.ClassNotFoundException: org.junit.runner.manipulation.Filter (where junit came from?).

Removing the jmockit dependency from the pom above will allow the pitest to run successfully.

opened by estekhin 20

Transfer Pitclipse repository to the Pitest organization
Hi,

I have recently started to contribute to Pitclipse, which provides an Eclipse IDE integration for Pitest. @philglover (the author of the project) and I think that it would be relevant to move the repository under the Pitest organization.

Would that be possible?

@philglover according to the docs I believe that it will be up to you to start the transfer

Moreover, we would like to benefit from the following CI tools:

Travis CI (the repo is already using it, but I guess that some paths will have to be changed)

CodeCov

SonarCloud

I am a collaborator of the project but it seems that it's not enough to be able to activate them (not sure about SonarCloud though) so one of the owners will have to do this.
opened by echebbi 19
JDK7 try-with-resources with catch produces failing mutations for autoclose code

From [email protected] on June 23, 2012 02:42:05

What steps will reproduce the problem? 1. Use a try-with-resources with a catch for IOException What is the expected output? What do you see instead? No failing mutations from autoclose What version of the product are you using? On what operating system? 0.27, Ubuntu 12.04 x64 Please provide any additional information below. try (StringWriter writer = new StringWriter()) { } catch (IOException e) { throw new RuntimeException(e); }

Original issue: http://code.google.com/p/pitestrunner/issues/detail?id=48
bug imported

opened by hcoles 19

Mutant survives due to wrong coverage analysis

I found a bug in coverage analysis. If there is a local variable involved in an if-else if-else construct, coverage analysis does not detect the correct test(s).

I simplified my real-world example to the following lines:

Code under Test

public class Simple {
    public static int method(int digits) {
        final int result;

        if (digits == 1) {
            final int day = 6;

            result = day * 7;
        } else if (digits == 2) {
            result = 13;
        } else {
            result = 0;
        }

        return result;
    }
}

Test Cases:

public class SimpleTest {
    @Test
    public void test1() {
        assertThat(Simple.method(1), is(42));
   }

    @Test
    public void test2() {
        assertThat(Simple.method(2), is(13));
    }

    @Test
    public void test3() throws Exception {
        assertThat(Simple.method(3), is(0));
    }
}

The Negate-Condition Mutation of the else-if construct (checking digits==2) survives although this would be killed by test2 or test3 --- but only test1 is executed against this mutation. The verbose log reveals the following (important lines highlighted in bold):

stdout  : Installing PIT agent
stderr  : 13:32:43 PIT >> FINE : Running mutation MutationDetails [id=MutationIdentifier [location=Location [clazz=net.cyphoria.Simple, method=method, methodDesc=(I)I], indexes=[0], mutator=org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator], fstderr  : ilename=Simple.java, block=1, lineNumber=22, description=negated conditional, testsInOrder=[net.cyphoria.SimpleTest.test2(net.cyphoria.SimpleTest), net.cyphoria.SimpleTest.test3(net.cyphoria.SimpleTest), net.cyphoria.SimpleTest.test1(net.cyphoria.SimpleTesstderr  : t)]]
stderr  : 13:32:43 PIT >> FINE : mutating method method
stderr  : 13:32:43 PIT >> FINE : 3 relevant test for method
stderr  : 13:32:43 PIT >> FINE : replaced class with mutant in 3 ms
stderr  : 13:32:43 PIT >> FINE : Running 1 units
stderr  : 13:32:43 PIT >> FINE : Finished
stderr  : 13:32:43 PIT >> FINE : Mutation MutationIdentifier [location=Location [clazz=net.cyphoria.Simple, method=method, methodDesc=(I)I], indexes=[0], mutator=org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator] detected = KILLED by net.cyphostderr  : ria.SimpleTest.test2(net.cyphoria.SimpleTest)
stderr  : 13:32:43 PIT >> FINE : processed mutation in 94 ms.
stderr  : 13:32:43 PIT >> FINE : Running mutation MutationDetails [id=MutationIdentifier [location=Location [clazz=net.cyphoria.Simple, method=method, methodDesc=(I)I], indexes=[1], mutator=org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator], fstderr  : ilename=Simple.java, block=3, lineNumber=26, description=negated conditional, testsInOrder=[net.cyphoria.SimpleTest.test1(net.cyphoria.SimpleTest)]]
stderr  : 13:32:43 PIT >> FINE : mutating method method
13:32:43 PIT >> FINE : MutationIdentifier [location=Location [clazz=net.cyphoria.Simple, method=method, methodDesc=(I)I], indexes=[0], mutator=org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator] KILLED by net.cyphoria.SimpleTest.test2(net.cyphoria.SimpleTest)
stderr  : 13:32:43 PIT >> FINE : 1 relevant test for method
stderr  : 13:32:43 PIT >> FINE : replaced class with mutant in 1 ms
stderr  : 13:32:43 PIT >> FINE : Running 1 units
stderr  : 13:32:43 PIT >> FINE : Finished
stderr  : 13:32:43 PIT >> FINE : Mutation MutationIdentifier [location=Location [clazz=net.cyphoria.Simple, method=method, methodDesc=(I)I], indexes=[1], mutator=org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator] detected = SURVIVED
stderr  : 13:32:43 PIT >> FINE : processed mutation in 8 ms.
stderr  : 13:32:43 PIT >> FINE : Running mutation MutationDetails [id=MutationIdentifier [location=Location [clazz=net.cyphoria.Simple, method=method, methodDesc=(I)I], indexes=[0], mutator=org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator], filename=stderr  : Simple.java, block=5, lineNumber=32, description=replaced return of integer sized value with (x == 0 ? 1 : 0), testsInOrder=[net.cyphoria.SimpleTest.test2(net.cyphoria.SimpleTest), net.cyphoria.SimpleTest.test3(net.cyphoria.SimpleTest), net.cyphoria.SimpleTest.test1(net.cyphoria.SimpleTest)]]
stderr  : 13:32:43 PIT >> FINE : MutationIdentifier [location=Location [clazz=net.cyphoria.Simple, method=method, methodDesc=(I)I], indexes=[1], mutator=org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator] SURVIVED
stderr  : 13:32:43 PIT >> FINE : mutating method method
stderr  : 13:32:43 PIT >> FINE : 3 relevant test for method
stderr  : 13:32:43 PIT >> FINE : replaced class with mutant in 1 ms
stderr  : 13:32:43 PIT >> FINE : Running 1 units
stderr  : 13:32:43 PIT >> FINE : Finished
stderr  : 13:32:43 PIT >> FINE : Mutation MutationIdentifier [location=Location [clazz=net.cyphoria.Simple, method=method, methodDesc=(I)I], indexes=[0], mutator=org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator] detected = KILLED by net.cyphoria.Simpstderr  : leTest.test2(net.cyphoria.SimpleTest)
13:32:43 PIT >> FINE : processed mutation in 10 ms.
13:32:43 PIT >> FINE : MutationIdentifier [location=Location [clazz=net.cyphoria.Simple, method=method, methodDesc=(I)I], indexes=[0], mutator=org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator] KILLED by net.cyphoria.SimpleTest.test2(net.cyphoria.SimpleTest)
13:32:43 PIT >> FINE : Exit code was - OK
13:32:43 PIT >> FINE : Slave exited ok
13:32:43 PIT >> FINE : Finished
13:32:43 PIT >> INFO : Completed in 1 seconds

Additionally I could make the following observations:

Removing the local variable day resolves the problem
Keeping the local variable but replacing the assignment to result with a return statement resolves the problem

opened by StefanPenndorf 17

How to ensure testng executes afterMethod or afterSuite

My project will start the beforeSuite several times, and the mock port will be started in the beforeSuite. However, if the variant fails to execute the test case, the afterSuite method will be skipped (even if I add the alwaysRun=true annotation), which causes me not to close the mock port in the afterSuite. How can I solve this problem? Does the test provide a configuration that does not skip the AfterSuite phase? I look forward to your answer. Thank you!

opened by wlopper 0
RemoveSwitchMutator: false positive when case points to default

Project: https://github.com/checkstyle/checkstyle

Mutations: All at https://github.com/checkstyle/checkstyle/blob/master/config/pitest-suppressions/pitest-annotation-suppressions.xml

Command: mvn test-compile org.pitest:pitest-maven:mutationCoverage -Ppitest-annotation

Source 1: https://github.com/checkstyle/checkstyle/blob/master/src/main/java/com/puppycrawl/tools/checkstyle/checks/annotation/AnnotationUseStyleCheck.java#L442

The report says case 1 through 3 is killed, but 4 is survived. This is an enum and the value is going against the source code ordering of the enums, so it references IGNORE which is attached to the default case. All the other enums are verified as being resolved. Moving IGNORE to any other case label results in tests failures, so this isn't what pitest is looking for. There is no way to kill this mutation since the case is directly attached to the default case.

Source 2: https://github.com/checkstyle/checkstyle/blob/master/src/main/java/com/puppycrawl/tools/checkstyle/checks/annotation/SuppressWarningsCheck.java#L250-L251

TokenTypes.IDENT and TokenTypes.DOT are the ones with the specific case values. While they are not directly attached to the default, they do have the same code as the default. I tried changing the code so all 3 cases are attached, but it still survives and thus there is no way to kill the mutation.

I would expect these mutations to not even be generated since they are basically non-killable no matter what is done during execution. It would be better if pitest was smart and saw the code between the movable case and the default is the same code, but I am ok if we just ignore cases directly tied to the default.

opened by rnveach 2

AspectJ mutators

Hello, With AspectJ, I have a false positive mutator. I think it is because of the null check that inserts AspectJ. At the generated class I see:

public class DemoClass {
  public DemoClass() {
  }

  @MyAspect
  public int init() {
    JoinPoint var1 = Factory.makeJP(ajc$tjp_0, this, this);
    MyAspectImpl var10002 = MyAspectImpl.aspectOf();
    ProceedingJoinPoint var10003 = (ProceedingJoinPoint)var1;
    Annotation var10004 = ajc$anno$0;
    if (var10004 == null) {
      var10004 = ajc$anno$0 = DemoClass.class.getDeclaredMethod("init").getAnnotation(MyAspect.class);
    }

    return Conversions.intValue(init_aroundBody1$advice(this, var1, var10002, var10003, (MyAspect)var10004));
  }

  static {
    ajc$preClinit();
  }
}

And the report returns:

I have created this demo repo for check the issue: https://github.com/sanmibuh/pitest-aspectj You could check running ./gradlew clean build pitest

opened by sanmibuh 3

Add throwables in CheckTestHasFailedResultListener
adding a new property on CheckTestHasFailedResultListener to store all throwbales encountered.

printing the list of throwables from the result listener in MutationTestWorker.
opened by VijayKrishna 0

ClassNotFoundException: org.pitest.functional.predicate.Predicate with command line

Hi I am trying to use command line version of pit to mutate a class in defects4j. My command line input is

java -cp ~/tools/pit/pitest-command-line-1.1.5.jar:~/tools/pit/pitest-1.1.5.jar:~/tools/junit/junit-4.12.jar \ 
org.pitest.mutationtest.commandline.MutationCoverageReport \
--reportDir ./output  \
--targetClasses org.apache.commons.lang3.math.NumberUtils \
--targetTests org.apache.commons.lang3.math.NumberUtilsTest \
--sourceDirs src/main/java

and the result shows an exception:

Exception in thread "main" java.lang.NoClassDefFoundError: org/pitest/functional/predicate/Predicate
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
        at java.lang.Class.getMethod0(Class.java:2866)
        at java.lang.Class.getMethod(Class.java:1676)
        at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.pitest.functional.predicate.Predicate
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

All the paths to the jars are valid and my java version is 1.8 I wonder what's wrong with this, thank you !

opened by Iroski 1

Mutations falsely killed by TIMED_OUT under high system CPU load
When running a parallel build in Gradle (or any other parallelizable build system), sometimes Pitest will report a timeout coverage for one or more of the modules. This is due to the fact that some tests don't get to run as far as necessary to determine the mutation to be surviving. This is sort of highlighted in https://pitest.org/faq/#:~:text=I%E2%80%99m%20seeing%20a%20lot%20of%20timeouts%2C%20what%E2%80%99s%20going%20on%3F but maybe this report could help anyway.

In a simple case, the production code could look like

for (int i = 0; i < 100000; i++) { System.out.println(i); }

When changing < to <= the test can time out if the system is under high load.

Unfortunately, Gradle has its own worker count calculation based on number of cores, and then Pitest will fork out further processes, which causes starvation. Reducing the build system workers and/or Pitest workers alleviates the problem, but it also causes under-utilization of the system during non-pathological cases.

What I'm envisioning is some sort of dynamic test timeout, which is calculated by overall system load vs VM load. Roughly speaking, the timeout should increase if the system load is increasing, and decrease if the VM load is increasing.
opened by davidburstrom 1

Releases(1.10.4)

1.10.4(Jan 4, 2023)
#1134 Add excludedRunners parameter to command line interface and Ant

#1131 Support surefire late property evaluation syntax for arglines

Source code(tar.gz)
Source code(zip)
1.10.1(Dec 2, 2022)
#1124 Fix for bug introduced in 1.10.0 where bridge methods were treated as valid lines of code for line coverage

Source code(tar.gz)
Source code(zip)
1.10.0(Nov 29, 2022)
#1067 Improved Quarkus and Roboelectric support

#1116 Option to exclude lines 0 and 1 of all classes

#1118 Detect missing plugins

As a result of #1067 it is important that mutations are only created for a single class for each JVM. The MutationGrouper extension point has therefore been removed as this allowed this constraint to be violated. Any third party plugins using this extension are no longer supported.

Releases also includes multiple changes to internal data structures. Any third party plugins relying on these structures may need to be updated.

1116 provides a coarse way to exclude auto-generated code not picked up by pitest's other filters. It is disabled by default as it may exclude code that is valid for mutation in some situations, but can be enabled by adding the feature string +nofirstline.
Source code(tar.gz)
Source code(zip)
1.9.11(Nov 16, 2022)
#1105 Aggregator resolves wrong file for out of package kotlin files with same name

#1111 Fix log typo (thanks @wangbo15)

Source code(tar.gz)
Source code(zip)
1.9.10(Nov 11, 2022)
#1106 Do not replace Map.of with Collections.emptyMap

#1109 Relax source file debug check

Source code(tar.gz)
Source code(zip)
1.9.9(Nov 10, 2022)

#1098 Include case values in switch mutation descriptions
Source code(tar.gz)
Source code(zip)
1.9.8(Oct 10, 2022)

#1094 Search for source files outside of package dirs

Adds support for languages such as Kotlin where source files may not be within a directory structure which matches their package. Note that if the same filename occurs in more than one location, pitest cannot guarantee to resolve the correct file.
Source code(tar.gz)
Source code(zip)
1.9.6(Oct 3, 2022)
1092 - Upgrade to ASM 9.4

1091 - Properly escape coverage XML

1088 - Remove unused dependency distance param

1085 - Do not produce timestamp reports by default

Note that 1085 changes the default behaviour so reports are no longer written to timestamped directories by default. If you require this behaviour you will need to explicitly enable it.
Source code(tar.gz)
Source code(zip)
1.9.5(Aug 22, 2022)
#1078 - Alter Config Updater interface

#1079 - Avoid JBoss logging by default

1078 is a breaking interface change, but is included in a point release as it is believed there are currently no external clients.
Source code(tar.gz)
Source code(zip)
1.9.4(Aug 3, 2022)
#1063 - Improve filtering of equivalent return mutants

#1064 - Remove legacy return val mutators and old defaults group

#1066 - Expand static initializer filtering

#1070 - Remove jacoco agent if present on argline

Source code(tar.gz)
Source code(zip)
1.9.3(Jul 18, 2022)
#1052 - Support maven argLine property and single string argLines

#1054 - Provide control over auto addition of -Djava.awt.headless=true

#1055 - Option to autoset number of threads

#1056 - Set timeout for socket communication

#1058 - Set defaults for input and output encoding for programmatic api

1054 Moves support of auto adding headless=true (to prevent keyboard focus being stolen on Macs) into a feature. It is enabled by default, but can be disabled by adding -MACOS_FOCUS to the features string.

1055 adds the option to guess the appropriate number of threads for the current machine by adding +auto_threads the features string. This option is disabled by default and designed for local use. It is not recommended for use on a CI server.
Source code(tar.gz)
Source code(zip)
1.9.2(Jul 8, 2022)
#1050 - Check minions are alive to prevent process hangs

Source code(tar.gz)
Source code(zip)
1.9.1(Jul 6, 2022)

#1047 Mitigation for intemittent process hangs
Source code(tar.gz)
Source code(zip)
1.9.0(Jun 23, 2022)
#1035 - Improve coverage memory consumption

#1038 - Support Maven Site Plugin 3.12.0+ (thanks IOs)

#1040 - Faster JUnit 5 coverage calculation

#1040 - Remove dependencyDistance parameter

This release greatly improves the speed and efficiency of the coverage analysis performed before mutation analysis begins (particularly if you are using JUnit 5).

As a result of #1040 any external pitest test plugins (eg JUnit 5, TestNG, arcmutate accelerator) must be updated to a compatible version (1.0.0) when upgrading.

The dependencyDistance parameter is no longer supported.

History files from earlier releases are incompatible and should be deleted when upgrading.
Source code(tar.gz)
Source code(zip)
1.8.1(Jun 16, 2022)
#637 - Support different input and output encodings (thanks @qxo)

#705 - Allow + in file paths (thanks @ali-ghanbari)

#903 - Filter mutants in singleton constructors

#1025 - Rework String Switch filtering

#1027 - Rework assert filtering and remove legacy filter mechanism

#1030 - Filter enum switch junk mutations

Source code(tar.gz)
Source code(zip)
1.8.0(May 20, 2022)
#1017 - Improve static infinite loop detection

#1020 - Rework NFA library and try with resource filtering

#1022 - Support quarkus test

Source code(tar.gz)
Source code(zip)
1.7.6(Apr 29, 2022)
#1008 Upgrade ASM for initial Java 19 support

#1016 Fail cleanly when no working test plugin supplied

#1015 Move TestNG support out of main codebase

As a result of #1015 the TestNG plugin must now be configured when using pitest in codebases tested by TestNG. In earlier releases support was included automatically.
Source code(tar.gz)
Source code(zip)
1.7.5(Mar 23, 2022)

#625 / #983 Fixed report aggregation (thanks @therealryan and @aurelien-baudet) #985 Handle empty surefire excludes #993 Move rv mutators to external plugin #994 / #995 Filter equivalent boxed return true mutants in try blocks #968 / 997 Improve filtering of equivalent empty return mutants #999 / 1003 New parameter to locate root of multi module projects

As a result of #993, anyone wishing to use the research orientated RV operators must configure the pitest-rv plugin

https://github.com/pitest/pitest-rv-plugin
Source code(tar.gz)
Source code(zip)
1.7.4(Feb 2, 2022)
#965 - BigDecimal Mutator (thanks @MarcinNowak-codes)

#750 - Support -DskipTests from commandline (thanks @marwin1991)

Source code(tar.gz)
Source code(zip)
1.7.3(Nov 8, 2021)
#952 Mutate map return to emptyMap instead of null

#954 Allow mutators to be excluded

#957 Filter equivalent mutations to Boolean.TRUE and Boolean.FALSE

#954 allows mutators to be excluded by prepending a - to their name when specifying the mutator set. e.g -TRUE_RETURNS
Source code(tar.gz)
Source code(zip)
1.7.2(Oct 12, 2021)
#943 Change default mutators - replace negate conditional with remove conditional

#946 Mutate stream returns to empty stream instead of null

Source code(tar.gz)
Source code(zip)
1.7.1(Sep 29, 2021)
#932 Improve switch mutation descriptions

#934 Configure console output verbosity

#940 Hide internal features

#942 Fix for 3rd party mutation operators not added to client classpath

#939 Improve caching of bytecode retrieval

Source code(tar.gz)
Source code(zip)
1.7.0(Sep 1, 2021)

#923 Internal interface changes #930 Pluggable mutators

Due to internal changes some third party plugins maybe incompatible with this release.

All history files should be deleted before upgrading.

The names of the remove conditionals mutators have changed slightly as a result of #930 and may need to be updated in build scripts if explicitly activated.
Source code(tar.gz)
Source code(zip)
1.6.9(Aug 12, 2021)
#922 Filter equivalent stream.empty mutants in flatMap calls

#921 Guarantee order of mutation operators

#919 Filter junk mutations in java records

Source code(tar.gz)
Source code(zip)
pitest-parent-1.6.8(Jul 26, 2021)
#917 - Add method to retrieve all mutator ids for pitclipse and other tooling

#913 - Bump ASM to 9.2

Source code(tar.gz)
Source code(zip)
1.6.7(Jun 3, 2021)

#897 Fix description when replacing value with empty set #900 Support multiple test engines #822 Fix regression of NPE when reading classpath manifest #798 Fix regression of jvmArgs parameter support via maven #797 Fix regression of line coverage on console

As a result of #900 the testPlugin parameter is now defunct, but pitest will continue to accept it without error for this release. Pitest will use any test-engines supplied on the classpath, falling back to the built in JUnit4 support when other test engines cannot handle the class.
Source code(tar.gz)
Source code(zip)
1.6.6(Apr 29, 2021)

#891 - Fix history performance #889 - Do not scan classpath for history when option not supplied #887 - Fix css for timeouts and memory errors #888 - Remove defunct max mutants per class perameter #890 - Remove defunct mutate static initalizers parameter
Source code(tar.gz)
Source code(zip)
1.6.5(Apr 20, 2021)

#882 Avoid running tests when no mutants possible
Source code(tar.gz)
Source code(zip)
1.6.4(Mar 11, 2021)

#862 Update ASM for Java 16
Source code(tar.gz)
Source code(zip)
1.6.3(Feb 16, 2021)
#853 Fix case insensitive feature check

#855 Make coverage data available to interceptors

#857 Pass stderr/out data by line (thanks @kgeilmann)

#858 Fix test not failing when maxSurviving is 0 (thanks @alexkoltz)

#860 Fix junk mutations in try with resources in java 11

Source code(tar.gz)
Source code(zip)