Tools for keeping your cloud operating in top form. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures.

Add a bunch more chaos types, using the new script support.

Also make sure that the Apache license is on all the files I added.

The chaos monkey army is documented here: https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-Army

Because not all ASG are created equally, we needed a way to tune the killing of instances and didn't want to add an entry in chaos.properties for every new ASG. This PR allows you to add a tag to your ASG with the key "chaosMonkey.aggressionCoefficient" and that value will be multiplied by your effective probability for that ASG. For example if my ASG effective probability is 0.5, I can tag my Cassandra ASG's [as an example] with chaosMonkey.aggressionCoefficient=0.5 so that my Cassandra ASG's will have an effective probability of 0.25.

It seems odd that the only notification of terminations is via SES, not SNS.

Suggested feature: for each termination, send a JSON message to some configured SNS topic, including at least the instance id terminated, and the arn of the Autoscaling Group of which it was a member.

There is no similar ASG concept in vSphere as in AWS and auto grouping all VMs by parrent folder name makes over-chaos to the whole vSphere. Specifying all VMs under the same absolute folder path as a group is a more precise way to inject faulty into target applications. So I add a VsphereChaosCrawler to crawl only VMs under target folder.

This request contains one commit that was merged in from SimianArmy on Netflix:master (removal of java 1.6 requirement). Tests run (and new tests written), no regressions. Comments welcome!

The purpose of this PR is to start work on abstraction of "cloud" and "cluster" ideas (all still abstract) from some of the tightly coupled AWS dependencies that exist in those base classes. This particular branch allows a non-aws user to send notifications through a specified SMTP server. Our particular use case is that we are a cloud provider (rather than an app developer who deploys to a cloud provider). SimianArmy has the same intrinsic value for cloud providers (taking out machines to see if the cluster will survive without interrupting the user experience), but we obviously can't route mail through the AWS email service.

This change uses a property to load a MonkeyRecorder, so an alternate can be used when SDB is not available/desired. This is critical for non-AWS cloud environments, where there may not be a natural equivalent for SDB.

[DEBUG] [org.gradle.configuration.project.BuildScriptProcessor] Timing: Running the build script took 3.716 secs [ERROR] [org.gradle.BuildExceptionReporter] [ERROR] [org.gradle.BuildExceptionReporter] FAILURE: Build failed with an exception. [ERROR] [org.gradle.BuildExceptionReporter] [ERROR] [org.gradle.BuildExceptionReporter] * What went wrong: [ERROR] [org.gradle.BuildExceptionReporter] nebula/plugin/netflixossproject/NetflixOssProjectPlugin : Unsupported major.minor version 51.0 [ERROR] [org.gradle.BuildExceptionReporter] [ERROR] [org.gradle.BuildExceptionReporter] * Try: [ERROR] [org.gradle.BuildExceptionReporter] Run with --stacktrace option to get the stack trace. [LIFECYCLE] [org.gradle.BuildResultLogger] [LIFECYCLE] [org.gradle.BuildResultLogger] BUILD FAILED [LIFECYCLE] [org.gradle.BuildResultLogger] [LIFECYCLE] [org.gradle.BuildResultLogger] Total time: 6.445 secs

java version "1.6.0_45" Java(TM) SE Runtime Environment (build 1.6.0_45-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

Hi,

We've made the following changes:

• CloudFormationChaosMonkey class to enable email notifications when no-suffix group names are used.
• Modified BasicChaosMonkey and BasicChaosEmailNotifier classes to enable global email notifications. eg. simianarmy.chaos.notification.global.receiverEmail = [email protected] is used to enable global email notifications. Added corresponding unit test.
• Refactored TestChaosMonkeyContext to remove duplicate code

Hi

When using cloud formation, the ASG created has a random string appended, such as:

foo-my-app-MyAutoScalingGroup-1727U49KMHGUT

We are currently using the opt in feature with Chaosmonkey to select a subset of our ASGs for Chaosmonkey to target.

So we put 'simianarmy.chaos.ASG.foo-my-app-MyAutoScalingGroup-1727U49KMHGUT.enabled =1' into the properties file.

However this can be a little problematic as occasionally we update the stack. This creates a new random string as a suffix to the ASG name and causes the opt-in configuration to fail to match the previously matched ASG.

Is there a way we can use some kind of pattern matching and if not could we submit a patch to solve this problem.

Dip

Hi Cory

I'm finally done with adapting ChaosMonkey to run against VSpehere. The configuration file for this is client.properties The basic strategy fot termination is to set a custom property in the VM and then reset it. This triggers a kickstart on reboot by our infrastructure code governing the VSphere installation. I'm pretty sure others will have their own strategies here in the future. I refactored the code so I could introduce the infastructure needed to make the client configurable in order introduce the VSphereClient. I also tried to make the change to the overall structure as small as possible. The persistence is still Amazon Simple DB. I may add MySQL support later, since this more how we roll here rather then using public clouds. I couldn't get my windows eclipse with jalopy to cooperate on the NewlineAtEndOfFile Rule, so I commented that out in codequality/checkstyle.xml It's probably best if you just ignore this change. I enabled the travisci build since it's free and a threeliner I added a "EventReport" to the context. I use that to log a final score after a monkey run on what VMs where terminated. This is pure convenience on my part since I find this easier to read in the logs. The test coverage on class VSphereServiceConnection is pretty low. This is the class that encapsulates the connection to vsphere. And the VSpehere API is pretty hard to unit test, I went as far as I deem feasible.

If you have any questions regarding my changes, do not hesitate to get back to me.

Cheers Ingmar

Before, the simianarmy.calendar.isMonkeyTime could have three states: true, false, or unset. If unset, the property is ignored; if true, it's always monkey time; if false, it's never monkey time.

Arguably, given the documentation, this is the correct behavior.

I am following the documentation to run simian army

Following the steps in https://github.com/Netflix/SimianArmy/wiki/Quick-Start-Guide#setup-simpledb-table & unleashing the money, instead of getting

Terminated i-8b55fbb8 from group monkey-target,

I am getting

2020-06-01 14:43:15.581 - INFO InstanceInSecurityGroup - [InstanceInSecurityGroup.java:153] Instance i-0fba7806cbf15b213 is in VPC and is ignored.

Hello, Your project uses some dependencies with CVEs. I found that the buggy methods of the CVEs are in the program execution path of your project, which makes your project at risk. I have suggested some version updates. See below for more details:

• Vulnerable Dependency: org.apache.httpcomponents : httpclient : 4.3

• Call Chain to Buggy Methods:

  • Some files in your project call the library method org.apache.http.impl.client.HttpClientBuilder.build(), which can reach the buggy method of CVE-2013-4366.

    • Files in your project: src/main/java/com/netflix/simianarmy/client/MonkeyRestClient.java
    • One of the possible call chain:

    org.apache.http.impl.client.HttpClientBuilder.build() [buggy method]
    

• Update suggestion: version 4.5.11 4.5.11 is a safe version without CVEs. From 4.3 to 4.5.11, 2 of the APIs (called by 2 times in your project) were modified.

# kubectl get pods -n chaos
NAME                                 READY   STATUS    RESTARTS   AGE
testforcw-test-test1530-v000-m8nhm   1/1     Running   0          122m

# chaosmonkey eligible testforcw chaos-account --cluster=testforcw-test-test1530 --region=chaos
testforcw-test-test1530-v000-m8nhm

# chaosmonkey terminate testforcw chaos-account --cluster=testforcw-test-test1530 --region=chaos
[ 8222] 2019/06/13 17:39:49 Picked: {testforcw chaos-account chaos test testforcw-test-test1530 testforcw-test-test1530-v000 testforcw-test-test1530-v000-m8nhm kubernetes}

but I still get error: "Failed to delete pod testforcw-test-test1530-v000-m8nhm in default" and the pod was not terminated and restart

does anyone have any ideal about this problem?

Travis are now recommending removing the sudo tag.

"If you currently specify sudo: false in your .travis.yml, we recommend removing that configuration"

Hi all, I am trying to use chaos monkey with spinnaker running on kubernates cluster. Spinnaker is running it's services separately in container .

I am trying to get chaos monkey support for spinnaker mentioned in below link:- https://netflix.github.io/chaosmonkey/How-to-deploy/

But settings.js file is not found in /var/www location.

I went inside deck container and changed setting.js file in /opt/deck/html (location inside container). But I still not chaos monkey enable feature in spinnaker UI.

I am using spinnaker version 1.7.6

Can anyone help me out?