This is a huge one. Thanks for reviewing it in advance. You might want to examine commit by commit and also add &w=1 or ?w=1 to the URL to ignore white spaces in the diff due to reformatting.

Aside a few minor optimizations mentioned in the commits, the major changes are:

1. Add SparseLabeledPoint for space efficient representation of feature vector
2. Add validation split into model trainer for better understanding of model convergence

Sparse Representation

It shall be noted this implementation is not ideal in any sense but as a compromise to best fit in Aerosolve existing architecture without rewriting all the transforms.

SparseLabeledPoint store feature vector in the form of

    public final int[] indices;
    public final float[] values;
    public final int[] denseIndices;
    public final float[][] denseValues;

where float and string features are stored in indices,values and dense features are stored in denseIndices,denseValues. The index of each feature is backed by featureIndexer in AdditiveModel of type Map<String, Map<String, Integer>>.

So instead of

Map(
  'family1' -> Map(
    `feature1' -> value1,
    `feature4' -> value2,
   ...),
  ...
)

we store features as

indices: [1, 4, ...]
values: [1, 2, ...]

Furthermore, to support efficient model updates with sparsely represented feature vector, we collect all features into Function[] weightVector where features are directly indexed by their corresponding index in the featureIndexer.

Validation Split

Because SGD loss is rather jittery in short window frame and we perform averaging of model weights among other regularizations during each iterations, training loss appears to be a poor proxy to model convergence. This PR adds an optional isTraining function to split training data into training and validation. After each iteration, validation loss is calculated without updating the model itself. This gives a much clearer guidance to both learning rate specification, its appropriate decay rate, early stopping, and if model of any iterations might have been overfitted. The final model is picked from the iteration with lowest validation loss instead.

Speed Optimization

With validation loss, we were able to experiment with parameters and sampling technique with much better confidence. This PR also adds option to sample_by_partition which take a sample of partitions. This played nicely when there are lots of partitions and data is randomly distributed when they are pulled. It appears model can converge much faster when a smaller sample is chosen and run with a few more iterations.

@deerzq @jq @timyitong

p.s. I will update hackpad once this is merged to outline a few techniques and findings.

The training and evaluation dataset is usually different and in a lot of the cases need to be separated by time not by random sample. e.g. training is data from 2015 and eval is data from 2016.

The current Aerosolve config doesn’t support adding evaluation dataset. This PR is to add this option to the user. As suggested by @deerzq , I refactored the makeTraningrun to make it more generic to make both the training and evaluation dataset(optional)

@jq @saurfang

• Implement AdditiveModel which takes more flexible forms of functions. In addition to Spline, it also takes linear functions.
• Add an abstract function class, Spline and Linear all inherit this class.
• Add unit test, fix a small bug in LinearModelTest.java
• implement additive model trainer

@hectorgon

This started as an attempt to refactor AdditiveModelTrainer to make it much faster. However I have come to the conclusion that the trainer implementation is not the bottleneck but rather the inefficient Example class is the culprit for the poor performance. It roughly takes 5~10x time to load Example data than the actually optimization routine at each iteration.

Before moving to a more efficient representation of feature vectors, I want to merge in the initial refactoring, clean up and documentation I added for the analysis. There should be no performance impact and logic changes for this PR.

@jq @deerzq @yolken

Let me know if you have any comment or suggestions on the documentation. You will probably find it easier to review commit by commit.

This is nowhere close to being finished, just wanted to push it up so that we can more easily discuss the design of the transform, whether an explicit output should be generated, and whether repeated tokens are desired (since the output is currently a set). We can talk about this tomorrow morning. @hectorgon @yolken

eval_test was an option in income pipeline but not part of Genericpipeline. The goal is to be able to evaluate the performance of model on a separate test data other than hold_out data as specified in eval_test.

Example usage:

make_testing {
  hive_query : ${generic_hive_query}" from "${training_table}${testing_filter}
  num_shards : 20
  output : ${testing_data}
}

eval_test {
  input : ${testing_data}
  subsample : ${eval_subsample}
  bins : 100
  model_config : ${model_config}
  is_probability : false
  is_regression : false
  metric_to_maximize : "!HOLD_F1"
  model_name : ${model_name}
}

to: @jq cc: @deerzq

Content: This is just a quick foundation of what I envisioned a float cross float transform might look like (essentially a MoveFloatToString transform followed by a StringCrossFloat transform, but without deleting the original float feature or creating an intermediate string feature which would occur if these two transforms were applied back to back). The ability to specify which keys to cross could be added as well instead of crossing the entire family.

Please let me know what you think / if this is what you need. Once we're on the same page I'll add comments and tests. Thanks!

While trying to use aerosolve for kaggle airbnb competition, I noticed in com.airbnb.aerosolve.core.transforms.Transformer.java that only string features are being copied from context to examples. Is this behavior coded on purpose ? In general, is the differentiation between context and item features only for memory optimization or does it impact the algorithms ranking task ? Thanks in advance for any advices to help.

What

Optimize the ROC-AUC calculation for dataset with uneven distributed scoring. AUC is calculated by first divide the whole area to 100 small vertical strips, approximate the areas for each strips and sum them up. In our existing implementation, we first get the min and max of all scores(probabilities) and evenly divide the space to 100 intervals, no matter how many records are in each interval.

This will make the AUC inaccurate for score distribution that is not uniformly distributed. During my experiment with short term labeling, our existing implementation returned AUC of 0.74. I take a sample offline and use sklearn to calculate, the result is 0.87. With this PR, the result is 0.85, which is far more reasonable.

For our base models, the error introduced by this approximation is much smaller.

How

Divide the score space unevenly according to the true distribution. More specifically, first sort the records by score, use zipWithIndex to get the rank of score, then use rank/bucketSize to get the bucket.

I also changed the bucket number from 100 to 10000, turns out not much difference. But I think 10000 is better since we do want more accurate estimation and can afford the computation resources.

@deerzq @saurfang

Revert code clean up that merges the evaluation of records from RDD and List. Instead of merging the code path using List, perform evaluation on RDD instead to handle large dataset.

There is no reliable way to get a SparkContext especially with the way we set up our tests. Since there is really no need to run evaluation on List of records and lists can always be parallelize with single function call, we are removing evaluation on list altogether.

Related to #237 #231

@jq @deerzq @luanjunyi

Adds an updateBounds function to KDTree so boundary of KDTree node can be updated after a KDTree model is trained. Split is not affected by this update. The boundary information is purely informational.

Reformatted code and applied static code analysis suggestions.

@jq @spencerde

Hi, there!

We are a research team working on third-party library analysis. We have found that some widely-used third-party libraries in your project have major/critical bugs, which will degrade the quality of your project. We highly recommend you to update those libraries to new versions.

We have attached the buggy third-party libraries and corresponding jira issue links below for you to have more detailed information. We have analyzed the api call related to the following libraries and found one library that is using the API call that might invoke buggy methods in the library of the history.

1. commons-codec commons-codec version: 1.4 API call in your project:org.apache.commons.codec.binary.Base64.setInitialBuffer(byte[],int,int)

Jira issues: Base64InputStream#read(byte[]) incorrectly returns 0 at end of any stream which is multiple of 3 bytes long version:1.4 ArrayIndexOutOfBoundsException when doing multiple reads() on encoding Base64InputStream version:1.4 Base64 encoding issue for larger avi files version:1.4 org.apache.commons.codec.net.URLCodec.ESCAPE_CHAR isn't final but should be version:1.2;1.3;1.4 org.apache.commons.codec.language.RefinedSoundex.US_ENGLISH_MAPPING should be package protected MALICIOUS_CODE version:1.4 org.apache.commons.codec.language.Soundex.US_ENGLISH_MAPPING should be package protected MALICIOUS_CODE version:1.4 Caverphone encodes names starting and ending with "mb" incorrectly. version:1.4 All links to fixed bugs in the "Changes Report" http://commons.apache.org/codec/changes-report.html point nowhere; e.g. http://issues.apache.org/jira/browse/34157. Looks as if all JIRA tickets were renumbered. version:1.1;1.2;1.3;1.4 Regression: Base64.encode(chunk=true) has bug when input length is multiple of 76 version:1.4 DigestUtils: MD5 checksum is not calculated correctly on linux64-platforms version:1.3;1.4 new Base64().encode() appends a CRLF; and chunks results into 76 character lines version:1.4 Base64 encode() method is no longer thread-safe; breaking clients using it as a shared BinaryEncoder version:1.4 Base64 default constructor behaviour changed to enable chunking in 1.4 version:1.4 Base64InputStream causes NullPointerException on some input version:1.4 Base64.encodeBase64String() shouldn't chunk version:1.4 2. org.apache.commons commons-lang3 version: 3.4 Jira issues: TypeUtils.ParameterizedType#equals doesn't work with wildcard types version:3.3.2;3.4 DateUtilsTest.testLang530 fails for some timezones version:3.4 StringUtils.stripAccents from "Ł" and "ł" version:3.4 No release notes for version 3.4 version:3.4 JsonToStringStyle doesn't handle chars and objects correctly version:3.4 ReflectionToStringBuilder doesn't throw IllegalArgumentException when the constructor's object param is null version:3.4 StrLookup.systemPropertiesLookup() no longer reacts on changes on system properties version:3.4 StringUtils#capitalize: Javadoc says toTitleCase; code uses toUpperCase version:3.4 Multiple calls of org.apache.commons.lang3.concurrent.LazyInitializer.initialize() are possible version:3.4;3.5 EnumUtils *BitVector issue with more than 32 values Enum version:3.4 StringUtils#equals fails with Index OOBE on non-Strings with identical leading prefix version:3.4 There are no tests for CharSequenceUtils.regionMatches version:3.4 ArrayUtils.removeAll(Object array; int... indices) should do the clone; not its callers version:3.4 TypeUtils.isAssignable throws NullPointerException when fromType has type variables and toType generic superclass specifies type variable version:3.4 FastDateFormat does not support the week-year component (uppercase 'Y') version:3.4 ordinalIndexOf("abc"; "ab"; 1) gives incorrect answer of -1 (correct answer should be 0) version:3.4 Fix implementation of StringUtils.getJaroWinklerDistance() version:3.4 parseDateStrictly does't pass specified locale version:3.4 ClassUtils.getClass(ClassLoader; String) fails for "void" version:3.4 NumberUtils.isNumber bug version:3.4 FastDateFormat doesn't respect summer daylight in localized strings version:3.4 StringUtils#normalizeSpace does not trim the string anymore version:3.4 DiffBuilder: Add null check on fieldName when appending Object or Object[] version:3.4 FastDatePrinter Memory allocation regression version:3.4 SerializationUtils.ClassLoaderAwareObjectInputStream should use static initializer to initialize primitiveTypes map. version:3.2;3.3;3.4 NumberUtils.isNumber and NumberUtils.createNumber resolve inconsistently version:3.4 ArrayUtils.contains returns false for instances of subtypes version:3.4 CompareToBuilder.append(Object;Object;Comparator) method is too big to be inlined version:3.4 StrBuilder#replaceAll ArrayIndexOutOfBoundsException version:3.2.1;3.4;3.5 NumberUtils#createNumber() returns positive BigDecimal when negative Float is expected version:3.x

Sincerely~ FDU Software Engineering Lab Marth 14th,2019

:core:compileJava (Thread[Task worker for ':',5,main]) started.

Task :core:compileJava Putting task artifact state for task ':core:compileJava' into context took 0.0 secs. Executing task ':core:compileJava' (up-to-date check took 0.006 secs) due to: Task has failed previously. All input files are considered out-of-date for incremental task ':core:compileJava'. Compiling with JDK Java compiler API. /Users/maximebodereau/Documents/Projects/Ux AI/aerosolve/core/src/main/java/com/airbnb/aerosolve/core/util/Weibull.java:13: error: cannot find symbol public WeibullBuilder defaultBuilder() { ^ symbol: class WeibullBuilder location: class Weibull 1 error

:core:compileJava (Thread[Task worker for ':',5,main]) completed. Took 0.262 secs.

FAILURE: Build failed with an exception.

• What went wrong: Execution failed for task ':core:compileJava'.

java.lang.NoSuchFieldError: pid

Hi, I'm a Stanford MS student trying to run the image impressionism and income classification demos. When running gradle shadowjar --info, I get multiple errors of the following type during the execution of the task :core:compileJava:

/Users/ei5h4/Documents/aerosolve/core/build/gen-java/com/airbnb/aerosolve/core/ModelRecord.java:1075: error: method hashCode in class Object cannot be applied to given types;
      hashCode = hashCode * 8191 + org.apache.thrift.TBaseHelper.hashCode(featureWeight);
                                                                ^
  required: no arguments
  found: double
  reason: actual and formal argument lists differ in length

My thrift version is 0.10.0. I tried downloading and installing an older version of thrift (0.9.0) from source since this demo is old and might rely on an older thrift (just a hypothesis). But that turned out to have some roadblocks as well since the older thrift uses some C code namespace tr1 that is no longer supported by C++11 on my OSX El Capitan. So I couldn't verify if thrift is the issue or something else. Basically I thought the hashCode function in the error above might have a changed signature from 0.9.0 to 0.10.0.

I think anyone else attempting to build the demo will run into this issue as well. Really hope to get this running on my machine soon. Aerosolve is super exciting!