Hi,

I tried out your Shapiro-Wilk implementation as I needed to calculate some values for a paper-submission. I did cross reference it with the Real-Statistics Excel Plugin (http://www.real-statistics.com/) as well as several online tools that allow use of Shapiro-Wilk.

If you run it with the following values: 488.0, 486.0, 492.0, 490.0, 489.0, 491.0, 488.0, 490.0, 496.0, 487.0, 487.0, 493.0 The results should be: W -> 0.944486 P -> 0.55826969...

However the P value is 0.44173030948

The offending line is here: https://github.com/datumbox/datumbox-framework/blob/957560901f3c87d3e9f6760263644a4d70b0a3b8/datumbox-framework-core/src/main/java/com/datumbox/framework/core/statistics/nonparametrics/onesample/ShapiroWilk.java#L257

It should actually be m-y pw=ContinuousDistributions.gaussCdf((m-y)/s)

So, as exposed in our discussion in the pull request, here is a minimal piece of code that shows when the thing throws an exception:

    public void testClassifier() {
        RandomGenerator.setGlobalSeed(42L);
        Configuration configuration = Configuration.getConfiguration();
        InMemoryConfiguration memConfiguration = new InMemoryConfiguration();
        final File f = new File(Datumbox.class.getProtectionDomain().getCodeSource().getLocation().getPath());
        memConfiguration.setDirectory(f.getAbsolutePath());
        configuration.setStorageConfiguration(memConfiguration);

        // List of positive and negative sentences, for training
        List<String> positives = new ArrayList<>();
        List<String> negatives = new ArrayList<>();
        positives.add("the rock is destined to be the 21st century's new \" conan \" and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .");
        positives.add("the gorgeously elaborate continuation of \" the lord of the rings \" trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .");
        positives.add("effective but too-tepid biopic");
        negatives.add("simplistic , silly and tedious .");
        negatives.add("it's so laddish and juvenile , only teenage boys could possibly find it funny .");
        negatives.add("exploitative and largely devoid of the depth or sophistication that would make watching such a graphic treatment of the crimes bearable .");

        // Construct the training parameters for the classifier
        TextClassifier.TrainingParameters trainingParameters = new TextClassifier.TrainingParameters();
        trainingParameters.setNumericalScalerTrainingParameters(new StandardScaler.TrainingParameters());
        trainingParameters.setCategoricalEncoderTrainingParameters(new CornerConstraintsEncoder.TrainingParameters());
        trainingParameters.setFeatureSelectorTrainingParametersList(Arrays.asList(new ChisquareSelect.TrainingParameters()));
        trainingParameters.setTextExtractorParameters(new NgramsExtractor.Parameters());
        trainingParameters.setModelerTrainingParameters(new BernoulliNaiveBayes.TrainingParameters());

        // Construct list of records form lists of positives/negatives
        AbstractTextExtractor textExtractor = AbstractTextExtractor.newInstance(trainingParameters.getTextExtractorParameters());
        List<Record> records = new ArrayList<>();
        for (String positive: positives) {
            AssociativeArray xData = new AssociativeArray(textExtractor.extract(StringCleaner.clear(positive)));
            records.add(new Record(xData, "positive"));
        }
        for (String negative: negatives) {
            AssociativeArray xData = new AssociativeArray(textExtractor.extract(StringCleaner.clear(negative)));
            records.add(new Record(xData, "negative"));
        }

        // Construct training dataframe
        Dataframe trainingData = new Dataframe(configuration);
        for (Record r: records)
            trainingData.set(trainingData.size(), r);

        // Construct and train the classifier
        TextClassifier classifier = MLBuilder.create(trainingParameters, configuration);
        classifier.fit(trainingData); // Here, you can follow the trace to see the problem
    }

Sometimes when creating a FlatDataList null values are added to the List, and consequently, when the library tries to retrieve those values it fails with an exception. For example, the method variance() in core.statistics.descriptivestatistics.Descriptive, fails in line 293 (double v = it.next()) when flatDataCollection has some of his values nulled.

Hello I'm new to machine learning.Datumbox is my first ML framework I'm working with But I did not find any documentation on setting config properties to read trainingdata from Disk, please share a code example of reading training data sets from the disk & setting up config properties.

How do I serialize a dataframe efficiently (records in bulk) onto the disk, with mapdb. My use case is, I have a large dataset for text classification, it takes a long time to deserialize and tokenize the text. I want to try out multiple experiments, without having to do tokenization again to convert to Record instances.

Hi, I'm new in machine learning and I'm building a small project for analyzing data with "Sentiment Analysis, Content Readability, Content Quality, Adult Content and Spam Detection" features. so I wanted a small help to use Pre-trained Models with Framework.

Bumps junit from 4.13 to 4.13.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hello,

I need to classify the text. I already tried your awesome example based on MultinomialNaiveBayes classifier https://github.com/datumbox/datumbox-framework-examples/blob/develop/src/main/java/com/datumbox/examples/TextClassification.java

I'd like to also test another algorithm - SVM. Could you please show an example how to transform the mentioned sample class in order to use SVM?

Is it as simple as changing to:

from

trainingParameters.setModelerTrainingParameters(new MultinomialNaiveBayes.TrainingParameters());

to:

trainingParameters.setModelerTrainingParameters(new SupportVectorMachine.TrainingParameters()); or do I need to change something else also?

Also, can I use the same text files for my SVM model that I used previously for MNB?

Hi I created my own sentiment model for twitter. For preparation of model, I collected 4mb data files of pos,neg,neu.

when I created the model the size of the model is increased to 191mb and when I tried out sentiment the by loading that 191 mb model, It is taking 40 sec time to give the output.

In order to load that 191mb model in tomcat, I increased java -xmx value to 2gb. I want to know why it is taking 40 sec to give the output & when I created model I did not find any fs0 folder. Does it make any difference ?

please give suggestions on following : Do I need to look at hardware or Do I need to decrease the size of raw files and create the model ? when I check your twitter-sentiment model in datumbox zoo,it as low size model (just 1.5 mb) not like me

I just need the Naive Bayes Classifier, and wanted to know if there are any imports or codes (haven't check tho) that would prevent it from working on Android (Dalvik or ART Java VMs)

Does Datumbox support cross-validation internally to tune the parameters? I see in this post (http://blog.datumbox.com/how-to-build-your-own-twitter-sentiment-analysis-tool/) you talk about a 10 fold cross validation. Do we need to do it on our own? I could not find any example for such. Currently, what is the best way for tuning parameters in Datumbox.

I am using the Cross Validation to estimate the performance for my model, right now the way I am using it is ClassificationMetrics vm = new Validator<>(ClassificationMetrics.class, configuration).validate(new KFoldSplitter(10).split(trainingDataframe), new MultinomialNaiveBayes.TrainingParameters());

in the com.datumbox.framework.core.machinelearning.common.abstracts.algorithms.AbstractNaiveBayes, I see there's a setLogPriors function which can probably be used to tune the model. (I want to create a DET graph for the model performance, by playing around with the prior probability). Is there a way to set the prior probability of different labels for cross validation? Thanks.