Original issue 76 created by ClearTK on 2009-03-20T17:22:01.000Z:

We need to provide a mechanism for returning K-best results from a classifier for sequential tagging tasks. There are two primary drivers for this. 1) a viterbi-style search through the space of possible solutions for a sequential tagging task such as part-of-speech tagging or chunking is very common because it improves the accuracy of the top choice which is returned. 2) It is often the case that you want to pass along the K-best sequences to downstream components. Right now we assume that a non-sequential tagger performing a sequential tagging task can return the best result by simply making the best decision it can at each decision point and returning the resulting sequence.

I was just reading the Ratnaparkhi parsing paper which describes the basic algorithm that the OpenNLP parser is based on. Here is an excerpt:

"It should be emphasize that if K > 1 (beam size), the parser does not commit to a single POS or chunk assignment for the input sentence before building constituent structure. All three of the passes described in section 2 are integrated in the search, i.e. when parsing a test sentence, the input to the second pass consists of K of the best distinct POS tag assignments for the input sentence."

Reimplementing this parser in ClearTK would be really difficult right now. We should think about how to provide search capability for sequential tagging tasks and a mechanism to "return" the K-best sequences. The former may not make sense for all non-sequential classifiers (i.e. LIBSVM doesn't provide probabilities with a classification) but should be quite straightforward for e.g. Maxent.

This issue was briefly raised in # 57 but was not addressed there because we used that issue to address a related problem of providing OutcomeFeatureExtractor.

Original issue 228 created by ClearTK on 2011-02-07T21:14:35.000Z:

So I've been playing around with the evaluation package, and it feels pretty clunky to me in a number of ways. I don't like the idea of labeling it as 0.9.9 (which is what it will get if it stays in cleartk-ml) because I think that may give the impression that the API are pretty much the way we want them. Could we move this out into a new module, say cleartk-ml-eval, and label it with 0.5.0 or 0.7.0 or something a little less committal than 0.9.9?

Some things that I think are clunky:

• I don't like that EngineFactory forces you to use Descriptions. In many cases, it may be possible to use a single AnalysisEngine for both data writing and classifying by simply flipping the isTraining flag (e.g. via a few setConfigurationParameterValue calls and a reconfigure call). But to support this, the EngineFactory methods would need to return AnalysisEngines instead of AnalysisEngineDescriptions, so that you could have a single instance and just reconfigure it when the interface methods were called.
• There aren't any default implementations of anything, so it makes it hard to be confident we have the abstraction right. Some things I think we should implement before we declare the APIs satisfactory:
  • A FilesCorpusFactory that takes a list of training files and a list of testing files (to be read in as plain text using a FilesCollectionReader) and fills in all the CorpusFactory methods using this information
  • An AnnotationSpanEvaluationFactory that takes an annotation type, and compares the spans of the gold annotations of this type to the model annotations using precision, recall and F-measure. And probably similarly for an AnnotationAttributeEvaluationFactory using accuracy.

Don't get me wrong - I don't think the evaluation package is horribly broken or anything. I just think we should label it as something less than a 0.9.9 release until we've pushed its boundaries a bit.

Original issue 55 created by ClearTK on 2009-02-18T17:54:29.000Z:

[Steve]

Consider this scenario:

• I'm working on a document classification task where I want to combine, say, word-based features with citation-based features.
• I want to apply Euclidean normalization to each group of features, that is, I want to normalize the word-based features separately from the citation-based features.

How should I go about this?

• Normalization is currently done in FeaturesEncoders, but FeaturesEncoders just see an Iterable of Features - they have no concept of groups.
• I know which features are in which groups in the AnnotationHandler, so maybe I should do the normalization by hand there? I'd basically have to re-implement FeatureVector.l2Norm() to work over Feature objects though...

Anyone have any better ideas?

• [x] Pull UIMA and uimaFIT versions up to 3.3.0
• [x] All tests are successful
• [x] Odd problem with uima.tcas.Annotation not being in the type system (slot 21 = null) in TimeML module

Original issue 226 created by ClearTK on 2011-02-03T18:31:48.000Z:

[Lee]

I was thinking since I have to do it anyway for my work, I should go ahead and write a CleartkMultiAnnotator class. The question is what can one reasonably expect from such a class? It looks like CleartkAnnotator doesn't provide much save for initialization of the classifier and data writer, and I was thinking CleartkMultiAnnotator would just provide help in dealing with collections of classifiers/ data writers. Based on conversations with Philip and my own needs, I was thinking that the pieces that differentiate CleartkMultiAnnotator from CleartkAnnotator would look something like this:

public abstract class CleartkAnnotator<KEY_TYPE, OUTCOME_TYPE> extends JCasAnnotator_ImplBase implements Initializable {

 protected Map<KEY_TYPE, Classifier<OUTCOME_TYPE> classifiers;
 protected Map<KEY_TYPE, DataWriter<OUTCOME_TYPE> dataWriters;

 // Retrieves or creates classifier as needed
 getClassifier(KEY_TYPE key);

// Retrieves or creates dataWriter as needed
getDataWriter(KEY_TYPE key);

}

Can you think of anything else that would be useful? I know when Philip implemented something similar for his own purposes he had an output directory configuration parameter so that he could squirrel away the classifiers and data writers in jar files.

[Steve] Thanks, this sounds like a great contribution!

This looks like basically what I had in mind, except I would just drop the generics for KEY_TYPE and use String everywhere. Then you can have a String[] PARAM_CLASSIFIER_NAMES @ConfigurationParameter, and the DataWriters can all delegate to PARAM_OUTPUT_DIRECTORY/name. Within those subdirectories, you should be able to package and unpackage jars using the usual ClearTK machinery.

By the way, I think you probably don't need the getter methods if the Maps are protected. No one but subclasses should really be asking for the DataWriters and Classifiers here, right?

[Philip] The getters can be protected. In my solution, I used getClassifier(keyType) to get a classifier if it was already available. If not, then it would initialize the classifier. I like doing this because you may not need every classifier that's been created for/from the training data when you are running it on your evaluation data because you may (perhaps inadvertently) have some key types that are very sparse and never appear in your evaluation data.

I agree that KEY_TYPE may be unnecessary and String may suffice. Although, I can see that a boolean would be quite reasonable for some scenarios.

I would rather not use the word "key" or "type" for the naming the key of the map as they are both overloaded terms. Maybe "name"?

Finally, it would be nice if we were able to abstract out the output directory as a parameter to this analysis engine as we do with CleartkAnnotator. I don't want to wire in our jar-based solution into this new class.

[Lee] I'm leaning toward just using String like Steve suggested. If someone really needed the boolean condition they could just create appropriate strings to describe their classifiers.

As per the getters, I go back and forth on whether or not they are needed. They are handy for the on-the-fly creation like Philip described, but they also muddle the semantics when using these classifier/datawriter maps. It's not clear when writing an extension of this proposed class whether one should use the maps directly or should only access them through the getters.

Lastly, does anyone have a good name for this class?

Original issue 197 created by ClearTK on 2011-01-07T21:36:41.000Z:

so - it seems like it's time to get a continuous integration server set up for this project so that we know that the code is compiling and tests are running. This project is already too big to be doing this manually and assuming that we are staying up-to-date. I have never done this before but am willing to help out with this anyways.

I don't know much about the tooling space for this - but my understanding is that Hudson is popular and easy to use. It's what Richard uses for uimaFIT. He might be some to ask. He gets cobutera test coverage reports generated too.

Original issue 57 created by ClearTK on 2009-02-21T00:36:40.000Z:

I think we have a problem in how we are thinking about using non-sequential classifiers for sequential tagging tasks. For starters, the notion that you are going to do separate feature extraction for the sequential case vs. the non-sequential case is just silly. The only difference between the two is that in the latter case you want add features based on the classifications of the previous instances. It would be easy enough to provide a generic wrapper for all non-sequential classifiers so that we aren't constantly breaking our code up into "sequential" and "non-sequential" mode.

Equally silly, though, is the notion of doing a sequential tagging task with a non-sequential classifier without a viterbi beam search of some kind. This is pretty basic.

I suggest that we create a sequential classifier that wraps any non-sequential classifier so that it:

• does something sensible with converting previous classifications into features
• provides a mechanism to specify a viterbi-style search of some kind. We should be able to come with a handful (one-per sequential classifier?) of these to cover all of our sequential classifiers.

Original issue 181 created by ClearTK on 2011-01-05T18:13:00.000Z:

One thing that is going to get annoying very quickly about the current arrangement is that building all of the projects requires some effort. It sure would be nice if we had a parent pom.xml file that we could run to compile, test, and possibly install all of the projects. I think uimaFIT does a nice job with this and we can model our approach on that. Any thoughts?

Original issue 218 created by ClearTK on 2011-01-31T09:22:46.000Z:

So, assuming we do the refactoring of cleartk-ml proposed in Issue 217, I think there is no longer any use for ClassifierFactory. It was originally introduced for Issue 135, where the problem was that Classifiers had to take a JarFile in the constructor, and Philip needed to create a test Classifier without a JarFile. That problem goes away with the refactoring in Issue 217 because Classifiers can now have any constructor parameters they want.

But why not just leave well enough alone? ;-) Because ClassifierFactory prevents us from easily using models as Resources on the classpath. JarClassifierFactory (the only real ClassifierFactory we have) currently does this:

InputStream stream;
try {
  stream = new URL(this.classifierJarPath).openStream();
} catch (MalformedURLException e) {
  stream = new FileInputStream(this.classifierJarPath);
}
stream = new BufferedInputStream(stream);
JarInputStream modelStream = new JarInputStream(stream);
ClassifierBuilder<?> builder = JarClassifierBuilder.fromManifest(modelStream.getManifest());
try {
  return superClass.cast(builder.loadClassifier(modelStream));
} finally {
  stream.close();
}

It would really be nice if before we try the FileInputStream we could try something like:

stream = annotatorClass.getResourceAsStream(this.classifierJarPath)

But this won't work because ClassifierFactory objects don't have access to the annotator class. If we get rid of ClassifierFactory and just move the bit of code above to CleartkAnnotator, we can easily make this work using this.getClass().

What do you think?

Original issue 7 created by ClearTK on 2008-12-05T18:21:14.000Z:

The following unit tests have data in the text of the .java file that should not be redistributed (e.g. song lyrics, treebank data, etc.) These unit tests are only available on the old repository and either need to be refactored or replaced.

Parent directory /ClearTK/test/src/

Affected Files org/cleartk/classifier/encoder/features/string/StringFeatureEncoderTests.java

Original issue 3 created by ClearTK on 2008-12-05T17:48:34.000Z:

Please use labels and text to provide additional information.

There is currently no documentation for the different feature extractors. We need to both complete the JavaDocs, and, more importantly, give a more tutorial-like overview of what is available and how it can be used.

previously feature name from sub-extractors in ClearTkExtractor were silently discarded, for reasons I can't discern (presumably accidental?). Compare with the out-of-bounds features a few lines below, which preserve the name from the sub-extractor. Please advise if this was in fact a principled decision (and I'll abandon the PR). If not, I can provide other supporting changes you need for this very simple patch

In the tutorial the class SimpleFeatureExtractor is part of the sample code but in the code (at least in the newest version) such a class does not seem to exist!

The parser adds two TopTreebankNode for each sentence and also for each TerminalTreebankNode annotation, it creates one extra TreebankNode annotation.