Minor fix of 'expected' test assert order

Further, it seems that the CsvSortOptions and up-to-date CsvExternalSort as described in the readme are not in the latest 0.4.0 release version.

I'm trying to use the merge method with a GZIPOutput Buffered writer. The method mergeSortedFiles(BufferedWriter fbw, final Comparator cmp, boolean distinct, List buffers) looks perfect, but the dependency to BinaryFileBuffer makes it impossible to use. As BinaryFileBuffer is a non-public child class, you cannot use it externally and you cannot extend it either. Could you just make the BinaryFileBuffer a standard public class in its own file?

Thanks

Including new class to deal with CsvExternal sorting using apache commons csv

I made small changes into your logic to be able to work with csv files better.

I'm using org.apache.commons:commons-csv:1.9.0 Could we update it in externalsortinginjava ? So when I have commons-csv:1.9.0 running in runtime, I know that externalsortinginjava was compiled against 1.9 as well.

Hey, I found that the estimation of available memory uses the current heap space, instead of the maximum heapspace (ie. I got lots of little batch files even though I had gigabytes of heap space). This worked for me.

See http://stackoverflow.com/questions/12807797/java-get-available-memory

Hi,

Thanks for this code. I could sort billions of lines. However, I identified a tiny issue while sorting integers of variable length like below..

1000000000 is smaller than 100000001 100000010 is smaller than 10000001 100000020 is smaller than 10000002 and so on.,

This is because of r1.compareTo(r2) method call in defaultcomparator. When they are compared as strings, 1000000000 is indeed smaller than 100000001. But it's incorrect when it is an integer.

This PR checks whether input contain all numerical digits & sort them as integers.

Looking forward to hear from you.

I was a bit confused with the skipheader and numHeader in the the CSVSortOptions. Couldn't find the description in your API documentation. Can you please elaborate?

1. Is it possible to sort a CSV file based on multiple columns, say firstName, lastName, and dob? For example in case multiple records have the same firstName, those records will be sorted based on the lastName value. If lastName is also same, they will be sorted based on the dob value, And so on.

2. Can this method also be applied to sort fixed length files based on multiple columns?

If yes, can you point me to where can I find some samples of those?

CsvExternalSort.mergeSortedFiles ignores CsvSortOptions.numHeader which could trigger issues in the provided Comparator as it will try to compare header line with other lines. The issue only occurs if the batch has size > 1.

README example:

import com.google.code.externalsorting.CsvExternalSort;
import com.google.code.externalsorting.CsvSortOptions;

// provide a comparator
Comparator<CSVRecord> comparator = (op1, op2) -> op1.get(0).compareTo(op2.get(0));
//... inputfile: input file name
//... outputfile: output file name
//...provide sort options
CsvSortOptions sortOptions = new CsvSortOptions
				.Builder(CsvExternalSort.DEFAULTMAXTEMPFILES, comparator, 1, CsvExternalSort.estimateAvailableMemory())
				.charset(Charset.defaultCharset())
				.distinct(false)
				.numHeader(1)
				.skipHeader(false)
				.format(CSVFormat.DEFAULT)
				.build();

// next two lines sort the lines from inputfile to outputfile
List<File> sortInBatch = CsvExternalSort.sortInBatch(file, null, sortOptions);;
CsvExternalSort.mergeSortedFiles(sortInBatch, outputfile, sortOptions, true);

In the case where if (currentBlock.get() < blocksize) is false, the CSVRecord e is lost. You'll notice this as a missing row for each block.

My solution was to remove the conditional logic for processing e, and add the conditional logic only in the part where the block is processed.

Here is my updated code: ` try (CSVParser parser = new CSVParser(fbr, sortOptions.getFormat())) { parser.spliterator().forEachRemaining(e -> { if (e.getRecordNumber() <= sortOptions.getNumHeader()) { header[0] = e; } else { tmplist.add(e); currentBlock.addAndGet(SizeEstimator.estimatedSizeOf(e)); } if (currentBlock.get() >= blocksize) { try { files.add(sortAndSave(tmplist, tmpdirectory, sortOptions, header[0])); } catch (IOException e1) { logger.warn(String.format("Error during the sort in batch"),e1); //$NON-NLS-1$ } tmplist.clear(); currentBlock.getAndSet(0); }

		});
	}

`

I have a test that call ExternalSort.sortInBatch with a very small maxMemory - 100 bytes. When I call that method on a small file, ExternalSort.sortInBatch creates 5 temporary files, one of which is empty. I then call ExternalSort.mergeSortedFiles on the result of ExternalSort.sortInBatch. The resulting merged file is correct, however ExternalSort.mergeSortedFiles doesn't delete the empty file.

That seems to be caused by the empty file still being in use (I'm running on Windows). The code only adds non-empty files to the priority queue:

`

       public static long mergeSortedFiles(BufferedWriter fbw,
            final Comparator<String> cmp, boolean distinct,
            List<IOStringStack> buffers)  ...
            for (IOStringStack bfb : buffers) {
                    if (!bfb.empty()) {
                            pq.add(bfb);
                    }
            }

` Since the empty file is not on the queue, it's not being closed so it cannot be deleted.

There is probably more than one way to fix this issue - e.g., if a file is empty, don't create an IOStringStack for it. I'm not sure what's the best way.

Currently fbw.newLine(); is used exclusively, but I require the ability to use single line feed on windows so the files can interchanged with other systems, without changing the default system line separator.