I want to use the FastCSV for implementing a large file CSV editor, using JavaFX. I plan to parse only separate rows, at render time. This means I need to use the CSV parser to parse a String line at a time. Is it possible to have something like this?

CSVParser parser = new CSVParser();
parse.parse( line98);
parser.reset();
parser.parse(line99)

I mean I parse a line, then I may parse another, etc. For getting optimal memory usage, I would instantiate the CSVParser only one time.

Thank you for your work on this great product! It's proven performance has significantly improved the performance of our application.

We had initially been using version 1.0.4, which greatly improved the performance of our CSV parsing (over commons-csv which we had been previously using). We recently tried upgrading to version 2.1.0. I like the new API, however, we noticed that there was a significant performance degradation over 1.0.4. We have a little bit of a unique data format that we deal with, which involves an embedded CSV list within a CSV column. This is how our data looks:

NAME,NUMBER,WIDGETS_LIST
john doe,123456,"""thequickbrownfoxjumpedoverthelazydog"""
john smith,7890123,"""thequickbrownfoxjumpedoverthelazydog1"",""thequickbrownfoxjumpedoverthelazydog2"""

The WIDGETS_LIST column is a variable length list that is formatted as an embedded csv string. Each item in the list is usually around 200 characters long.

With fastcsv 1.0.4 we would parse the data with code like this:

class Parser {

  Client parseCsv(Path file) {
    List<Client> clients = new ArrayList<>();
    CsvReader csvReader = new CsvReader();
    try(var parser = csvReader.parse(file, StandardCharsets.UTF_8)) {
      CsvRow row;
      while( (row = parser.nextRow()) != null) {
        String name = row.getField(0);
        String number = row.getField(1);
        List<String> widgets = parseWidgets(row.getField(2));
        clients.add(new Client(name, number, widgets));
      }
    }
    return clients;
  }
  List<String> parseWidgets(String data) {
    CsvReader csvReader = new CsvReader();
    CsvParser parser = csvParser.parser(new StringReader(data));
    CsvRow row = parser.nextRow();
    return row != null ? List.copyOf(row.getFields()) : List.of();
  }

}

With fastcsv 2.1.0 we parse with code like this:

class Parser {

  Client parseCsv(Path file) {
    try(var parser = CsvReader.builder().build(file)) {
      return parser.stream()
         .map(row -> {
             String name = row.getField(0);
             String number = row.getField(1);
             List<String> widgets = parseWidgets(row.getField(2));
             return new Client(name, number, widgets));
         })
         .toList();
  }
  List<String> parseWidgets(String data) {
    return CsvReader.builder().build(data)
        .stream().flatMap(r -> r.getFields().stream())
        .toList();
  }

}

Very surprisingly, the fastcsv 2.1.0 code takes around twice as long to parse the CSV data than version 1.0.4. It seems to be related to the embedded CSV string since for other data without the embedded CSV, 2.1.0 is actually faster than 1.0.4. However, I cannot figure out why the embedded CSV is causing such a significant slow down. To get meaningful performance results we benchmarked with a CSV file containing about 1 million rows, and processed the same file 10 times per run.

Additional context Java distribution and version to be used (output of java -version).

openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment Temurin-17.0.2+8 (build 17.0.2+8)
OpenJDK 64-bit Server VM Temurin-17.0.2+8 (build 17.0.2+8, mixed mode, sharing)

I need a way to enforce per-field quoting in order to generate CSV for PostgreSQL COPY statement because quoted empty string is treated as NULL, while totally empty field is treated as an empty string:

1,,3 ---> ""
1,"",3 ---> NULL

The current CsvAppender API doesn't support such behavior. Possible solutions:

• add an additional flag to appendField
• add a new method appendDelimitedField
• make alwaysDelimitText field mutable so that the consumer can turn it off/on before appending the specific field

Many Writer implementations are already buffered and fast enough. Also, many use cases start with an already existing BufferedWriter or similar, adding an extra layer only adds another temporary buffer copy. For example, you may have code like this working for different uses cases:

GZIPOutputStream zout = new GZIPOutputStream(
new CipherOutputStream(new FileOutputStream(file), c));
		
return new BufferedWriter(new OutputStreamWriter(zout, "utf-8"));

Assuming you can't change the code above or you don't want to be forced to do this:

if (usingFasctCsv)
   return new OutputStreamWriter(zout, "utf-8")
else 
   return new BufferedWriter(new OutputStreamWriter(zout, "utf-8"));

Can you add some way to construct an appender without wrapping the writer? Or I will try a pull request...

If a field at the end of a row was quoted but empty, it was silently dropped. For example, take this CSV:

"foo",""

This would only end up having 1 field in the resulting row instead of the expected 2. This PR fixes this bug.

I think that I found a bug inside the class CsvWriter:

"Caused by: java.lang.NoSuchMethodError: No virtual method toPath()Ljava/nio/file/Path; in class Ljava/io/File; or its super classes (declaration of 'java.io.File' appears in /system/framework/core-oj.jar)
                                                                             at de.siegmar.fastcsv.writer.CsvWriter.append(CsvWriter.java:148)"

I doesn't work on android 7.0 but it works on android 8.0, same phone. I already tried forcing Android Studio working with Java VERSION_1_8 and VERSION_1_7 but it's still the same.

Maybe I'm missing something obvious .. I want to simply add a row of data to a CSV file that already exists on disk.

Using CsvWriter and writeRow does not create a row at the bottom of the file. I noticed things changed in the version overhaul, and the CsvAppender and appendLine stuff is gone.

So, using the CsvWriter how can you open an existing CSV file and add a row of new data?

Describe the bug I am trying to read a file having size of approx 340 MBs. After I reach line number 809, I get this error: Caused by: java.io.IOException: Maximum buffer size 8388608 is not enough to read data

To Reproduce Try reading a csv with 800k rows.

Code:

CsvReader reader = CsvReader.builder()
            .fieldSeparator('\t')
            .quoteCharacter('"')
            .commentStrategy(CommentStrategy.NONE)
            .skipEmptyRows(true)
            .errorOnDifferentFieldCount(true)
            .build(path, charset);

reader.forEach(System.out::println);

Additional context java version "1.8.0_201"

Is your feature request related to a problem? Please describe.

Basically CsvWriter fails the Single Responsibility Principle by tackling both high level csv formatting and low level IO buffering, creating problems for non trivial uses.
Some use cases rely on a Writer obtained previously which is used to write more data than just a single CSV file. For example the same stream can contain multiple CSVs or some other data at the end.
The user can not do a writer.flush() because the writer is internally wrapped with CachingWriter so the last bytes may never get written until a csvWriter.close() is issued which closes the writer as well.

Once a Writer is passed its state is unknown until a call to CavWriter.close() is made!

Describe the solution you'd like

Let construct a CsvWriter without messing in any way with the writer passed. Let CsvWriter deal with building the csv data structure, and let the passed Writer deal with the low level stuff.

Describe alternatives you've considered

No alternative possible except a source code modification.

RFC 4180 compliance Would this feature comply to RFC 4180?

Yes, but the important part is that the code will be more correct, because it won't mess with external code passed to it.

Hi,

I am trying to read from a csv file containing a bit more than 2 million rows, then make a simple mapping to something i can use to finally insert it to a database. However, i am getting an erro: "GC limit overhead exceeded", as it creates a lot of temporary objects.

I read the other issue regarding temporary objects, however as i could understand, it is regarding writing to a csv file, but i am getting this error while reading from an csv file.

I want to create a big csv file using the CSV Appender. I'm using this code:

`for (int i = 0; i < writeBuffer.size(); i++) {
	String array[] = writeBuffer.get(i).toArray(new String[writeBuffer.get(i).size()]);
	csvAppender.appendLine(array);
   }`

being writeBuffer a List<List>. This buffer can have more than 500 lines. When I finish with the processing, the resulting file only has 148 lines and the last one is incomplete.

I also have try to flush at 100 lines, but then is not writting the next lines.

Maybe i am using the library in a incorrect way?

Thanks in advance.

The feature added with #57 had to be removed by #59 as it lacked support for unicode. Think of a better concept and implement it by keeping performance characteristics of reading regular files.