Instead of only having a boolean true/false for how duplicate header values are handled, this uses an enum instead.

Previously the only possibilities were:

• true: To allow duplicates.
• false: To disallow duplicates, except empty cells which were allowed to be duplicates.

This pull request makes an enum with three options:

• ALLOW_ALL: To always allow duplicates. (Same as true previously.)
• ALLOW_EMPTY: To allow duplicates only if they're empty cells. (Same as false previously.)
• DISALLOW: To disallow all duplicates.

This provides a little more flexibility in the strictness for the parser, and makes what the options do clearer.

Jira Issue: https://issues.apache.org/jira/browse/CSV-264

CSVRecord implements get(int) and size() methods so it’s already pretty much a list. However, because it does not implement the List interface, users cannot take advantage of convenience methods such as subList nor can they pass records around to methods accepting lists.

Make CSVRecord extend AbstractList so that it implements List interface.

Because AbstractList pravides iterator() implementation, this allows us to delete said method (alongside toList() method) reducing amount of code.

The support of duplicate header entries allows processing a CSV file do not worry about the presence of duplicate headers. It is enough to just call CSVFormat.DEFAULT.withIgnoreDuplicateHeaderEntries() that has to be first in the forming chain of the CSVFormat.

What is the need for this?! Here are two examples from real life.

1. There is a well-known set of columns from which to extract data. And there is no information about the potential presence of other columns (possibly duplicates) and their sequences in a document. The use of this feature will avoid such exceptions as java.lang.IllegalArgumentException: The header contains a duplicate name when the contents of the document are not fully known and there is a need to get by name. Example: Well-known columns set: [A, B, D]. Actual document columns set: [Z, A, B, C, D, C] Updated header structure: Z->[0], A->[1], B->[2], C->[3, 5], D->[4] Summarizing: This approach avoids exceptions for columns that do not even participate in processing. At the same time allows saving the possibility of getting by name.

2. There is a pivot table that aggregates other tables with the partially identical column names and there is a need to perform an aggregate function with the same columns. Example: Table1: [A, B, C] Table2: [B, C] Pivot table: [A, B, C, B, C] Task: need to perform an XOR for duplicate columns Updated header structure: A->[0], B->[1, 3], C->[2, 4]. Summarizing: This approach allows storing duplicates as an ordered set. Thus it will allow to perform xor(B[1], B[3]) & xor (C[2], C[4]).

…e.ALL is specified

In my opinion this is only a small change. If I already set the option (QuoteMode.ALL), then it should also be used.

JIRA-Reference

These are the changes:

• add getHeaderNames returns all headers in column order including repeats which are allowed in general as per RFC 4180
• add CSVFormat.withAllowDuplicateHeaderNames(). CSVFormat.DEFAULT now allows duplicate header names because RFC 4190 allows non-unique header names. This is a behavioural change but not a breaking API change anywhere because there is no API contract for it (e.g. javadoc). Because CSVFormat.DEFAULT should reflect RFC 4190 I'd classify this as a bug fix.
• CSVFormat is Serializable which means adding new fields to it (allowDuplicateHeaderNames) is theoretically a breaking change. I propose we allow this minor breaking change and also propose that CSVFormat does not implement Serializable in 2.x
• fix CSVRecord.toMap javadoc
• fix bug in CSVParser where an IAE is thrown with a message about duplicate headers when the problem was actually a missing header name
• add test coverage

Question:

• do we need to talk about HeaderNames when we could just say Header?

Not addressed:

• would be nice if CSVRecord.toMap returned a Map whose entries are iterable in column order but this involves quite a bit of rework so will leave for another PR (probably for 2.x).
• CSVRecord.get(String) should ideally throw when two columns with that header name exist

Notes for 2.x:

• for consistency CSVFormat.withAllowMissingColumnNames should be CSVFormat.withAllowMissingHeaderNames
• remove Serializable from CSVFormat
• CSVFormat.withIgnoreHeaderCase creates problems and lacks flexibility. I'd suggest CSVRecord.getIgnoreCase(int) instead

Fix for incorrect handling of embedded quotes when printing from Reader and to related test; added new test class to further exercise CSVFormat.printWithQuotes(Reader,Appendable) for expected behavior. CSV-263

fix [CSV-195] and [CSV-149] when stream end with normal char but not lineseparator will add 1 to currentlinenumber.

https://issues.apache.org/jira/browse/CSV-149 https://issues.apache.org/jira/browse/CSV-195

I tested in psql 14.5 Homebrew in Mac M1.

CSVFormat.POSTGRESQL_CSV - special characters are not escaped. CSVFormat.POSTGRESQL_TEXT - values are not quoted.

drop table COMMONS_CSV_PSQL_TEST;
create table COMMONS_CSV_PSQL_TEST (ID INTEGER, COL1 VARCHAR, COL2 VARCHAR, COL3 VARCHAR, COL4 VARCHAR);
insert into COMMONS_CSV_PSQL_TEST select 1, 'abc', 'test line 1' || chr(10) || 'test line 2', null, '';
insert into COMMONS_CSV_PSQL_TEST select 2, 'xyz', '\b:' || chr(8) || ' \n:' || chr(10) || ' \r:' || chr(13), 'a', 'b';
insert into COMMONS_CSV_PSQL_TEST values (3, 'a', 'b,c,d', '"quoted"', 'e');
copy COMMONS_CSV_PSQL_TEST TO '/tmp/psql.csv' WITH (FORMAT CSV);
copy COMMONS_CSV_PSQL_TEST TO '/tmp/psql.tsv';

cat /tmp/psql.csv
1,abc,"test line 1
test line 2",,""
2,xyz,"\b:^H \n:
\r:^M",a,b
3,a,"b,c,d","""quoted""",e

cat /tmp/psql.tsv
1    abc    test line 1\ntest line 2               \N
2    xyz    \\b:\b \\n:\n \\r:\r       a           b
3    a      b,c,d                      "quoted"    e

Being able to appropriately translate an absent value in CSV input with a Java null value. Previously, there was no way to do this, such a value would at best become a zero-length string when parsing. This made it impossible to correctly parse CSV output from say databases.

This PR is in reference to CSV-253.

The PR addresses the issue by adding a flag on Token so that it becomes possible to distinguish between a token which is the result of an absent value in input or an actual zero-length string. A new modifier, absentIsNull is introduced on CSVFormat. All existing formats and functionality are kept as-is, meaning the new feature is fully based on opt-in.

As a possible next step the pre-defined CSV formats for databases (i.e. INFORMIX_UNLOAD_CSV, MYSQL, ORACLE and POSTGRESQL_CSV) should be reviewed. I suspect that at least POSTGRESQL_CSV has always been incorrect in this matter. With this PR it can be corrected (if need be).

I've taken the liberty of adding "since 1.8" in the Javadocs since I see some commits for preparing such a release. Thus, hoping to include this.

Ensure that the resulting Commons CSV JAR file can be used in JPMS based projects

Prior to this change the JAR is missing the Automatic-Module-Name entry in its manifest meaning it cannot be used in JPMS based projects (e.g. rvesse/airline#106) yielding errors like the following:

Error occurred during initialization of boot layer
java.lang.module.FindException: Module org.apache.commons.csv not found, required by com.github.rvesse.airline.examples

With this change JPMS based projects can successfully use Commons CSV without errors.

Bumps opencsv from 5.5.1 to 5.5.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

As per issue https://issues.apache.org/jira/browse/CSV-141 and based on what we did in Apache OpenOffice https://bz.apache.org/ooo/show_bug.cgi?id=126805

This adds a setting, allowTrailingText (for lack of a better name) that allows CSV fields to have trailing text after the closing quote, up to the next separator, which can contain anything except the separator character, and this extra text is appended as-is to the field contents (any further quoting is ignored). This is exactly how Excel behaves.

As this is a non-standard setting with surprising behaviour, I've made it off by default. Only CSVFormat.EXCEL has it on by default.

This doesn't fully fix CSV-141 yet as that has line ending issues too, but I'd like to investigate how Excel handles that first.

• CSVFormat.duplicateHeaderMode requires default DISALLOW for backward compatibility
• The field does not allow null values
• Deserialization of version 1.9.0 CSVFormat objects now set duplicateHeaderMode based on the previous member boolean duplicateHeaderMode (done in readResolve)
• Add JUnit test for CSVFormat deserialization
• Fix a small bug in setNullString where member quotedNullString was inconsistently written (missing write in setQuote)

Bumps checkstyle from 9.3 to 10.3.4.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This pull request solves ticket CSV-303 by adding org.revapi:revapi-maven-plugin to the build section of the library's maven POM. Please note that config file revapi-config.json is added to exclude CSVFormat.serialVersionUID from checks. The field should probably be updated to something != 1 along with custom de-serialization for backward compatibility (see ticket CSV-302).

• https://issues.apache.org/jira/projects/CSV/issues/CSV-150

Hi, @garydgregory In the source code, it is estimated that for performance reasons, Character is converted to char, so null is mapped to \ufffe which cannot appear in the middle of a normal file. Now if you have the \ufffe character in the middle of the file, there's a problem.

added ignoreQuoteInToken support to ignore quotes in strings even when there are few encapsulatedTokens with comma within. This will help in parsing csv values like abc,"xyz" 123 bar,3,11961034,"First author, Second Author"