Co-Authored-By: Ismaël Mejía [email protected]

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

What changes were proposed in this pull request?

Enable precommit tests for github PR against branch-2.3.

Why are the changes needed?

Adding a new Jenkinsfile for the repo. This is almost the same file used in the master branch, except changing timeout from 6 to 12 hours.

Does this PR introduce any user-facing change?

No

How was this patch tested?

N/A

What changes were proposed in this pull request?

This PR aims to upgrade Apache ORC from 1.5.6 to 1.5.8.

Why are the changes needed?

This will bring eleven bug fixes.

• ORC 1.5.7: https://issues.apache.org/jira/projects/ORC/versions/12345702
• ORC 1.5.8: https://issues.apache.org/jira/projects/ORC/versions/12346462

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CI with the existing test cases.

What changes were proposed in this pull request?

This PR proposes to upgrade Guava to 27 in Hive 2.3 branch.

Why are the changes needed?

When trying to upgrade Guava in Spark, found the following error. A Guava method became package-private since Guava version 20. So there is incompatibility with Guava versions > 19.0.

sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator
	at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:108)
	at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)

Does this PR introduce any user-facing change?

Yes. This upgrades Guava to 27.

How was this patch tested?

Built Hive locally.

What changes were proposed in this pull request?

Bump apache ORC version to latest 1.6 release (1.6.7)

Why are the changes needed?

So Apache HIVE can take advantage of the latest features and bug fixes

Does this PR introduce any user-facing change?

• Integer to Timestamp conversion uses seconds while ORC-1.5 used milliseconds This results in different expected output in queries like: schema_evol_orc_nonvec_part_all_primitive.q

Non user-facing changes:

• CacheWritter bufferSize is now decoupled from llap.max.alloc with the former being 8Mb and the later 16Mb
• ZeroCopy Tests for ORC files disabled until ORC-701

How was this patch tested?

Internal tests + q files

What changes were proposed in this pull request?

This PR proposes to upgrade Guava to 27 in Hive branch-2. This is basically used to trigger test for #1394.

Why are the changes needed?

When trying to upgrade Guava in Spark, found the following error. A Guava method became package-private since Guava version 20. So there is incompatibility with Guava versions > 19.0.

sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator
	at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:108)
	at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)

Does this PR introduce any user-facing change?

Yes. This upgrades Guava to 27.

How was this patch tested?

Built Hive locally.

What changes were proposed in this pull request?

• This fixes https://issues.apache.org/jira/browse/HIVE-25522
• There are two options, either make the initialization static and kill HMS if there is an error, or keep it lazy. Went with second approach as there seem to be db connections that are taken and don't need to be if nobody uses any txnHandler methods.
• Make the initailization setConf method be idempotent by checking each of the static variables it sets and able to resume if a particular variable is not set.
• Also some refactor to push down verbose catch blocks as much as possible.

Why are the changes needed?

• See https://issues.apache.org/jira/browse/HIVE-25522

Does this PR introduce any user-facing change?

No

How was this patch tested?

Running unit tests

What changes were proposed in this pull request?

This PR proposes to shade Guava from hive-exec in Hive 2.3 branch.

Why are the changes needed?

When trying to upgrade Guava in Spark, found the following error. A Guava method became package-private since Guava version 20. So there is incompatibility with Guava versions > 19.0.

sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator
	at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:108)
	at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)

This is a problem for downstream clients. Hive project noticed that problem too in HIVE-22126, however that only targets 4.0.0. It'd be nicer if we can also shade Guava from current Hive versions, e.g. Hive 2.3 line.

Does this PR introduce any user-facing change?

Yes. Guava will be shaded from hive-exec.

How was this patch tested?

Built Hive locally and checked jar content.

Successor to https://github.com/apache/spark/pull/23888

See discussion there for some more details about the Hive side of this, in particular my comment here about existing StackOverflow answers and here:

My conclusion is that it's eminently ambiguous whether the intended behavior in either Hive or SparkSQL is to treat the delimiters as regular expressions.

BUT the behavior has been around for 8 years and at least going off of the SO answers, it seems to be accepted as "known" behavior so things will probably break if we change it.

Thus, this PR intends to solidify the interpretation of delimiter1 and delimiter2 as regular expressions once and for all.

If the non-regexp behavior is strongly desired, eventually there could be a fixed: bool argument that behaves like the identically-named argument in R regular expression functions like gsub and strsplit...

What changes were proposed in this pull request?

Upgrade calcite version due to CVE - https://issues.apache.org/jira/browse/HIVE-26254

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

What changes were proposed in this pull request?

HIVE-26892: Backport HIVE-25243 to 3.2.0: Handle nested values in null struct.

This differs from the original patch as follows:

• There are no changes in TestJdbcWithMiniLlapVectorArrow. This file does not exist in branch-3. The original patch was only a whitespace change though.
• I have not included the new test suite TestMiniLlapVectorArrowWithLlapIODisabled. This test suite depends on an additional feature that is only present in master: HIVE-20300 / https://github.com/apache/hive/commit/a8ef2147fad5aeaaf01279230da9c584db6a2337. This is a fairly large patch that I have not chosen to try backporting at this time.

Why are the changes needed?

On branch-3, we've seen a failure in TestArrowColumnarBatchSerDe while trying to serialize a row of null values. It fails while trying to serialize the fields of a null struct. This was fixed in 4.0 by HIVE-25243. This issue tracks a backport to branch-3.

Does this PR introduce any user-facing change?

Null structs are now serialized to Arrow format correctly without error.

How was this patch tested?

Unfortunately, we are left with no specific new tests as part of this backport. However, I applied this patch locally in combination with HIVE-26840 / #3859. After that, all tests in TestArrowColumnarBatchSerDe are passing.

What changes were proposed in this pull request?

Because of Jackson upgrade to 2.12.0 there are unit test failure in TestArrowColumnarBatchSerDe. We cannot directly upgrade the arrow version as it was giving compilation error. These commits are interdependent hence these 6 commits are required.

Why are the changes needed?

There are 2 ways to fix these tests:

1. Downgrade Jackson Version to 2.10.0
2. Upgrade arrow version from 0.8.0 to 0.10.0. We are going ahead with this.

Does this PR introduce any user-facing change?

No

How was this patch tested?

On my local machine

What changes were proposed in this pull request?

Implement array_join function in Hive

Why are the changes needed?

This enhancement is already implemented in Spark

Does this PR introduce any user-facing change?

How was this patch tested?

Created Junit tests as well as qtests as part of this change

This was done as part of HIVE-21456 on oss/master. This test also fails with the same error in Hive-3.1.3 release. So we can safely ignore this test from branch-3. Please find the JIRA link : https://issues.apache.org/jira/browse/HIVE-26890

HIVE-26887 In the QueryResultsCache function of class QueryResultsCache, there is the following code segment

  private QueryResultsCache(HiveConf configuration) throws IOException {
    ......
    FileSystem fs = cacheDirPath.getFileSystem(conf);
    FsPermission fsPermission = new FsPermission("700");
    fs.mkdirs(cacheDirPath, fsPermission);
    ......
}

It can be seen that the function will use the mkdirs to create cacheDirPath, and the parameters passed in include the path variable cacheDirPath and a permission 700. But we haven't confirmed whether the permission is correctly assigned to the file.

The above question is raised because there are two mkdir functions of hadoop,mkdirs(Path f, FsPermission permission) with mkdirs(FileSystem fs, Path dir, FsPermission permission)and the first one is used here. The permissions of this function will be affected by the underlying umask. Although 700 here will hardly be affected by umask, but I think from a rigorous point of view, we should have one more permission check and permission grant here. So I judge whether the file permission is correctly granted by detecting the influence of umask on the permission, and if not, give it the correct permission through setPermission.

    if (!permission.equals(permission.applyUMask(FsPermission.getUMask(conf)))) {
        fs.setPermission(dir, permission);
    }

And I find same issue in other three methods here. In class Context

private Path getScratchDir(String scheme, String authority,
      boolean mkdir, String scratchDir) {
          ......
          FileSystem fs = dirPath.getFileSystem(conf);
          dirPath = new Path(fs.makeQualified(dirPath).toString());
          FsPermission fsPermission = new FsPermission(scratchDirPermission);

          if (!fs.mkdirs(dirPath, fsPermission)) {
            throw new RuntimeException("Cannot make directory: "
                + dirPath.toString());
          ......
  }

In class SessionState

  static void createPath(HiveConf conf, Path path, String permission, boolean isLocal,
      boolean isCleanUp) throws IOException {
    FsPermission fsPermission = new FsPermission(permission);
    FileSystem fs;
    ......
    if (!fs.mkdirs(path, fsPermission)) {
      throw new IOException("Failed to create directory " + path + " on fs " + fs.getUri());
    }
    ......
  }

and in class TezSessionState

private Path createTezDir(String sessionId, String suffix) throws IOException {
    ......
    Path tezDir = new Path(hdfsScratchDir, TEZ_DIR);
    FileSystem fs = tezDir.getFileSystem(conf);
    FsPermission fsPermission = new FsPermission(HiveConf.getVar(conf, HiveConf.ConfVars.SCRATCHDIRPERMISSION));
    fs.mkdirs(tezDir, fsPermission);
    ......
  }

What changes were proposed in this pull request?

Implement array_slice function in Hive

Why are the changes needed?

This enhancement is already implemented in Spark

Does this PR introduce any user-facing change?

No

How was this patch tested?

Created Junit tests as well as qtests as part of this change