Description

Solr provides a BackupRepository interface with which users can create backups to arbitrary backends. There is now a GCS implementation (see https://github.com/apache/solr/pull/39), but no S3 impl yet.

Solution

This PR adds a BackupRepository implementation for communicating with S3.

Tests

We've added new unit tests at the BackupRepository level as well as tests for the S3 interactions (using S3Mock framework).

Checklist

Please review the following and check all that apply:

• [x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [x] I have created a Jira issue and added the issue ID to my pull request title.
• [x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [x] I have developed this patch against the main branch.
• [x] I have run ./gradlew check.
• [x] I have added tests for my changes.
• [x] I have added documentation for the Reference Guide

We have not yet done the work of adding license files for all the newly added libraries/dependencies. These will be added in a future commit.

The test :solr:validateConfigFileSanity checks that <luceneMatchVersion> is correct in various solrconfig files. Now that lucene version is different from solr version (e.g. on main branch right now), the build needs to know what lucene version we have.

I tried using Lucene's Version.LATEST.toString(), but the build found a 8_10_0 version in my env. So now I try to just pull it from versions.props, and that works.

However, it may noe be compatible with the local lucene version override feature in lucene-dev-repo-composite.gradle??

https://issues.apache.org/jira/browse/SOLR-15955

Summary:

• Upgrades to Jetty 10.0.12
• dropwizard metrics 4.2.12 for dropwizard-metrics9 -> dropwizard-metrics10
• log4j 2.19.0 and slf4j 2.0.3
• for s3mock specifically upgrade spring-boot 2.5.14 and spring 5.3.23 to handle Jetty 10

https://issues.apache.org/jira/browse/SOLR-15824

Description

While sending a query in solr, it is very convenient that the q field can be resizable(because it's in textarea tag), while the fq and raw query parameters fields are not resizable, which can cause difficulties in long query parameters. To solve this made improvement on query.html and query.css.

Solution

I put fq and raw query parameters inside textarea tag and with css I made it vertical resizable because of the line between query and result. Because if it was resizable both vertical and horizontal or only horizontal, it would cause the appearance of the panel to deteriorate.

Checklist

Please review the following and check all that apply:

• [X] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [X] I have created a Jira issue and added the issue ID to my pull request title.
• [X] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [X] I have developed this patch against the main branch.
• [X] I have run ./gradlew check.
• [ ] I have added tests for my changes.
• [ ] I have added documentation for the Reference Guide

EDIT

I changed only raw query parameters section because of the comments on PR. Added +/- buttons for raw query parameters.

https://issues.apache.org/jira/browse/SOLR-16271

Description

Remove wildcard imports from the build. In later issue, will add a spotless based check.

Solution

made the changes.

Tests

manually run tests.

Checklist

Please review the following and check all that apply:

• [ ] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [ ] I have created a Jira issue and added the issue ID to my pull request title.
• [ ] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [ ] I have developed this patch against the main branch.
• [ ] I have run ./gradlew check.
• [ ] I have added tests for my changes.
• [ ] I have added documentation for the Reference Guide

https://issues.apache.org/jira/browse/SOLR-15982

Description

Adding new field endTime to collection backup response (response is actually an aggregated result of responses from multiple nodes).

Solution

After backup finished (regardless is it sync or async) it will write backup.properties file to repository. Actually it writes the content of the org.apache.solr.core.backup.BackupProperties. Before writing backup properties data the value endTime is filled at org.apache.solr.core.backup.BackupProperties.store(Writer):

    public void store(Writer propsWriter) throws IOException {
        properties.put("indexSizeMB", String.valueOf(indexSizeMB));
        properties.put("indexFileCount", String.valueOf(indexFileCount));
        properties.put(BackupManager.END_TIME_PROP, Instant.now().toString());
        properties.store(propsWriter, "Backup properties file");
    }

Exactly after writing backup properties file an additional field endTime will be appended to the response within org.apache.solr.cloud.api.collections.BackupCmd.call(ClusterState, ZkNodeProps, NamedList<Object>) call:

  public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results) throws Exception {
    ...
    try (BackupRepository repository = cc.newBackupRepository(repo)) {
      ...
      backupMgr.writeBackupProperties(backupProperties);

      if(backupProperties != null) {
        NamedList<Object> response = (NamedList<Object>) results.get("response");
        response.add("endTime", backupProperties.getEndTime());
      }
      ...
    }
  }

Tests

None

Checklist

Please review the following and check all that apply:

• [x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [x] I have created a Jira issue and added the issue ID to my pull request title.
• [x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [x] I have developed this patch against the main branch.
• [x] I have run ./gradlew check.
• [ ] I have added tests for my changes.
• [x] I have added documentation for the Reference Guide

https://issues.apache.org/jira/browse/SOLR-15342

I open this PR in draft to share the evolution of the work. I closed the old PR since it has been open for a while, there have been quite a few changes since then in relation to ZooKeeper and some of the work has been contributed in a separate PR. I resume the work done in the latter in this one

https://issues.apache.org/jira/browse/SOLR-15842

Description

Adding new field to org.apache.solr.handler.admin.CoreAdminHandler.TaskObject to hold operation results.

Solution

I apply a simple changes that adds an additional field operationRspInfo to org.apache.solr.handler.admin.CoreAdminHandler.TaskObject. It is filled with operation results in case of operation finished successfully (it is exactly the same results that are used in case of sync request). This way we store the results within TaskObject.

Later, when request status is sent, besides adding standard value Response, an additional value response will be added to request response. Value response will hold operation results preserved in TaskObject.

Tests

None

Checklist

Please review the following and check all that apply:

• [x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [x] I have created a Jira issue and added the issue ID to my pull request title.
• [x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [x] I have developed this patch against the main branch.
• [ ] I have run ./gradlew check.
• [ ] I have added tests for my changes.
• [ ] I have added documentation for the Reference Guide

https://issues.apache.org/jira/browse/SOLR-10452

Description

setQueryParams should be deprecated in favor of SolrClientBuilder methods.

Solution

I moved the the setQueryParams over for the Http2SolrClient, it already had been done for HttpSolrClient. I noticed we have a addQueryParams method, which I marked deprecated and we shouldn't use, as that goes against the idea of a Solr Client being immutable.

One area I tried to fix and gave up on was the DelegationTokenHttpSolrClient, I couldn't quite figure out what to do there, would love a suggestion or a fix ;-)

Tests

Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.

Checklist

Please review the following and check all that apply:

• [ ] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [ ] I have created a Jira issue and added the issue ID to my pull request title.
• [ ] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [ ] I have developed this patch against the main branch.
• [ ] I have run ./gradlew check.
• [ ] I have added tests for my changes.
• [ ] I have added documentation for the Reference Guide

https://issues.apache.org/jira/browse/SOLR-16574

Description

Enrich the films example to demonstrate how to use the Dense Vectors feature.

Solution

Added the field film_vector to the films dataset. This is an embedding vector created to represent the movie with 10 dimensions. The vector is created by combining the first 5 dimensions of a pre-trained BERT sentence model applied on the name of the movies plus the name of the genres, followed by an item2vec 5-dimensions model of itemset co-occurrence of genres in the movies, totaling 10 dimensions. Even though it is expected that similar movies will be close to each other, this is just a "toy example" model to serve as source for creating the films vectors.

The README of the example was also updated to include the specification of the Dense Vector field in the schema. Also, a new section was created, with examples showing how to make KNN queries with the vectors.

Tests

• Added the new field film_vector to the 3 dataset formats (JSON, XML, CSV), making sure to preserve the exact same data from the original datasets, so that the "diff" will be only the appendage of the new field.
• Checked the creation of the collection for the 3 dataset formats. Regardless of the format all the 1100 films were indexed, and the film_vector field was correctly parsed and indexed as well.
• Checked the KNN example queries for all the 3 dataset formats.

Checklist

Please review the following and check all that apply:

• [X] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [X] I have created a Jira issue and added the issue ID to my pull request title.
• [X] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [X] I have developed this patch against the main branch.
• [ ] I have run ./gradlew check.
• [ ] I have added tests for my changes.
• [ ] I have added documentation for the Reference Guide

https://issues.apache.org/jira/browse/SOLR-16368

Description

Part of working on reducing the use of legacy HttpSolrClient in the tests everywhere is seeing if mutating the client can be reduced by embracing the Builder pattern.

Solution

Use the Builder where possible.

I couldn't figure out how to untangle the logic in TestRandomFlRTGCloud and would love some eyes on that one!

Tests

ran the tests

Checklist

Please review the following and check all that apply:

• [ ] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [ ] I have created a Jira issue and added the issue ID to my pull request title.
• [ ] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [ ] I have developed this patch against the main branch.
• [ ] I have run ./gradlew check.
• [ ] I have added tests for my changes.
• [ ] I have added documentation for the Reference Guide

https://issues.apache.org/jira/browse/SOLR-16613

Tested with:

# With Temurin 17
./gradlew test --tests TestRSAKeyPair
./gradlew test --tests TestPKIAuthenticationPlugin

# With OpenJ9 17
RUNTIME_JAVA_HOME=/Library/Java/JavaVirtualMachines/ibm-semeru-open-17.jdk/Contents/Home ./gradlew test --tests TestRSAKeyPair
RUNTIME_JAVA_HOME=/Library/Java/JavaVirtualMachines/ibm-semeru-open-17.jdk/Contents/Home ./gradlew test --tests TestPKIAuthenticationPlugin

https://issues.apache.org/jira/browse/SOLR-16532

I created a new PR where we can gather all followup cleanups that may arrive after the initial merge in #1168

https://issues.apache.org/jira/browse/SOLR-16610

Description

This is an example of the types of changes we would need to make to be more in line with Antora's handling of command lines and being able to cut n paste them.

Solution

Conversion

Tests

manual

https://issues.apache.org/jira/browse/SOLR-16608

Description

This PR provides the ability to configured a minimum size of state.json above which it will be compressed when written to Zookeeper. Solr will be able to handle compressing/decompressing in ZLib format in all areas where Solr reads state.json from Zookeeper.

Solution

This uses ZLib compression to optionally compress state.json, the core of this logic is contained within a new class called CompressionUtil that handles both compression and decompression as well as some logic to check if bytes are compressed in a very efficient way. This can handle reading/writing both compressed and uncompressed data so it changing the configuration for compression is backward compatible and will not break reading state.json.

Tests

There are tests added to check various layers to ensure compression/decompression is working, including unit tests for CompressionUtil and tests for ZkStateReader and ZkStateWriter.

Checklist

Please review the following and check all that apply:

• [x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [x] I have created a Jira issue and added the issue ID to my pull request title.
• [ ] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [x] I have developed this patch against the main branch.
• [x] I have run ./gradlew check.
• [x] I have added tests for my changes.
• [x] I have added documentation for the Reference Guide

https://issues.apache.org/jira/browse/SOLR-788 I think this is a somewhat reliable solution for the problem with distributed mlt component. Reminder, now it naively relies on parsing booleanQuery.toString() that's unacceptable.