OpenSearch is an open source distributed and RESTful search engine.

Overview

OpenSearch

Hello!

OpenSearch is an open source search and analytics engine derived from Elasticsearch 7.10.2, and is currently in an alpha state. While we're still in the process of getting set up, we are excited to build the best distributed search engine on the planet with you.

We officially began work on the new fork on January 21st, 2021. Since then, we've been removing non-Apache 2.0 compliant code and doing a full rename of the project. Feel free to take a look around at what we've been up to, and then head over to look at the open issues. Or you could jump right in and start opening issues or contributing.

Getting Started

If you would like to install or run this project please see the Developer Guide. There will be binary distributions coming soon which will enable faster installations.

Guiding Principles

Here are the principles we use to guide our development. From opensearch.org:

When we (the contributors) are successful, the OpenSearch project will be:

1. Great software. If it doesn’t solve your problems, everything else is moot. It’s going to be software you love to use.

2. Open source like we mean it. We are invested in this being a successful open source project for the long term. It’s all Apache 2.0. There’s no Contributor License Agreement. Easy.

3. A level playing field. We will not tweak the software so that it runs better for any vendor (including AWS) at the expense of others. If this happens, call it out and we will fix it as a community.

4. Used everywhere. Our goal is for as many people as possible to use it in their business, their software, and their projects. Use it however you want. Surprise us!

5. Made with your input. We will ask for public input on direction, requirements, and implementation for any feature we build.

6. Open to contributions. Great open source software is built together, with a diverse community of contributors. If you want to get involved at any level - big, small, or huge - we will find a way to make that happen. We don’t know what that looks like yet, and we look forward to figuring it out together.

7. Respectful, approachable, and friendly. This will be a community where you will be heard, accepted, and valued, whether you are a new or experienced user or contributor.

8. A place to invent. You will be able to innovate rapidly. This project will have a stable and predictable foundation that is modular, making it easy to extend.

How you can help

Look for the tag "help wanted"

If you're excited to jump in, we've marked a few issues that would be really helpful.

Watch the forum, because there are a bunch of things we want to talk about

As we've been working, we've come up with a bunch of questions that we wanted to get community feedback on. Now that we're done with this first pass of renaming, you'll see us posting those questions to the forums. Please let us know your thoughts!

Questions? Feedback?

Let us know in the forums.

Comments
  • Fix resource leak issues suggested by Amazon CodeGuru

    Fix resource leak issues suggested by Amazon CodeGuru

    Description

    Address a kind of issue suggested by Amazon CodeGuru Reviewer.

    1. Add try-with-resources block to automatically close the resources after using to avoid resource leak.
    • Close FileInputStream in SymbolicLinkPreservingTarIT, LicenseAnalyzer, SymbolicLinkPreservingUntarTransform and ConcurrentSeqNoVersioningIT classes
    • Close InputStream in VersionProperties and GeoFilterIT classes
    • Close one variable of InputStream in XContentHelper class
    • Close OutputStream in Json class
    • Close PrintStream and BytesStreamOutput in IndexShard class
    1. Add try-finally block to close the resources after using to avoid resource leak.
    • Close SocketChannel in ServerChannelContext class
    1. Add try-catch block to close the resources when exception occurs.
    • Close other InputStream in XContentHelper class when XContentFactory.xContentType throws an exception ~~- Close SeekableByteChannel in FsBlobContainer class~~
    1. Close resources when assertion error occurs.
    • Close SocketChannel in ServerChannelContext class

    Reference: The change in LicenseAnalyzer class is copied from #692

    Issues Resolved

    [List any issues this PR will resolve]

    Check List

    • [ ] New functionality includes testing.
      • [x] All tests pass
    • [ ] New functionality has been documented.
      • [ ] New functionality has javadoc added
    • [x] Commits are signed per the DCO using --signoff

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    security 
    opened by tlfeng 132
  •  [Bug]: gradle check failing with java heap OutOfMemoryError

    [Bug]: gradle check failing with java heap OutOfMemoryError

    Signed-off-by: Andriy Redko [email protected]

    Description

    Fixing Gradle check failing with java heap OutOfMemoryError (hopefully). It turns out that Gradle was not using options.forkOptions.memoryMaximumSize property and always falled back to default of 512m:

    [..., -Djava.awt.headless=true, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dtests.artifact=rest-api-spec, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=D349409
    4BD808CE3, -Dtests.task=:rest-api-spec:test, -XX:+HeapDumpOnOutOfMemoryError, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -esa, -javaagent:../../tmp/expandedArchives/org.jacoco.agent-0.8.8.jar_a33b649e552c51298e5a242c2f0d0e3c/jacocoagent.jar=destfile=../../jacoco
    /test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, , -Duser.country=CA, -Duser.language=en, -Duser.variant, -ea]
    

    By alterning the JavaForkOptions tasks, the -Xmx setting seems to be respected

    project.tasks.withType(JavaForkOptions) {
      maxHeapSize project.property('options.forkOptions.memoryMaximumSize')
    }
    
    
    [-Djava.awt.headless=true, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dtests.artifact=rest-api-spec, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=86BB942
    FF1760033, -Dtests.task=:rest-api-spec:test, -XX:+HeapDumpOnOutOfMemoryError, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -esa, -javaagent:../../tmp/expandedArchives/org.jacoco.agent-0.8.8.jar_a33b649e552c51298e5a242c2f0d0e3c/jacocoagent.jar=destfile=../../jacoco
    /test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx2g, -Dfile.encoding=UTF-8, -Duser.country=CA, -Duser.language=en, -Duser.variant, -ea]  
    

    Issues Resolved

    Closes https://github.com/opensearch-project/OpenSearch/issues/3973

    Check List

    • [ ] New functionality includes testing.
      • [ ] All tests pass
    • [ ] New functionality has been documented.
      • [ ] New functionality has javadoc added
    • [X] Commits are signed per the DCO using --signoff
    • [X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    bug CI backport 2.x backport 2.0 backport 2.1 backport 2.2 
    opened by reta 128
  • Concurrent Searching (Experimental)

    Concurrent Searching (Experimental)

    Signed-off-by: Andriy Redko [email protected]

    Description

    Allows to use experimental Apache Lucene low-level API which allows to parallelize execution of the search across segment.

    • [x] Implement multi-collectors query context (possibly, needs API change to use CollectorManagers instead)

    • [X] Implement profilers support

      The query profiling is supported but it has some limitations and differences with respect to naming. The profilers capture the time each collection type has taken and return it as the cumulative summary. The fact that segments may have been searched concurrently is not reflected (it requires significant overhauling of the profiling implementation), so the "real" perceived time may be smaller then profilers will report. The collector names are replaced with collector manager names (fe MinimumScoreCollector vs MinimumCollectorManager).

      To compare, here is the usual search profile

      "collector" : [
         {
           "name" : "MinimumScoreCollector",
           "reason" : "search_min_score",
           "time_in_nanos" : 4702942,
           "children" : [
             {
               "name" : "SimpleTopScoreDocCollector",
               "reason" : "search_top_hits",
               "time_in_nanos" : 3450252
             }
           ]
         }
       ]
      

      vs the one where concurrent segment search is enabled

      "collector" : [
        {
          "name" : "MinimumCollectorManager",
          "reason" : "search_min_score",
          "time_in_nanos" : 74827,
          "children" : [
            {
              "name" : "SimpleTopDocsCollectorManager",
              "reason" : "search_top_hits",
              "time_in_nanos" : 98044
            }
          ]
        }
      ]
      

      Additionally, since TopDocs are merged, the query's shardIndex field is also populated, usual search profile has it set to -1, for example:

      "query" : [
        {
          "type" : "ConstantScoreQuery",
          "description" : "ConstantScore(SearchAfterSortedDocQuery(sort=<int: \"rank\">, afterDoc=doc=287 score=NaN shardIndex=-1 fields=[-2062713761]))",
          "time_in_nanos" : 1009712,
          ...
        }
      ]
      

      vs the one where concurrent segment search is enabled

      "query" : [
        {
          "type" : "ConstantScoreQuery",
          "description" : "ConstantScore(SearchAfterSortedDocQuery(sort=<int: \"rank\">, afterDoc=doc=287 score=NaN shardIndex=1 fields=[-2062713761]))",
          "time_in_nanos" : 35667,
            ....
        }
      ]
      
    • [X] Implement early termination and timeout support

      The forced early termination is simulated (without abrupt search termination) using Apache Lucene normal collection termination mechanism. The number of results is reduced to the desired one (if necessary) at the post search step. Essentially, to summarize, when early termination is requested and concurrent segment search is enabled, be the engine is going to do more work and (very likely) come up with more search hits than have been asked.

      If query finished by timeout, no results are being returned, even if returning partial results is acceptable. The reason for that exception propagation mechanism does not leave a chance for reducers to execute.

    • [x] Add benchmarks suite

    Splitting into subtasks:

    • [x] Extract changes related to profiler support (https://github.com/opensearch-project/OpenSearch/pull/1673)
    • [x] Extract changes related to query context support
    • [x] Move the concurrent search implementation to sandbox

    Future Enhancements:

    • [ ] https://github.com/opensearch-project/OpenSearch/issues/2585
    • [ ] https://github.com/opensearch-project/OpenSearch/issues/2586

    Issues Resolved

    Fixes https://github.com/opensearch-project/OpenSearch/issues/1286

    Check List

    • [X] New functionality includes testing.
      • [X] All tests pass
    • [X] New functionality has been documented.
      • [X] New functionality has javadoc added
    • [X] Commits are signed per the DCO using --signoff

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    v2.0.0 WIP feature Indexing & Search backport 2.x 
    opened by reta 118
  • Merge shard level Indexing back-pressure feature branch

    Merge shard level Indexing back-pressure feature branch

    Description

    This PR aims to merge the changes for the shard level indexing back-pressure changes from the feature branch. The changes were divided into small manageable chunks as part of the following PRs against the feature branch.

    • #716
    • #717
    • #718
    • #838
    • #945
    • #1015
    • #1084
    • #1113
    • #1171
    • #1198

    Issues Resolved

    #478

    Check List

    • [x] New functionality includes testing.
      • [x] All tests pass
    • [ ] New functionality has been documented.
      • [x] New functionality has javadoc added
    • [x] Commits are signed per the DCO using --signoff

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    v1.2.0 
    opened by adnapibar 108
  • Create PIT API

    Create PIT API

    Description

    This PR contains Create PIT API changes. Context: A Point In Time is a set of segments which are frozen in time taken at the moment of creation. We lock the segments of those indices’ shards and create contexts to access and query only those shards. The Create PIT API returns PIT Id which later can be used as part of search requests to get results on that view of data.

    Issues Resolved

    https://github.com/opensearch-project/OpenSearch/issues/1147

    Check List

    • [ ] New functionality includes testing.
      • [ ] All tests pass
    • [ ] New functionality has been documented.
      • [ ] New functionality has javadoc added
    • [x] Commits are signed per the DCO using --signoff

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    opened by bharath-techie 96
  • Support Gradle 7

    Support Gradle 7

    Signed-off-by: Andriy Redko [email protected]

    Description

    Support Gradle 7

    Issues Resolved

    Closes https://github.com/opensearch-project/OpenSearch/issues/1246

    Check List

    • [X] New functionality includes testing.
      • [X] All tests pass
    • [ ] New functionality has been documented.
      • [ ] New functionality has javadoc added
    • [X] Commits are signed per the DCO using --signoff

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    backport backport 1.x 
    opened by reta 94
  • [Upgrade] Lucene 9.0.0 release

    [Upgrade] Lucene 9.0.0 release

    This PR upgrades the core codebase from Lucene 8.10.1 to lucene-9.0.0 release. It includes all necessary refactoring of features and API changes when upgrading to the new major Lucene release.

    relates #1067 depends on #2452

    v2.0.0 >upgrade 
    opened by nknize 77
  • Add changes for graceful node decommission

    Add changes for graceful node decommission

    Signed-off-by: pranikum [email protected]

    Description

    Changes for graceful decommission of nodes. This includes call to WRR api. Can be merged once decommission API and WRR changes are merged.

    Depends on : https://github.com/opensearch-project/OpenSearch/pull/4272 https://github.com/opensearch-project/OpenSearch/pull/4241

    Issues Resolved

    https://github.com/opensearch-project/OpenSearch/issues/4083

    Check List

    • [x] New functionality includes testing.
      • [x] All tests pass
    • [ ] New functionality has been documented.
      • [ ] New functionality has javadoc added
    • [X] Commits are signed per the DCO using --signoff
    • [X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    opened by pranikum 75
  • Allowing custom folder name for plugin installation

    Allowing custom folder name for plugin installation

    Signed-off-by: Vacha Shah [email protected]

    Description

    This PR allows the user to add a custom folder name for plugin cli installation. This custom folder name can be specified in the plugin properties file for the plugin, when this property is specified, the plugin would be installed with the custom folder name otherwise it will be installed with the name of the plugin. This feature would be available for versions after 1.0.0.

    The feature includes:

    1. Support for custom folder name for plugin installation after 1.0.0 versions with backwards compatibility.
    2. The install plugin command will display the folder name where the plugin is being installed.
    3. The list plugin command will continue to display the plugin name to maintain backwards compatibility.
    4. The remove plugin command will continue to take in the plugin name to remove it even though the plugin is installed in a custom folder to maintain backwards compatibility.
    5. During installation or removal, when checking if a plugin already exists, any folder previously installed with the plugin name or custom name would be determined.

    Testing

    • For manual testing, I added the customFolderName property in build.gradle for job-scheduler plugin as follows:
    opensearchplugin {    
      name 'opensearch-job-scheduler'    
      description 'OpenSearch Job Scheduler plugin'    
      classname 'org.opensearch.jobscheduler.JobSchedulerPlugin'
      customFolderName 'custom-folder'
    }
    
    1. When installing the plugin with the above properties using ./opensearch-plugin install <path to build file>, this resulted in the plugin being installed in folder: custom-folder with an output message as follows:
    -> Installed opensearch-job-scheduler with folder name custom-folder
    
    1. When the plugins are being listed using ./opensearch-plugin list, this resulted in the output: opensearch-job-scheduler.
    2. To remove the plugin, I used ./opensearch-plugin remove opensearch-job-scheduler.

    Issues Resolved

    #686

    Check List

    • [X] New functionality includes testing.
      • [X] All tests pass
    • [ ] New functionality has been documented.
      • [ ] New functionality has javadoc added
    • [X] Commits are signed per the DCO using --signoff

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    v2.0.0 feature documentation pending v1.1.0 
    opened by VachaShah 74
  • Test PR - DCO / Check testing

    Test PR - DCO / Check testing

    DCO / Check testing

    Signed-off-by: Peter Nied [email protected]

    By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

    opened by peternied 73
  • [Backport] Support for translog pruning based on retention leases

    [Backport] Support for translog pruning based on retention leases

    Description

    Backport of https://github.com/opensearch-project/OpenSearch/pull/1038

    Issues Resolved

    N/A

    Check List

    • [x] New functionality includes testing.
      • [x] All tests pass
    • [x] New functionality has been documented.
      • [x] New functionality has javadoc added
    • [x] Commits are signed per the DCO using --signoff

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    Severity-Blocker backport v1.1.0 distributed framework 
    opened by saikaranam-amazon 70
  • Flaky test failure `SegmentReplicationIT.testAddNewReplicaFailure`

    Flaky test failure `SegmentReplicationIT.testAddNewReplicaFailure`

    Coming from https://github.com/opensearch-project/OpenSearch/pull/4959, which enables yml based feature flag settings; reveals flakyness in testAddNewReplicaFailure

    REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationIT.testAddNewReplicaFailure" -Dtests.seed=D3DBF1C239661FF6 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=it -Dtests.timezone=Brazil/West -Druntime.java=19
    
    org.opensearch.indices.replication.SegmentReplicationIT > testAddNewReplicaFailure FAILED
        java.lang.AssertionError
            at __randomizedtesting.SeedInfo.seed([D3DBF1C239661FF6:F35C3295F5C37311]:0)
            at org.junit.Assert.fail(Assert.java:87)
            at org.junit.Assert.assertTrue(Assert.java:42)
            at org.junit.Assert.assertFalse(Assert.java:65)
            at org.junit.Assert.assertFalse(Assert.java:75)
            at org.opensearch.indices.replication.SegmentReplicationIT.testAddNewReplicaFailure(SegmentReplicationIT.java:267)
    

    For tracking added to meta #1715

    bug untriaged 
    opened by dreamer-89 0
  • Add additional logging for future debugging of HA proxy mode flaky test

    Add additional logging for future debugging of HA proxy mode flaky test

    Signed-off-by: Ryan Bogan [email protected]

    Description

    The current logging is not sufficient to fix flaky test failures for testHAProxyModeConnectionWorks . The failure occurs due to the number of connected sockets being 0. However, the RemoteConnectionInfo containing this data is obtained through a transport request, so the cause of the sockets not connecting is unknown. Since the error is non-reproducible, this PR adds additional logging when there are no connected sockets, printing out the cluster health at the time of failure. If the flaky test failure occurs again, there will be more logged information that can hopefully lead to a solution.

    Check List

    • [X] New functionality includes testing.
      • [X] All tests pass
    • [X] New functionality has been documented.
      • [X] New functionality has javadoc added
    • [X] Commits are signed per the DCO using --signoff
    • [X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    opened by ryanbogan 3
  • Update Gradle to 8.0-rc-1

    Update Gradle to 8.0-rc-1

    Signed-off-by: Andriy Redko [email protected]

    Description

    The first Gradle 8.0-rc-1 is out, starting the draft pull request to understand required efforts for migration.

    Issues Resolved

    N/A

    Check List

    • [ ] New functionality includes testing.
      • [ ] All tests pass
    • [ ] New functionality has been documented.
      • [ ] New functionality has javadoc added
    • [X] Commits are signed per the DCO using --signoff
    • [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    skip-changelog 
    opened by reta 1
  • Fix Graph Filter Error in Search

    Fix Graph Filter Error in Search

    Description:

    1. Reproducing the issue:
    • [1] . creating index:

    curl -XPUT localhost:9200/test-index --data '{ "settings": { "index": { "analysis": { "analyzer": { "search_analyzer": { "filter": [ "stop", "asciifolding", "elision", "lowercase", "synonym_graph", "snowball", "unique" ], "tokenizer": "standard", "type": "custom" }, "index_analyzer": { "filter": [ "stop", "asciifolding", "elision", "word_delimiter", "flatten_graph", "lowercase", "snowball", "unique" ], "tokenizer": "standard", "type": "custom" } }, "filter": { "synonym_graph": { "lenient": "true", "synonyms": [ "head board, bed head, bedhead, headboard" ], "type": "synonym_graph" }, "synonyms": { "synonyms": [ "head board, bed head, bedhead, headboard" ], "type": "synonym" }, "word_delimiter": { "preserve_original": "true", "type": "word_delimiter" } } } } }, "mappings": { "dynamic": "strict", "properties": { "merchandising_keywords": { "analyzer": "index_analyzer", "index_options": "docs", "search_analyzer": "search_analyzer", "type": "text" } } } } ' -H "Content-Type:Application/json"

    • [2 ] querying by curl 'localhost:9200/test-index/_search?pretty' --data '{"query": {"match": {"merchandising_keywords": "Gas Lift Storage Bed Frame with Arched Bed Head in King"}}}' -H "Content-Type:Application/json"
    • [3 ] outcome 'get 404 error' assertion error
    1. Testing with logging information. the Input of the function is a graph created from lucene, GraphTokenStreamFiniteStringsorg.apache.lucene.util.graph.GraphTokenStreamFiniteStrings. Found out when there is no document added, the graph has no path, only the source node. so while walking through the graph to find side path get assertion error, start =0, end = -1, state =0, next_state = 0

    [2022-12-29T15:07:35,251][INFO ][o.o.p.PluginsService     ] [runTask-0] PluginService:onIndexModule index:[test-index/jVsetp_JTSuPkUBIeKFuYg][2022-12-29T15:07:35,290][DEBUG][o.o.c.c.C.CoordinatorPublication] [runTask-0] publication ended successfully: Publication{term=1, version=3} [2022-12-29T15:07:35,423][DEBUG][o.o.c.c.PublicationTransportHandler] [runTask-0] received diff cluster state version [4] with uuid [aSNgAs0GRIK8AkGNiu_TjQ], diff size [620] [2022-12-29T15:07:35,453][DEBUG][o.o.c.c.C.CoordinatorPublication] [runTask-0] publication ended successfully: Publication{term=1, version=4} [2022-12-29T15:07:49,047][INFO ][o.o.i.s.MatchQuery       ] [runTask-0]  check the sourceCachingTokenFilter@52e91aa3 term=,bytes=[],startOffset=55,endOffset=55,positionIncrement=0,positionLength=1,type=word,termFrequency=1,keyword=false [2022-12-29T15:07:49,048][INFO ][o.o.i.s.MatchQuery       ] [runTask-0]  check the GraphTokenStreamFiniteStringsorg.apache.lucene.util.graph.GraphTokenStreamFiniteStrings@6389dd20 [2022-12-29T15:07:49,048][INFO ][o.o.i.s.MatchQuery       ] [runTask-0] articulationPoints.length: 0 [2022-12-29T15:07:49,048][INFO ][o.o.i.s.MatchQuery       ] [runTask-0] i: 0 [2022-12-29T15:07:49,049][INFO ][o.o.i.s.MatchQuery       ] [runTask-0] lastState: -1 [2022-12-29T15:07:49,049][INFO ][o.o.i.s.MatchQuery       ] [runTask-0] end: -1 [2022-12-29T15:07:49,049][INFO ][o.o.i.s.MatchQuery       ] [runTask-0] catch assertion error: java.lang.AssertionError: state=0 nextState=0

    1. Applying the fix to catch the error raising from lucene, and return empty query. Because there is no path found, no need to go further to analyze the graph, return empty query results.

    ** this is the query result as expected after the fix:

    curl 'localhost:9200/test-index/_search?pretty' --data '{"query": {"match": {"merchandising_keywords": "Gas Lift Storage Bed Frame with Arched Bed Head in King"}}}' -H "Content-Type:Application/json"

    { "took" : 68, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 0, "relation" : "eq" }, "max_score" : null, "hits" : [ ] } }

    Issues Resolved

    https://github.com/opensearch-project/OpenSearch/issues/5379

    Check List

    • [ ] New functionality includes testing.
      • [ ] All tests pass
    • [ ] New functionality has been documented.
      • [ ] New functionality has javadoc added
    • [ ] Commits are signed per the DCO using --signoff
    • [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    opened by mingshl 8
  • (Re-)Add coalesceToEmpty method to o.o.common.Strings util class

    (Re-)Add coalesceToEmpty method to o.o.common.Strings util class

    Is your feature request related to a problem? Please describe.

    A relatively common use case during REST handling is to replace a null String with an empty String:

    public static String coalesceToEmpty(@Nullable String s) {
        return s == null ? "" : s;
    }
    

    Such a method exists in ml-commons and anomaly-detection plugins and is being added to job-scheduler.

    This exists in the current release (2.4.1) of the Strings class here and is referenced by the alerting plugin but was removed in #5288 as "unused". (This broke the AD plugin which prompted re-adding its own method.)

    Describe the solution you'd like

    Don't repeat code. Add this method back to the Strings class.

    Additionally, I'm curious why this util class is marked "internal" as it's useful for dependent classes to use (which at least the alerting plugin does). The removal was not marked deprecated nor mentioned in the change log (likely because of its "internal" usage) but there is a lot of use of other methods in this class.

    Describe alternatives you've considered

    It may be a better idea to create a util class with all the RestHandlerUtils in the above-linked plugin packages.

    Additional context

    The method appears to have originated in com.google.common.base.Strings and was moved to the Strings class (in open-source ES pre-fork) to remove that external dependency.

    If we want to remove "internal" util classes/methods like this, we should consider either using external dependencies or creating a separate supported (API) project with util classes to provide the same functionality without duplicating code.

    enhancement untriaged 
    opened by dbwiddis 0
  • RemoteFSTranslog Trimming and GC Logic

    RemoteFSTranslog Trimming and GC Logic

    Signed-off-by: Gaurav Bafna [email protected]

    Description

    After any segment uploaded to remote store , we will update minSeqNoRequired for Translog. This is implemented only for RemoteFSTranslog .

    minSeqNoRequired will be used to preserve generations to be trimmed till it is uploaded to remote store.

    Once uploaded , post trimming, remote translog will be cleaned up as well along with local tlog files.

    Issues Resolved

    https://github.com/opensearch-project/OpenSearch/issues/5567

    Check List

    • [ ] New functionality includes testing.
      • [ ] All tests pass
    • [ ] New functionality has been documented.
      • [ ] New functionality has javadoc added
    • [ ] Commits are signed per the DCO using --signoff
    • [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

    opened by gbbafna 1
Releases(1.3.7)
  • 1.3.7(Dec 16, 2022)

  • 2.4.1(Dec 13, 2022)

  • 2.4.0(Nov 16, 2022)

  • 1.3.6(Oct 6, 2022)

  • 2.3.0(Sep 14, 2022)

  • 2.2.1(Sep 1, 2022)

  • 1.3.5(Sep 1, 2022)

  • 2.2.0(Aug 11, 2022)

  • 1.3.4(Jul 14, 2022)

  • 2.1.0(Jul 7, 2022)

  • 2.0.1(Jun 16, 2022)

  • 2.0.0(May 26, 2022)

  • 2.0.0-rc1(May 3, 2022)

    Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-2.0.0-rc1.md

    Full Changelog: https://github.com/opensearch-project/OpenSearch/compare/1.3.1...2.0.0-rc1

    Source code(tar.gz)
    Source code(zip)
  • 1.3.1(Apr 8, 2022)

    Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.3.1.md

    What's Changed

    • Bump the version to 1.3.1 by @zelinh in https://github.com/opensearch-project/OpenSearch/pull/2509
    • Exclude man page symlink in distribution by @andrross in https://github.com/opensearch-project/OpenSearch/pull/2602

    Full Changelog: https://github.com/opensearch-project/OpenSearch/compare/1.3.0...1.3.1

    Source code(tar.gz)
    Source code(zip)
  • 1.3.0(Mar 17, 2022)

    Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.3.0.md

    Full Changelog: https://github.com/opensearch-project/OpenSearch/compare/1.2.4...1.3.0

    Source code(tar.gz)
    Source code(zip)
  • 1.2.4(Jan 18, 2022)

    What's Changed

    • Prepare for next development iteration, 1.2.4 by @dblock in https://github.com/opensearch-project/OpenSearch/pull/1792
    • [Backport] [1.2] Update to log4j 2.17.1 (#1820) by @reta in https://github.com/opensearch-project/OpenSearch/pull/1823
    • [Backport] [1.2] RestIntegTestTask fails because of missed log4j-core dependency (#1815) by @reta in https://github.com/opensearch-project/OpenSearch/pull/1819
    • [Backport 1.2] Upgrading bouncycastle to 1.70 (#1832) by @saratvemulapalli in https://github.com/opensearch-project/OpenSearch/pull/1889
    • [Backport 1.2] Upgrading Netty to 4.1.72-Final by @saratvemulapalli in https://github.com/opensearch-project/OpenSearch/pull/1890
    • [Backport 1.2] Replace JCenter with Maven Central. (#1057) and update plugin repository order. by @mch2 in https://github.com/opensearch-project/OpenSearch/pull/1894
    • [1.2] Update FIPS API libraries of Bouncy Castle (#1853) by @tlfeng in https://github.com/opensearch-project/OpenSearch/pull/1888

    Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.2.4.md

    Full Changelog: https://github.com/opensearch-project/OpenSearch/compare/1.2.3...1.2.4

    Source code(tar.gz)
    Source code(zip)
  • 1.2.3(Jan 13, 2022)

    What's Changed

    • [backport] [1.2] [plugin] repository-azure is not working properly hangs on basic operations by @reta in https://github.com/opensearch-project/OpenSearch/pull/1745
    • Increment version to 1.2.3 by @dblock in https://github.com/opensearch-project/OpenSearch/pull/1758
    • [Backport] [1.2] Update to log4j 2.17.0 (#1771) by @reta in https://github.com/opensearch-project/OpenSearch/pull/1774

    Full Changelog: https://github.com/opensearch-project/OpenSearch/compare/1.2.2...1.2.3 Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes.1.2.3.md

    Source code(tar.gz)
    Source code(zip)
  • 1.2.2(Dec 16, 2021)

    What's Changed

    • Increment version to 1.2.2 and backport log4j upgrade to 2.16. by @dblock in https://github.com/opensearch-project/OpenSearch/pull/1728

    Full Changelog: https://github.com/opensearch-project/OpenSearch/compare/1.2.1...1.2.2 Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.2.2.md

    Source code(tar.gz)
    Source code(zip)
  • 1.2.1(Dec 14, 2021)

    What's Changed

    • Increment version to 1.2.1 by @dblock in https://github.com/opensearch-project/OpenSearch/pull/1700
    • [Backport 1.2] Upgrade to log4j 2.15.0 by @andrross in https://github.com/opensearch-project/OpenSearch/pull/1704

    Full Changelog: https://github.com/opensearch-project/OpenSearch/compare/1.2.0...1.2.1

    Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.2.1.md

    Source code(tar.gz)
    Source code(zip)
  • 1.2.0(Nov 23, 2021)

    Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.2.0.md

    Source code(tar.gz)
    Source code(zip)
  • 1.1.0(Oct 6, 2021)

    Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.1.0.md

    Source code(tar.gz)
    Source code(zip)
  • 1.0.0(Jul 13, 2021)

    Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.0.0.md

    Source code(tar.gz)
    Source code(zip)
  • 1.0.0-rc1(Jun 8, 2021)

    Downloads: https://opensearch.org/downloads.html Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.0.0-rc1.md

    Source code(tar.gz)
    Source code(zip)
  • 1.0.0-beta1(Apr 27, 2021)

  • 1.0.0-alpha2(Apr 26, 2021)

  • 1.0.0-alpha1(Apr 22, 2021)

Owner
null
filehunter - Simple, fast, open source file search engine

Simple, fast, open source file search engine. Designed to be local file search engine for places where multiple documents are stored on multiple hosts with multiple directories.

null 32 Sep 14, 2022
Apache Lucene and Solr open-source search software

Apache Lucene and Solr have separate repositories now! Solr has become a top-level Apache project and main line development for Lucene and Solr is hap

The Apache Software Foundation 4.3k Jan 7, 2023
🔍An open source GitLab/Gitee/Gitea code search tool. Kooder 是一个为 Gitee/GitLab 开发的开源代码搜索工具,这是一个镜像仓库,主仓库在 Gitee。

Kooder is a open source code search project, offering code, repositories and issues search service for code hosting platforms including Gitee, GitLab and Gitea.

开源中国 350 Dec 30, 2022
GitHub Search Engine: Web Application used to retrieve, store and present projects from GitHub, as well as any statistics related to them.

GHSearch Platform This project is made of two subprojects: application: The main application has two main responsibilities: Crawling GitHub and retrie

SEART - SoftwarE Analytics Research Team 68 Nov 25, 2022
A simple fast search engine written in java with the help of the Collection API which takes in multiple queries and outputs results accordingly.

A simple fast search engine written in java with the help of the Collection API which takes in multiple queries and outputs results accordingly.

Adnan Hossain 6 Oct 24, 2022
Apache Lucene is a high-performance, full featured text search engine library written in Java.

Apache Lucene is a high-performance, full featured text search engine library written in Java.

The Apache Software Foundation 1.4k Jan 5, 2023
Apache Solr is an enterprise search platform written in Java and using Apache Lucene.

Apache Solr is an enterprise search platform written in Java and using Apache Lucene. Major features include full-text search, index replication and sharding, and result faceting and highlighting.

The Apache Software Foundation 630 Dec 28, 2022
A proof-of-concept serverless full-text search solution built with Apache Lucene and Quarkus framework.

Lucene Serverless This project demonstrates a proof-of-concept serverless full-text search solution built with Apache Lucene and Quarkus framework. ✔️

Arseny Yankovsky 38 Oct 29, 2022
Simple full text indexing and searching library for Java

indexer4j Simple full text indexing and searching library for Java Install Gradle repositories { jcenter() } dependencies { compile 'com.haeun

Haeun Kim 47 May 18, 2022
Path Finding Visualizer for Breadth first search, Depth first search, Best first search and A* search made with java swing

Path-Finding-Visualizer Purpose This is a tool to visualize search algorithms Algorithms featured Breadth First Search Deapth First Search Gready Best

Leonard 11 Oct 20, 2022
Free and Open, Distributed, RESTful Search Engine

Elasticsearch A Distributed RESTful Search Engine https://www.elastic.co/products/elasticsearch Elasticsearch is a distributed RESTful search engine b

elastic 62.3k Dec 31, 2022
🔍 Open Source Enterprise Cognitive Search Engine

OpenK9 OpenK9 is a new Cognitive Search Engine that allows you to build next generation search experiences. It employs a scalable architecture and mac

SMC 24 Dec 10, 2022
filehunter - Simple, fast, open source file search engine

Simple, fast, open source file search engine. Designed to be local file search engine for places where multiple documents are stored on multiple hosts with multiple directories.

null 32 Sep 14, 2022
RESTEasy is a JBoss project that provides various frameworks to help you build RESTful Web Services and RESTful Java applications

RESTEasy RESTEasy is a JBoss.org project aimed at providing productivity frameworks for developing client and server RESTful applications and services

RESTEasy 1k Dec 23, 2022
OpenGrok is a fast and usable source code search and cross reference engine, written in Java

Copyright (c) 2006, 2020 Oracle and/or its affiliates. All rights reserved. OpenGrok - a wicked fast source browser OpenGrok - a wicked fast source br

Oracle 3.8k Jan 8, 2023
Search API with spelling correction using ngram-index algorithm: implementation using Java Spring-boot and MySQL ngram full text search index

Search API to handle Spelling-Corrections Based on N-gram index algorithm: using MySQL Ngram Full-Text Parser Sample Screen-Recording Screen.Recording

Hardik Singh Behl 5 Dec 4, 2021
Drools is a rule engine, DMN engine and complex event processing (CEP) engine for Java.

An open source rule engine, DMN engine and complex event processing (CEP) engine for Java™ and the JVM Platform. Drools is a business rule management

KIE (Drools, OptaPlanner and jBPM) 4.9k Dec 31, 2022
Apache Lucene and Solr open-source search software

Apache Lucene and Solr have separate repositories now! Solr has become a top-level Apache project and main line development for Lucene and Solr is hap

The Apache Software Foundation 4.3k Jan 7, 2023