Hi All,

I am looking for OpenGrok index update documentation, both manual and automated, please advise. Ideally I'd like to have it configured each time source files are updated, or on a scheduled basis. If not supported, manual option will do too as long as it doesn't rebuild index from scratch every time.

Thanks, YM

Hey, after indexing using java -Djava.util.logging.config.file=/opengrok/etc/logging.properties -jar /opengrok/dist/lib/opengrok.jar -c /usr/local/bin/ctags -s /opengrok/src -d /opengrok/data -H -P -S -G -W /opengrok/etc/configuration.xml -U http://localhost:8080/source I can't select projects to search on

also, I have attached my configration.xml configuration.zip

Thank you for your help

Hello,

Please consider for integration this patch with some improvements in error handling and changes to reduce the time interval where the Lucene database is inconsistent with the xref file data and to speed up deletions for issue #1908.

1. Add PendingFileCompleter and use in IndexDatabase to queue up file deletions and file renamings to be executed during the Lucene 2-phase commit. IndexDatabase is changed to write to in-progress files (i.e., with an "._org_opengrok" extension) while xref-ing and to queue up the moves to final names (without the pending extension) during the 2-phase commit. This minimizes the time interval of inconsistency between the xref data and the committed Lucene data.

2. Do not run Lucene commit() on each file deletion, but instead once as the last step of IndexDatabase.update() so that Lucene and PendingFileCompleter can be well-coordinated.

3. Do not swallow IndexWriter exceptions for prepareCommit(), commit(), or close() in either the IndexDatabase update() or optimize() methods.
4. Support NativeFSLockFactory, and support the --lock switch via the OpenGrok script (on|off|native|simple with the default still off, and on as an alias for simple). Configuration is revised to support the new lock mode, with the existing boolean usingLuceneLocking preserved but @Deprecated to be retired at some future point — i.e., after a sufficient upgrade time interval is deemed to have past.
5. Ensure a meaningful non-zero exit code if a non-fatal error occurs in the OpenGrok script.

Thank you.

Environment: OpenGrook 0.12.1, Linux, Java 7 64-bit, >JAVA_OPTS="-Xmx16g -Xms16g"<

Scanning about 300 projects, small , to medium size, total 20GB of all source code Scan results with "java.lang.OutOfMemoryError: Requested array size exceeds VM limit"

Describe the bug I have a project jdk8-b120 which is causing opengrok server to shutdown.

I want to disable the sync and reindex for this project. Please find mirror.yml below

# Empty config file for opengrok-mirror
ignore_errors: true

projects:
  jdk8.*:
    disabled: true
    incoming_check: false

Despite doing this I still see reindex getting triggered(screenshot below) for this project. What am I missing here? Also, how I can find what is causing the shutdown? Can suggester be the reason for the same?

I have used 1.4 for quite sometime and never had any such issues.

Screenshots

Today I think I hit a race condition when doing the parallel indexing and using the sync.py script (mostly from the wiki).

My flow is following:

1. Delete all unwanted projects with projadm.py
2. Download all sources from the SCM
3. Add all projects with projadm.py (no index at this step)
4. Generate groups and publish them to the webapp
5. Index the sources using sync.py script with mirroring.

I'm using 10 repositories, some of them take longer time to index but most of them can finish in almost the same time. I'm using the number of workers 4.

When the sync.py script is finished, with no errors, the page itself displays only 9 (sometimes 8) projects most of the time, with one of them missing randomly. After looking in the webapp configuration the project itself is not marked as indexed (therefore not visible).

However, looking at the indexer logs, there is a 10 times a mention that the uri projects/{p}/indexed has been fired and return 204.

-bash-4.1$ cat /opengrok/log/**/opengrok0.0.log | grep "indexed HTTP/1.1" | wc -l
10

Also looking at the tcpdump communication (posted below), I can see 10 calls for the indexed url.

I couldn't dig any deeper at this point, but there's something wrong going on.

Sync-config.yml:

commands:
- command:
  - http://localhost:8150/source/api/v1/messages
  - POST
  - cssClass: info
    duration: PT2H
    tags: ['%PROJECT%']
    text: resync + reindex in progress
- command: [ /opengrok/bin/mirror.py, -U, 'http://localhost:8150/source', -c, /opengrok/etc/mirror-config.yml ]
- command: [
    /opengrok/bin/reindex-project.py,
    -j,
    '/usr/java/jdk1.8.0_60/bin/java',
    -J=-d64,
    '-J=-XX:-UseGCOverheadLimit',
    -J=-Xmx12g,
    -J=-server,
    -J=-Dorg.opengrok.indexer.history.git=/opt/git/git283/bin/git,
    --jar,
    /opengrok/lib/opengrok.jar,
    -t,
    /opengrok/etc/logging.properties.template,
    -p,
    '%PROJ%',
    -d,
    /opengrok/log/%PROJECT%,
    -P,
    '%PROJECT%',
    --uri,
    'http://localhost:8150/source',
    --,
    -U,
    'http://localhost:8150/source',
    --renamedHistory,
    'off',
    -r,
    dirbased,
    --memory,
    '256',
    -c,
    /opt/universal_ctags/bin/ctags,
    -i,
    'Assets.webtest',
    -P,
    -H,
    '%PROJECT%',
  ]
  env: {LC_ALL: en_US.UTF-8}
- command: ['http://localhost:8150/source/api/v1/messages?tag=%PROJECT%', DELETE, '']
cleanup:
  command: ['http://localhost:8150/source/api/v1/messages?tag=%PROJECT%', DELETE, '']

Tcpdump:

[root@notebook]# tcpdump -s 0 -A -i lo -v -l -n  port 8150 | egrep -i "POST /|GET /|PUT /|DELETE /|Host:"
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
..@o..@oPOST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
..@p..@pPOST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
..@z..@zPOST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
..@[email protected] /source/api/v1/messages HTTP/1.1
Host: localhost:8150
..C'..C'GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
..C3..C3GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
..CZ..CZGET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
..Cm..CmGET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
..C...C.GET /source/api/v1/projects/api.fuel/repositories HTTP/1.1
Host: localhost:8150
..C...C.GET /source/api/v1/repositories/type?repository=%2Fapi.fuel HTTP/1.1
Host: localhost:8150
..C...C.GET /source/api/v1/projects/api.apitest/repositories HTTP/1.1
Host: localhost:8150
..D...D.GET /source/api/v1/repositories/type?repository=%2Fapi.apitest HTTP/1.1
Host: localhost:8150
..D%..D$GET /source/api/v1/projects/api.avails/repositories HTTP/1.1
Host: localhost:8150
..D)..D)GET /source/api/v1/repositories/type?repository=%2Fapi.avails HTTP/1.1
Host: localhost:8150
..D7..D7GET /source/api/v1/projects/api.analytics/repositories HTTP/1.1
Host: localhost:8150
..D;..D;GET /source/api/v1/repositories/type?repository=%2Fapi.analytics HTTP/1.1
Host: localhost:8150
..G...G.GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
..HI..HIGET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
..H...H.GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
..J...J.GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
..O...O.PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
..Pt..PqPUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
..P...P.PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
..Sc..S`PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
..{E..{DPUT /source/api/v1/projects/api.apitest/indexed HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/projects/api.avails/indexed HTTP/1.1
Host: localhost:8150
...@..{VPUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...v...rDELETE /source/api/v1/messages?tag=api.apitest HTTP/1.1
Host: localhost:8150
...~...~POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
...V...VGET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/projects/api.fuel_js/repositories HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/repositories/type?repository=%2Fapi.fuel_js HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...T...TDELETE /source/api/v1/messages?tag=api.avails HTTP/1.1
Host: localhost:8150
..._..._POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
...a...aGET /source/api/v1/projects/api.navajo/repositories HTTP/1.1
Host: localhost:8150
...g...fGET /source/api/v1/repositories/type?repository=%2Fapi.navajo HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
...'....PUT /source/api/v1/projects/api.fuel_js/indexed HTTP/1.1
Host: localhost:8150
.......6PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
........DELETE /source/api/v1/messages?tag=api.fuel_js HTTP/1.1
Host: localhost:8150
........POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
...e...eGET /source/api/v1/projects/api.orion/repositories HTTP/1.1
Host: localhost:8150
...k...kGET /source/api/v1/repositories/type?repository=%2Fapi.orion HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
...,...#PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
...>...=PUT /source/api/v1/projects/api.analytics/indexed HTTP/1.1
Host: localhost:8150
...m...hPUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...j...fDELETE /source/api/v1/messages?tag=api.analytics HTTP/1.1
Host: localhost:8150
...r...rPOST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/projects/api.orion-gateway/repositories HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/repositories/type?repository=%2Fapi.orion-gateway HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
...t...kPUT /source/api/v1/projects/api.orion/indexed HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
.......TPUT /source/api/v1/projects/api.orion-gateway/indexed HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...-...%DELETE /source/api/v1/messages?tag=api.orion-gateway HTTP/1.1
Host: localhost:8150
...9...1POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...@...@GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
........DELETE /source/api/v1/messages?tag=api.orion HTTP/1.1
Host: localhost:8150
........POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
...m...mGET /source/api/v1/projects/api.orionapitest/repositories HTTP/1.1
Host: localhost:8150
...u...qGET /source/api/v1/repositories/type?repository=%2Fapi.orionapitest HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/projects/api.oxy/repositories HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/repositories/type?repository=%2Fapi.oxy HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
...q...mPUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
...k....PUT /source/api/v1/projects/api.orionapitest/indexed HTTP/1.1
Host: localhost:8150
.. ... .PUT /source/api/v1/projects/api.fuel/indexed HTTP/1.1
Host: localhost:8150
..&.....PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
..)...).DELETE /source/api/v1/messages?tag=api.orionapitest HTTP/1.1
Host: localhost:8150
..N...N.PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
..O[..O[DELETE /source/api/v1/messages?tag=api.fuel HTTP/1.1
Host: localhost:8150
..O...O.PUT /source/api/v1/projects/api.oxy/indexed HTTP/1.1
Host: localhost:8150
..`#..O.PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
..a<..a<DELETE /source/api/v1/messages?tag=api.oxy HTTP/1.1
Host: localhost:8150
...{...yPUT /source/api/v1/projects/api.navajo/indexed HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...t...tDELETE /source/api/v1/messages?tag=api.navajo HTTP/1.1
Host: localhost:8150
[root@notebook]# 

Hey :)

After indexing using this command java -Djava.util.logging.config.file=/opengrok/etc/logging.properties -jar /opengrok/dist/lib/opengrok.jar --ctags /usr/local/bin/ctags -s /opengrok/src -d /opengrok/data -H -P -S -G -W /opengrok/etc/configuration.xml -U http://localhost:8080/source/ some files can't be open although it could be read and exist in src folder

Can't find anything special in log, but when checked found this for almost all projects in catalina.out SEVERE [ForkJoinPool-331-worker-19] org.opengrok.suggest.Suggester.lambda$getInitRunnable$1 Could not initialize suggester data for aftershock java.io.FileNotFoundException: /opengrok/data/suggester/aftershock/defs.wfst (Permission denied) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187) at org.opengrok.suggest.SuggesterProjectData.store(SuggesterProjectData.java:273) at org.opengrok.suggest.SuggesterProjectData.build(SuggesterProjectData.java:250) at org.opengrok.suggest.SuggesterProjectData.init(SuggesterProjectData.java:154) at org.opengrok.suggest.Suggester.lambda$getInitRunnable$1(Suggester.java:229) at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

Appreciate your help Basel

I'm running in a container based on a docker image built from opengrok/docker, and an altered index.sh that currently looks like this:

#!/bin/bash

LOCKFILE=/var/run/opengrok-indexer
URI="http://localhost:8080"
OPS=${INDEXER_FLAGS:='-H -P -G'}

if [ -f "$LOCKFILE" ]; then
        date +"%F %T Indexer still locked, skipping indexing"
        exit 1
fi

touch $LOCKFILE

date +"%F %T Indexing starting"
opengrok-indexer \
    -J=-Xmx8g \
    -J=-server \
    -a /opengrok/lib/opengrok.jar -- \
    -i hugeMatlabFile.mat \
    -m 256 \
    -v \
    --repository /opengrok/src/master \
    -s /opengrok/src \
    -d /opengrok/data \
    --remote on \
    -W /opengrok/etc/configuration.xml \
    -U "$URI" \
    $OPS \
    $INDEXER_OPT "$@"
date +"%F %T Indexing finished"

The index forms ok, and seems fine to a cursory examination, but the history link doesn't work, and there's no list of "current version" with last commit on the web app front page, which I get with the 1.5 year old version, if I run it on the same files.

The only severe I found in the log is a handful of these:

WARNING: CTags parsing problem: java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) at java.io.BufferedInputStream.read(BufferedInputStream.java:336) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) at java.io.BufferedReader.readLine(BufferedReader.java:389) at org.opengrok.indexer.analysis.Ctags.readTags(Ctags.java:547) at org.opengrok.indexer.analysis.Ctags.lambda$doCtags$2(Ctags.java:455) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Mar 16, 2020 11:31:08 PM org.opengrok.indexer.analysis.Ctags readTags SEVERE: CTag reader cycle was interrupted!

Is this a settings issue? Did I put in the wrong flag somewhere? I mean, as far as I understand it, the -H means history is on, no?

Or is it the severe?

Either way, please advise... and please excuse my being a pain.

status NEW severity minor in component scm for --- Reported in version unspecified on platform ANY/Generic Assigned to: Trond Norbye

On 2012-02-05 19:09:36 +0000, Johan wrote:

phb@opengrok:/var/lib/opengrok/src/opstools-misc$ git log fatal: bad default revision 'HEAD'

java.lang.RuntimeException: Failed to acquire random test lock; please verify filesystem for lock directory '/var/opengrok/data/spellIndex/opstools-misc' supports locking at org.apache.lucene.store.NativeFSLockFactory.acquireTestLock(NativeFSLockFactory.java:103) at org.apache.lucene.store.NativeFSLockFactory.makeLock(NativeFSLockFactory.java:142) at org.apache.lucene.store.Directory.makeLock(Directory.java:106) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1058) at org.apache.lucene.index.IndexWriter.(IndexWriter.java:882) at org.apache.lucene.search.spell.SpellChecker.setSpellIndex(SpellChecker.java:142) at org.apache.lucene.search.spell.SpellChecker.(SpellChecker.java:110) at org.apache.lucene.search.spell.SpellChecker.(SpellChecker.java:124) at org.apache.jsp.search_jsp._jspService(search_jsp.java:689) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.lucene.store.LockReleaseFailedException: Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /var/opengrok/data/spellIndex/opstools-misc/lucene-....--3y5zar-test.lock at org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:329) at org.apache.lucene.store.NativeFSLockFactory.acquireTestLock(NativeFSLockFactory.java:95) ... 26 more

phb@opengrok:~$ ls -al /var/opengrok/data/spellIndex/opstools-misc total 24 drwxr-xr-x 2 opengrok opengrok 4096 Jan 31 02:33 . drwxr-xr-x 548 opengrok opengrok 20480 Feb 4 02:34 ..

Workaround is simple, ensure no repositories like that exist in the directory that is being indexed. (We use git show-ref -q --verify refs/remotes/origin/master )

On 2012-02-06 08:28:43 +0000, Lubos Kosco wrote:

this is a bit weird I think it shouldn't happen with latest binaries

afaik our executors have a timeout on forked commands

below stack shows this: an indexer is stuck indexing "opstools-misc" and another indexer was started in between (within maybe the timeout of above)

so basically you have 2 indexers running on the same box and they seem to clash, check ps axf (or ptree) and let us know

• locking on lucene is supposed to be automatic, I removed forced options on lock clean ups because of that (and no one complained, so you're the first one)

the other Q can be, whether you use latest opengrok ;)

hmm? L

note that current 0.11rc and 0.10 didn't have big changes in this area, so the locking should be the same, however older opengrok versions had old locking, which was problematic sometimes

On 2012-02-06 09:43:10 +0000, Johan wrote:

(In reply to comment # 1)

this is a bit weird I think it shouldn't happen with latest binaries

It is using opengrok 0.11, or at the very least 0.10 if I failed the upgrade somehow (There seems to be no UI-visible version # to verify?)

afaik our executors have a timeout on forked commands

below stack shows this: an indexer is stuck indexing "opstools-misc" and another indexer was started in between (within maybe the timeout of above)

so basically you have 2 indexers running on the same box and they seem to clash, check ps axf (or ptree) and let us know

No indexers running at all. It happened for us on 5 repositories, all exhibiting the above problem, so I'm really suspecting something w.r.t the git scm history.

All the other directories under /var/opengrok/data/spellIndex has a couple of files ( 672 -rw-r--r-- 1 opengrok opengrok 684629 Nov 28 23:07 _0.cfs 128 -rw-r--r-- 1 opengrok opengrok 130345 Nov 28 23:07 _0.cfx 4 -rw-r--r-- 1 opengrok opengrok 20 Nov 28 23:07 segments.gen 4 -rw-r--r-- 1 opengrok opengrok 246 Nov 28 23:07 segment)

• locking on lucene is supposed to be automatic, I removed forced options on lock clean ups because of that (and no one complained, so you're the first one)

the other Q can be, whether you use latest opengrok ;) Well, most probably 0.11rc2 unless my installation failed and then it would be running 0.10.

hmm? L

note that current 0.11rc and 0.10 didn't have big changes in this area, so the locking should be the same, however older opengrok versions had old locking, which was problematic sometimes

Hello.

I wan't use OpenGrok for create repository browser on Tomcat server on my Ubuntu. Indexer works fine - but the problem I don't know how integrate it with Perforce.

I run this command: $JAVA -Xms2G -Xmx16G -Dorg.opensolaris.opengrok.history.Perforce=/usr/bin/p4 -jar "${OpenGrokJar}" -r on -H -m 2048 -c "$CTAGS" -w "myProject" -s "$VIEW" -d "$DB_DIR50" $IGNORE_PATTERNS $VERBOSE_PROGRES -z 1000 -t 4 -P -vvv -W "$CONF_FILE" -a on -O on

I was run this command : p4 dir /$VIEW and it return that this location is under P4 Client.

In log file I have lines: INFO: Generating history cache for all repositories ... Sep 27, 2018 10:20:14 AM org.opensolaris.opengrok.history.HistoryGuru createCacheReal INFO: Creating historycache for 0 repositories INFO: Done historycache for all repositories (took 6 ms)

Please, could you point me how integrate OpenGrok with Perforce - step by step.

Is your feature request related to a problem? Please describe. Let's write yes. I am reindexing all my company Git repos (30k) with latest OG 1.5.11 / Tomcat 9 / Java 11 / RHEL 8.3. Historycache generation is extremely long...

Describe the solution you'd like I would like when reindexing: 1a run indexer without historycache 1b run another indexer process in parallel with historycache only that's to say skipping file indexing since this is done in a short time by process of step 1a 2 each night run indexer without historycache 3 when historycache started in step 1b finishes ok, then and only then, update job run nightly in step 2 adding historycache too

Describe alternatives you've considered Optimizing the historycache generation? Issue to be opened further to your answer to this one. But in all case, because of the size of our code sources, historycache process has always been very long.

Additional context Some of our repos have very long history, with many tags and also many files. This does not help generating quickly the history.

Specifically, Docker build fails when running ./dev/dockerhub_readme.py with:

INFO:__main__:DOCKER_USERNAME is empty, exiting
Error: Process completed with exit code 1.

Seen e.g. in #4139.

The ConfigMerge tool should make sure that (canonicalized) paths for the base and new files are different to avoid nasty surprises.

Inspired by https://github.com/oracle/opengrok/discussions/4133#discussioncomment-4402540_

Describe the bug I have a Opengrok instance, but the Scope and Navigate windows don't work in my main instance.

Opengrok: v1.7.38 Tomcat: v10..

I just get an empty window for any project I open.

To Reproduce I have two instances, one with 7 projects(lets call it dev instance) and the other with ~1300 projects( lets call it qa instance), more to come. In the dev instance, scope doesn't work properly but navigation works, but in the qa instance, nothing works, even the syntax highlighting is not clearly there in the qa instance, but in dev instance, it seems to be correct.

Expected behavior Working Scope and Navigation Window

Additional context This seems to happen during indexing I think, but I am not sure what do I need to change to have this working.

As identified in https://github.com/oracle/opengrok/discussions/4118#discussioncomment-4346257, for parallel scan the scanning needs to take into account that it is starting one directory below source root.

When working on #4075, I noticed that some tests from SuggesterControllerTest fail occasionally, namely these:

• testPhraseAfter
• testPhraseBefore
• testPhraseMiddle

And also the testSloppyPhrase, however that one I have seen in Github Action failed builds before I think, unlike these 3, although they might share common root cause.

Interestingly, these can be stepped through in a debugger and they would still sometimes fail. The failures look like this:

java.lang.AssertionError: 
Expected: iterable containing ["file"]
     but: no item was "file"

	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at org.opengrok.web.api.v1.controller.SuggesterControllerTest.testPhraseAfter(SuggesterControllerTest.java:335)

In this case the API call got empty result.

When tracing the failed tests, it was always caused by interrupted being set to true in SuggesterSearcher, e.g. https://github.com/oracle/opengrok/blob/c10182859ee0b2d541135b28e90df09aca1a13d7/suggester/src/main/java/org/opengrok/suggest/SuggesterSearcher.java#L140-L143 however also in different locations where Thread.currentThread().isInterrupted() or the interrupted field is checked.

The thread being interrupted comes from https://github.com/oracle/opengrok/blob/c10182859ee0b2d541135b28e90df09aca1a13d7/suggester/src/main/java/org/opengrok/suggest/Suggester.java#L107-L115

I tried commenting out the https://github.com/oracle/opengrok/blob/c10182859ee0b2d541135b28e90df09aca1a13d7/suggester/src/main/java/org/opengrok/suggest/Suggester.java#L620 and significantly bumping the timeout in https://github.com/oracle/opengrok/blob/c10182859ee0b2d541135b28e90df09aca1a13d7/suggester/src/main/java/org/opengrok/suggest/Suggester.java#L466 however neither helped.

Observing a opengrok-mirror run for a bunch of non-local Mercurial repositories, with the -I option, I noticed that each repository takes couple of seconds to check (via repo.incoming()). Since the list of repositories is known beforehand in utils/mirror.py#process_changes(), this piece of code could be parallelized: https://github.com/oracle/opengrok/blob/c10182859ee0b2d541135b28e90df09aca1a13d7/tools/src/main/python/opengrok_tools/utils/mirror.py#L324-L330

The top-level repo check needs some thought, though. Also, will need to take care of error reporting. There is no exception (re)thrown, however will need to make sure that errors are properly returned as FAILURE_EXITVAL.

Lastly, will need to determine the parallelism level.

Similarly, same should be done for the repository synchronization part.