Today I think I hit a race condition when doing the parallel indexing and using the sync.py script (mostly from the wiki).
My flow is following:
- Delete all unwanted projects with projadm.py
- Download all sources from the SCM
- Add all projects with projadm.py (no index at this step)
- Generate groups and publish them to the webapp
- Index the sources using sync.py script with mirroring.
I'm using 10 repositories, some of them take longer time to index but most of them can finish in almost the same time. I'm using the number of workers 4.
When the sync.py script is finished, with no errors, the page itself displays only 9 (sometimes 8) projects most of the time, with one of them missing randomly. After looking in the webapp configuration the project itself is not marked as indexed (therefore not visible).
However, looking at the indexer logs, there is a 10 times a mention that the uri projects/{p}/indexed
has been fired and return 204.
-bash-4.1$ cat /opengrok/log/**/opengrok0.0.log | grep "indexed HTTP/1.1" | wc -l
10
Also looking at the tcpdump communication (posted below), I can see 10 calls for the indexed url.
I couldn't dig any deeper at this point, but there's something wrong going on.
Sync-config.yml:
commands:
- command:
- http://localhost:8150/source/api/v1/messages
- POST
- cssClass: info
duration: PT2H
tags: ['%PROJECT%']
text: resync + reindex in progress
- command: [ /opengrok/bin/mirror.py, -U, 'http://localhost:8150/source', -c, /opengrok/etc/mirror-config.yml ]
- command: [
/opengrok/bin/reindex-project.py,
-j,
'/usr/java/jdk1.8.0_60/bin/java',
-J=-d64,
'-J=-XX:-UseGCOverheadLimit',
-J=-Xmx12g,
-J=-server,
-J=-Dorg.opengrok.indexer.history.git=/opt/git/git283/bin/git,
--jar,
/opengrok/lib/opengrok.jar,
-t,
/opengrok/etc/logging.properties.template,
-p,
'%PROJ%',
-d,
/opengrok/log/%PROJECT%,
-P,
'%PROJECT%',
--uri,
'http://localhost:8150/source',
--,
-U,
'http://localhost:8150/source',
--renamedHistory,
'off',
-r,
dirbased,
--memory,
'256',
-c,
/opt/universal_ctags/bin/ctags,
-i,
'Assets.webtest',
-P,
-H,
'%PROJECT%',
]
env: {LC_ALL: en_US.UTF-8}
- command: ['http://localhost:8150/source/api/v1/messages?tag=%PROJECT%', DELETE, '']
cleanup:
command: ['http://localhost:8150/source/api/v1/messages?tag=%PROJECT%', DELETE, '']
Tcpdump:
[root@notebook]# tcpdump -s 0 -A -i lo -v -l -n port 8150 | egrep -i "POST /|GET /|PUT /|DELETE /|Host:"
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
..@o..@oPOST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
..@p..@pPOST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
..@z..@zPOST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
..@[email protected] /source/api/v1/messages HTTP/1.1
Host: localhost:8150
..C'..C'GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
..C3..C3GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
..CZ..CZGET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
..Cm..CmGET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
..C...C.GET /source/api/v1/projects/api.fuel/repositories HTTP/1.1
Host: localhost:8150
..C...C.GET /source/api/v1/repositories/type?repository=%2Fapi.fuel HTTP/1.1
Host: localhost:8150
..C...C.GET /source/api/v1/projects/api.apitest/repositories HTTP/1.1
Host: localhost:8150
..D...D.GET /source/api/v1/repositories/type?repository=%2Fapi.apitest HTTP/1.1
Host: localhost:8150
..D%..D$GET /source/api/v1/projects/api.avails/repositories HTTP/1.1
Host: localhost:8150
..D)..D)GET /source/api/v1/repositories/type?repository=%2Fapi.avails HTTP/1.1
Host: localhost:8150
..D7..D7GET /source/api/v1/projects/api.analytics/repositories HTTP/1.1
Host: localhost:8150
..D;..D;GET /source/api/v1/repositories/type?repository=%2Fapi.analytics HTTP/1.1
Host: localhost:8150
..G...G.GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
..HI..HIGET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
..H...H.GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
..J...J.GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
..O...O.PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
..Pt..PqPUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
..P...P.PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
..Sc..S`PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
..{E..{DPUT /source/api/v1/projects/api.apitest/indexed HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/projects/api.avails/indexed HTTP/1.1
Host: localhost:8150
...@..{VPUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...v...rDELETE /source/api/v1/messages?tag=api.apitest HTTP/1.1
Host: localhost:8150
...~...~POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
...V...VGET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/projects/api.fuel_js/repositories HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/repositories/type?repository=%2Fapi.fuel_js HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...T...TDELETE /source/api/v1/messages?tag=api.avails HTTP/1.1
Host: localhost:8150
..._..._POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
...a...aGET /source/api/v1/projects/api.navajo/repositories HTTP/1.1
Host: localhost:8150
...g...fGET /source/api/v1/repositories/type?repository=%2Fapi.navajo HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
...'....PUT /source/api/v1/projects/api.fuel_js/indexed HTTP/1.1
Host: localhost:8150
.......6PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
........DELETE /source/api/v1/messages?tag=api.fuel_js HTTP/1.1
Host: localhost:8150
........POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
...e...eGET /source/api/v1/projects/api.orion/repositories HTTP/1.1
Host: localhost:8150
...k...kGET /source/api/v1/repositories/type?repository=%2Fapi.orion HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
...,...#PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
...>...=PUT /source/api/v1/projects/api.analytics/indexed HTTP/1.1
Host: localhost:8150
...m...hPUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...j...fDELETE /source/api/v1/messages?tag=api.analytics HTTP/1.1
Host: localhost:8150
...r...rPOST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/projects/api.orion-gateway/repositories HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/repositories/type?repository=%2Fapi.orion-gateway HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
...t...kPUT /source/api/v1/projects/api.orion/indexed HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
.......TPUT /source/api/v1/projects/api.orion-gateway/indexed HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...-...%DELETE /source/api/v1/messages?tag=api.orion-gateway HTTP/1.1
Host: localhost:8150
...9...1POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...@...@GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
........DELETE /source/api/v1/messages?tag=api.orion HTTP/1.1
Host: localhost:8150
........POST /source/api/v1/messages HTTP/1.1
Host: localhost:8150
...m...mGET /source/api/v1/projects/api.orionapitest/repositories HTTP/1.1
Host: localhost:8150
...u...qGET /source/api/v1/repositories/type?repository=%2Fapi.orionapitest HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration/sourceRoot HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/projects/api.oxy/repositories HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/repositories/type?repository=%2Fapi.oxy HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
........GET /source/api/v1/configuration HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
...q...mPUT /source/api/v1/configuration/projectsEnabled HTTP/1.1
Host: localhost:8150
...k....PUT /source/api/v1/projects/api.orionapitest/indexed HTTP/1.1
Host: localhost:8150
.. ... .PUT /source/api/v1/projects/api.fuel/indexed HTTP/1.1
Host: localhost:8150
..&.....PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
..)...).DELETE /source/api/v1/messages?tag=api.orionapitest HTTP/1.1
Host: localhost:8150
..N...N.PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
..O[..O[DELETE /source/api/v1/messages?tag=api.fuel HTTP/1.1
Host: localhost:8150
..O...O.PUT /source/api/v1/projects/api.oxy/indexed HTTP/1.1
Host: localhost:8150
..`#..O.PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
..a<..a<DELETE /source/api/v1/messages?tag=api.oxy HTTP/1.1
Host: localhost:8150
...{...yPUT /source/api/v1/projects/api.navajo/indexed HTTP/1.1
Host: localhost:8150
........PUT /source/api/v1/system/refresh HTTP/1.1
Host: localhost:8150
...t...tDELETE /source/api/v1/messages?tag=api.navajo HTTP/1.1
Host: localhost:8150
[root@notebook]#
bug