Mp4grep - a CLI for transcribing and searching audio/video files

Hi there, I struggle on the usage of your tool. After the installation of the last release, I encounter the following error mp4grep --model $MP4GREP_MODEL --transcribe mytestfile.mp3

Exception in thread "Thread-2" com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected ':' at line 3 column 25 path $.result[0].861850
	at com.google.gson.Gson.fromJson(Gson.java:947)
	at com.google.gson.Gson.fromJson(Gson.java:897)
	at com.google.gson.Gson.fromJson(Gson.java:846)
	at com.google.gson.Gson.fromJson(Gson.java:817)
	at Transcribe.VoskAdapter.getJsonObject(VoskAdapter.java:163)
	at Transcribe.VoskAdapter.writeToCacheFiles(VoskAdapter.java:139)
	at Transcribe.VoskAdapter.writeMainRecognizerResults(VoskAdapter.java:120)
	at Transcribe.VoskAdapter.transcribeAudio(VoskAdapter.java:40)
	at Transcribe.VoskProxy.getSearchableTranscript(VoskProxy.java:14)
	at Transcribe.Cache.TranscriptCache.getSearchable(TranscriptCache.java:43)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:960)
	at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:934)
	at java.base/java.util.stream.AbstractTask.compute(AbstractTask.java:327)
	at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:686)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:927)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at Transcribe.TranscriptAdapter.lambda$getTranscription$1(TranscriptAdapter.java:140)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.google.gson.stream.MalformedJsonException: Expected ':' at line 3 column 25 path $.result[0].861850
	at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1562)
	at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:529)
	at com.google.gson.stream.JsonReader.peek(JsonReader.java:424)
	at com.google.gson.internal.bind.TypeAdapters$29.read(TypeAdapters.java:700)
	at com.google.gson.internal.bind.TypeAdapters$29.read(TypeAdapters.java:723)
	at com.google.gson.internal.bind.TypeAdapters$29.read(TypeAdapters.java:715)
	at com.google.gson.internal.bind.TypeAdapters$29.read(TypeAdapters.java:723)
	at com.google.gson.internal.bind.TypeAdapters$29.read(TypeAdapters.java:698)
	at com.google.gson.internal.bind.TypeAdapters$35$1.read(TypeAdapters.java:894)
	at com.google.gson.Gson.fromJson(Gson.java:932)
	... 24 more

I tried different audio files and different models, always with this error. If I call the tool again, it just raises 100% transcription, but no text is plottet.

im using debian 11 and ive downloaded 0.1.1 and extracted it. i cd into the folder and did source install.sh pls see below the output

readlink: invalid option -- 'b'
Try 'readlink --help' for more information.
Completed environment setup for mp4grep:
MP4GREP_CACHe=./.mp4grep_cache
MP4GREP_MODEL=./model
PATH=$PATH:./bin

Variable exported in ~/.bashrc

i tried running mp4grep and it says command not found

Hello, great tool, I really like it. This is maybe user error, but I'm running into a problem when feeding it a directory:

$ ls test
test1.wav*  test2.wav*  test3.wav*

$ mp4grep "test" test/
test3.wav does not exist, not transcribing.
test2.wav does not exist, not transcribing.
test1.wav does not exist, not transcribing.

If I point it directly to one of the files, it works as expected, but pointing it to a directory doesn't seem to work.

I encounter the same behavior with the prebuilt binary as well as one I build myself. I'm using version 0.1.4 on Arch.

Heard about this project on Linux Unplugged, wanted to give it a try. I use zsh, and the install.sh butchered my prompt by trying to source ~/.bashrc into a zsh session. I re-wrote the script to detect if the user is running bash or zsh and change the output file accordingly, but this install.sh doesn't seem to be in the git tree anywhere so I wasn't able to make a pull request.

Also, before editing a user's bashrc/zshrc we should give them the opportunity to do it manually if they wish. I know a lot of people (myself included) have a lot of custom stuff in their shell startup and the thought of an automated script messing with it always makes me nervous. (Yes, this script just appends lines to the end, but still.)

Here's what I wrote, it could definitely be improved upon but I wanted to at least pass on the two minutes of work I did!

#!/bin/sh

SCRIPT_DIRECTORY=$(dirname "$(readlink -f "$0")")

if [ "$SHELL" = "/bin/bash" ]; then
    RCFILE=~/.bashrc
elif [ "$SHELL" = "/usr/bin/zsh" ]; then
    RCFILE=~/.zshrc
else
    RCFILE=~/.profile # if we can't detect the shell, use a sane default that all modern shells look at
fi

echo "This installation script will try and set things up automatically."
echo "In order to do so, some environment variables will be appended to $RCFILE"
echo "If you want to do this manually, hit CTRL-C now and examine this script."
echo "Otherwise, just hit enter and we'll do the work for you!"
read wait # just wait for the user to read

echo "" >> $RCFILE # in case the rcfile doesn't end with a newline, forcefully add one
echo "# mp4grep environment variables -------" >> $RCFILE
echo "export MP4GREP_CACHE='$SCRIPT_DIRECTORY/.mp4grep_cache'" >> $RCFILE
echo "export MP4GREP_MODEL='$SCRIPT_DIRECTORY/model'" >> $RCFILE

echo -n 'export PATH="$PATH:' >> $RCFILE
echo "$SCRIPT_DIRECTORY/bin\"" >> $RCFILE
echo "# --------------------------------------" >> $RCFILE

echo "Completed environment setup for mp4grep: "
echo "MP4GREP_CACHE=$SCRIPT_DIRECTORY/.mp4grep_cache"
echo "MP4GREP_MODEL=$SCRIPT_DIRECTORY/model"
echo -n 'PATH=$PATH:'
echo "$SCRIPT_DIRECTORY/bin"
echo ""
echo "Variables exported in $RCFILE"

source $RCFILE

Whilst transcribing a 00:05:34:09 long video, the last transcribed line had a 00:05:25:47 timestamp. In other words, a 5 and a half hour long video had its transcribtion timestamps delayed more and more as the time went on.

Video: 00:00:10:50, Timestamp 00:00:10:35 Video: 00:00:38:03, Timestamp 00:00:37:04 Video: 00:01:39:41, Timestamp 00:01:37:12 Video: 00:03:45:28, Timestamp 00:03:39:50 Video: 00:05:34:09, Timestamp 00:05:25:47

Based on last results: ((5*60+34)60+9)/((560+25)*60+47)=1.02568169028 times increase in timestamp clock (approximately).

I am using WSL Ubuntu 20.04, as well as the "vosk-model-ru-0.22" Russian 1.5Gb Vosk Model.

Adds a yml file to execute an approval workflow whenever a pull request or push is made to the "release" branch. This workflow can be added as a action in the protected branch settings.

I monitored around 13GB RAM needed to use the 1.8GB vosk-model-en-us-0.22 to process a 209MB wav file with roughly 2 hours of audio.

Until I upped the VM's RAM I couldn't get it to run without 'Segmentation fault'-ing. It didn't work well with Hyper-V's dynamic memory either, the process was always killed before memory was ballooned up to what was needed.

With the RAM pre-allocated it didn't have any problem, just took a while with no indication of progress apart from occasional 'will do an exact optimization' messages.

Thanks for the work done to create this tool!

mp4grep does not support arm64 until https://github.com/ocaml-multicore/ocaml-multicore/issues/86 is resolved.

We will take up arm64 after we hit OCaml 5.0 MVP.

HI, I really appreciate your tool. It's such a great solution to make recordings more accessible for further investigations :smiley:

I read that Vosk has also a speaker identification / detection and I'm wondering, if you could add this to mp4grep as well? For myself there are a lot of nice usecases to track / analyse discussions (TV shows, movies, phone recordings, podcasts, web conferences, ...) and that allow great research like NLP or knowledge base and making multimedia content more accessible to users with handicaps. Done with privacy in mind and not contributing to major tech company algorithms.

My understanding so far is, that Vosk needs fingerprinting for different speakers and maybe multiple fingerprints per person. So we will need a way to assign lines within a transcription to fingerprinted speakers and to label this fingerprints with human readable labels. In a second step, there might be a final processing, that assigns this labels to every transcription line. Maybe we need also an extended transcription format like WebVTT to share this assigned lines and timecodes?

Hello ! When I run this program with a mp3 file I get the following error : ws.schild.jave.EncoderException: java.io.IOException: Cannot run program "/var/folders/gj/7wnvhh_92fn2cs9thjy0t0y40000gn/T/jave/ffmpeg-x86_64-3.2.0-osx": error=2, No such file or directory