diff --git a/README.md b/README.md index 584f3d0..25e14db 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ Please, cite us. [Bibtex citation](http://www.afnlp.org/conferences/ijcnlp2023/p 1) ``pip install librosa`` -- audio processing library -2) Whisper backend. +2) **Whisper backend**. Two alternative backends are integrated. The most recommended one is [faster-whisper](https://github.com/guillaumekln/faster-whisper) with GPU support. Follow their instructions for NVIDIA libraries -- we succeeded with CUDNN 8.5.0 and CUDA 11.7. Install with `pip install faster-whisper`. @@ -41,9 +41,16 @@ Alternative, less restrictive, but slower backend is [whisper-timestamped](https The backend is loaded only when chosen. The unused one does not have to be installed. +Or: **Seamless Streaming** -- alternative to Whisper-Streaming, wrapped to enable the same operation modes and input/output format as Whisper-Streaming. + +`pip install fairseq2 pydub sentencepiece git+https://github.com/facebookresearch/seamless_communication.git` + +Installation suggested [here](https://github.com/facebookresearch/seamless_communication/blob/main/Seamless_Tutorial.ipynb), for special torch version cases refer to [fairseq2](https://github.com/facebookresearch/fairseq2#variants). + + 3) Optional, not recommended: sentence segmenter (aka sentence tokenizer) -Two buffer trimming options are integrated and evaluated. They have impact on +Two buffer trimming options are integrated and evaluated for Whisper backends. They have impact on the quality and latency. The default "segment" option performs better according to our tests and does not require any sentence segmentation installed. @@ -68,8 +75,9 @@ In case of installation issues of opus-fast-mosestokenizer, especially on Window ### Real-time simulation from audio file ``` -usage: whisper_online.py [-h] [--min-chunk-size MIN_CHUNK_SIZE] [--model {tiny.en,tiny,base.en,base,small.en,small,medium.en,medium,large-v1,large-v2,large-v3,large}] [--model_cache_dir MODEL_CACHE_DIR] [--model_dir MODEL_DIR] [--lan LAN] [--task {transcribe,translate}] - [--backend {faster-whisper,whisper_timestamped}] [--vad] [--buffer_trimming {sentence,segment}] [--buffer_trimming_sec BUFFER_TRIMMING_SEC] [--start_at START_AT] [--offline] [--comp_unaware] +usage: whisper_online.py [-h] [--min-chunk-size MIN_CHUNK_SIZE] [--model {tiny.en,tiny,base.en,base,small.en,small,medium.en,medium,large-v1,large-v2,large-v3,large}] [--model_cache_dir MODEL_CACHE_DIR] + [--model_dir MODEL_DIR] [--lan LAN] [--task {transcribe,translate}] [--backend {faster-whisper,whisper_timestamped,seamless}] [--vad] [--buffer_trimming {sentence,segment}] + [--buffer_trimming_sec BUFFER_TRIMMING_SEC] [--start_at START_AT] [--offline] [--comp_unaware] audio_path positional arguments: @@ -78,24 +86,26 @@ positional arguments: options: -h, --help show this help message and exit --min-chunk-size MIN_CHUNK_SIZE - Minimum audio chunk size in seconds. It waits up to this time to do processing. If the processing takes shorter time, it waits, otherwise it processes the whole segment that was received by this time. + Minimum audio chunk size in seconds. It waits up to this time to do processing. If the processing takes shorter time, it waits, otherwise it processes the whole segment that was received + by this time. Applicable both to Whisper and seamless. --model {tiny.en,tiny,base.en,base,small.en,small,medium.en,medium,large-v1,large-v2,large-v3,large} - Name size of the Whisper model to use (default: large-v2). The model is automatically downloaded from the model hub if not present in model cache dir. + Name size of the Whisper model to use (default: large-v2). The model is automatically downloaded from the model hub if not present in model cache dir. Not applicable to seamless. --model_cache_dir MODEL_CACHE_DIR - Overriding the default model cache dir where models downloaded from the hub are saved + Overriding the default model cache dir where models downloaded from the hub are saved. Not applicable to seamless. --model_dir MODEL_DIR - Dir where Whisper model.bin and other files are saved. This option overrides --model and --model_cache_dir parameter. + Dir where Whisper model.bin and other files are saved. This option overrides --model and --model_cache_dir parameter. Not applicable to seamless. --lan LAN, --language LAN - Language code for transcription, e.g. en,de,cs. + Language code for transcription, e.g. en,de,cs. Seamless backend has its own 3-letter language codes, e.g. eng, deu, ces. --task {transcribe,translate} Transcribe or translate. - --backend {faster-whisper,whisper_timestamped} - Load only this backend for Whisper processing. - --vad Use VAD = voice activity detection, with the default parameters. + --backend {faster-whisper,whisper_timestamped,seamless} + Load only this backend for Whisper processing, or Seamless Streaming. + --vad Use VAD = voice activity detection, with the default parameters. Not applicable to seamless. --buffer_trimming {sentence,segment} - Buffer trimming strategy -- trim completed sentences marked with punctuation mark and detected by sentence segmenter, or the completed segments returned by Whisper. Sentence segmenter must be installed for "sentence" option. + Buffer trimming strategy -- trim completed sentences marked with punctuation mark and detected by sentence segmenter, or the completed segments returned by Whisper. Sentence segmenter + must be installed for "sentence" option. Not applicable to seamless. --buffer_trimming_sec BUFFER_TRIMMING_SEC - Buffer trimming length threshold in seconds. If buffer length is longer, trimming sentence/segment is triggered. + Buffer trimming length threshold in seconds. If buffer length is longer, trimming sentence/segment is triggered. Not applicable to seamless. --start_at START_AT Start processing audio at this time. --offline Offline mode. --comp_unaware Computationally unaware simulation. diff --git a/seamless_integration.py b/seamless_integration.py index f4e1a67..9534fe2 100644 --- a/seamless_integration.py +++ b/seamless_integration.py @@ -1,5 +1,4 @@ #!/usr/bin/env python3 - import sys import numpy as np @@ -9,13 +8,11 @@ from simuleval.data.segments import SpeechSegment, EmptySegment from simuleval.utils.arguments import cli_argument_list from simuleval import options - from typing import Union, List from simuleval.data.segments import Segment, TextSegment from simuleval.agents.pipeline import TreeAgentPipeline from simuleval.agents.states import AgentStates - SAMPLE_RATE = 16000 def reset_states(system, states): @@ -26,7 +23,6 @@ def reset_states(system, states): for state in states_iter: state.reset() - def get_states_root(system, states) -> AgentStates: if isinstance(system, TreeAgentPipeline): # self.states is a dict @@ -35,7 +31,6 @@ def get_states_root(system, states) -> AgentStates: # self.states is a list return system.states[0] - def build_streaming_system(model_configs, agent_class): parser = options.general_parser() parser.add_argument("-f", "--f", help="a dummy argument to fool ipython", default="1") @@ -76,7 +71,7 @@ from seamless_communication.streaming.agents.silero_vad import SileroVADAgent from seamless_communication.streaming.agents.unity_pipeline import UnitYAgentPipeline class FixDetokenizerAgent(DetokenizerAgent): def decode(self, x: str) -> str: - return x.replace(" ", "").replace("\u2581", " ") + return x.replace(" ", "").replace("\u2581", " ") # .strip() is removed class FixSeamlessStreamingS2TVADAgent(UnitYAgentPipeline): pipeline = [ @@ -88,9 +83,28 @@ class FixSeamlessStreamingS2TVADAgent(UnitYAgentPipeline): ] ################################## -#class SeamlessProcessor(OnlineASRProcessorBase): # TODO: there should be a common base class +# the next pieces of are copypasted from the tutorial and put to the corresponding methods + +#class SeamlessProcessor(OnlineASRProcessorBase): # TODO: there should be a common base class. But the code would not be simple anymore. class SeamlessProcessor: - def __init__(self, tgt_lan, logfile=sys.stderr): + ''' + Wrapping SeamlessStreaming for the same operation modes as + Whisper-Streaming's OnlineASRProcessor. + + ''' + def __init__(self, tgt_lan, task, logfile=sys.stderr): + ''' + tgt_lan: must be 3-letter language code that Seamless-Streaming supports for text output mode. + task: see below + logfile + ''' + if task in ("transcribe","asr"): + task_arg = "asr" + elif task in ("translate","s2tt"): + task_arg = "s2tt" + else: + raise ValueError("task argument must be 'transcribe' or 'translate', or 'asr' or 's2tt'") + self.logfile = logfile agent_class = FixSeamlessStreamingS2TVADAgent @@ -105,7 +119,7 @@ class SeamlessProcessor: no_early_stop=True, max_len_a=0, max_len_b=100, - task="s2tt", + task=task_arg, tgt_lang=tgt_lan, block_ngrams=True, detokenize_only=True, @@ -143,11 +157,11 @@ class SeamlessProcessor: return (None, None, "") - def process_iter(self): + def process_iter(self, finished=False): input_segment = SpeechSegment( content=self.audio_buffer, sample_rate=SAMPLE_RATE, - finished=False, + finished=finished, ) self.audio_buffer = np.array([],dtype=np.float32) input_segment.tgt_lang = self.tgt_lan @@ -155,7 +169,4 @@ class SeamlessProcessor: return self.process_segment(input_segment) def finish(self): - segment = EmptySegment( - finished=True, - ) - return self.process_segment(segment) + return self.process_iter(finished=True) diff --git a/whisper_online.py b/whisper_online.py index 5b2e53b..ec27137 100644 --- a/whisper_online.py +++ b/whisper_online.py @@ -209,6 +209,7 @@ class HypothesisBuffer: return self.buffer class OnlineASRProcessorBase: + '''Showing minimum common public interface for various specialized subclasses.''' def init(self): raise NotImplemented() def insert_audio_chunk(self, audio): @@ -458,16 +459,16 @@ def add_shared_args(parser): """shared args for simulation (this entry point) and server parser: argparse.ArgumentParser object """ - parser.add_argument('--min-chunk-size', type=float, default=1.0, help='Minimum audio chunk size in seconds. It waits up to this time to do processing. If the processing takes shorter time, it waits, otherwise it processes the whole segment that was received by this time.') - parser.add_argument('--model', type=str, default='large-v2', choices="tiny.en,tiny,base.en,base,small.en,small,medium.en,medium,large-v1,large-v2,large-v3,large".split(","),help="Name size of the Whisper model to use (default: large-v2). The model is automatically downloaded from the model hub if not present in model cache dir.") - parser.add_argument('--model_cache_dir', type=str, default=None, help="Overriding the default model cache dir where models downloaded from the hub are saved") - parser.add_argument('--model_dir', type=str, default=None, help="Dir where Whisper model.bin and other files are saved. This option overrides --model and --model_cache_dir parameter.") - parser.add_argument('--lan', '--language', type=str, default='en', help="Language code for transcription, e.g. en,de,cs.") + parser.add_argument('--min-chunk-size', type=float, default=1.0, help='Minimum audio chunk size in seconds. It waits up to this time to do processing. If the processing takes shorter time, it waits, otherwise it processes the whole segment that was received by this time. Applicable both to Whisper and seamless.') + parser.add_argument('--model', type=str, default='large-v2', choices="tiny.en,tiny,base.en,base,small.en,small,medium.en,medium,large-v1,large-v2,large-v3,large".split(","),help="Name size of the Whisper model to use (default: large-v2). The model is automatically downloaded from the model hub if not present in model cache dir. Not applicable to seamless.") + parser.add_argument('--model_cache_dir', type=str, default=None, help="Overriding the default model cache dir where models downloaded from the hub are saved. Not applicable to seamless.") + parser.add_argument('--model_dir', type=str, default=None, help="Dir where Whisper model.bin and other files are saved. This option overrides --model and --model_cache_dir parameter. Not applicable to seamless.") + parser.add_argument('--lan', '--language', type=str, default='en', help="Language code for transcription, e.g. en,de,cs. Seamless backend has its own 3-letter language codes, e.g. eng, deu, ces.") parser.add_argument('--task', type=str, default='transcribe', choices=["transcribe","translate"],help="Transcribe or translate.") - parser.add_argument('--backend', type=str, default="faster-whisper", choices=["faster-whisper", "whisper_timestamped", "seamless"],help='Load only this backend for Whisper processing, or SeamlessM4T Streaming backend.') - parser.add_argument('--vad', action="store_true", default=False, help='Use VAD = voice activity detection, with the default parameters.') - parser.add_argument('--buffer_trimming', type=str, default="segment", choices=["sentence", "segment"],help='Buffer trimming strategy -- trim completed sentences marked with punctuation mark and detected by sentence segmenter, or the completed segments returned by Whisper. Sentence segmenter must be installed for "sentence" option.') - parser.add_argument('--buffer_trimming_sec', type=float, default=15, help='Buffer trimming length threshold in seconds. If buffer length is longer, trimming sentence/segment is triggered.') + parser.add_argument('--backend', type=str, default="faster-whisper", choices=["faster-whisper", "whisper_timestamped", "seamless"],help='Load only this backend for Whisper processing, or Seamless Streaming.') + parser.add_argument('--vad', action="store_true", default=False, help='Use VAD = voice activity detection, with the default parameters. Not applicable to seamless.') + parser.add_argument('--buffer_trimming', type=str, default="segment", choices=["sentence", "segment"],help='Buffer trimming strategy -- trim completed sentences marked with punctuation mark and detected by sentence segmenter, or the completed segments returned by Whisper. Sentence segmenter must be installed for "sentence" option. Not applicable to seamless.') + parser.add_argument('--buffer_trimming_sec', type=float, default=15, help='Buffer trimming length threshold in seconds. If buffer length is longer, trimming sentence/segment is triggered. Not applicable to seamless.') ## main: @@ -538,10 +539,10 @@ if __name__ == "__main__": asr.transcribe(a) else: - print(f"Loading SeamlessM4T Streaming backend model",file=logfile,flush=True) + print(f"Loading Seamless Streaming backend model",file=logfile,flush=True) from seamless_integration import SeamlessProcessor - online = SeamlessProcessor(language, logfile=logfile) + online = SeamlessProcessor(language, args.task, logfile=logfile) beg = args.start_at start = time.time()-beg diff --git a/whisper_online_server.py b/whisper_online_server.py index b2f5120..c8cb685 100644 --- a/whisper_online_server.py +++ b/whisper_online_server.py @@ -20,60 +20,60 @@ args = parser.parse_args() SAMPLING_RATE = 16000 -size = args.model language = args.lan - -t = time.time() -print(f"Loading Whisper {size} model for {language}...",file=sys.stderr,end=" ",flush=True) - -if args.backend == "faster-whisper": - from faster_whisper import WhisperModel - asr_cls = FasterWhisperASR -else: - import whisper - import whisper_timestamped -# from whisper_timestamped_model import WhisperTimestampedASR - asr_cls = WhisperTimestampedASR - -asr = asr_cls(modelsize=size, lan=language, cache_dir=args.model_cache_dir, model_dir=args.model_dir) - -if args.task == "translate": - asr.set_translate_task() - tgt_language = "en" -else: - tgt_language = language - -e = time.time() -print(f"done. It took {round(e-t,2)} seconds.",file=sys.stderr) - -if args.vad: - print("setting VAD filter",file=sys.stderr) - asr.use_vad() - - min_chunk = args.min_chunk_size -if args.buffer_trimming == "sentence": - tokenizer = create_tokenizer(tgt_language) -else: - tokenizer = None -online = OnlineASRProcessor(asr,tokenizer,buffer_trimming=(args.buffer_trimming, args.buffer_trimming_sec)) +if args.backend != "seamless": # loading Whisper backend + size = args.model + t = time.time() + print(f"Loading Whisper {size} model for {language}...",file=sys.stderr,end=" ",flush=True) + if args.backend == "faster-whisper": + from faster_whisper import WhisperModel + asr_cls = FasterWhisperASR + else: + import whisper + import whisper_timestamped + # from whisper_timestamped_model import WhisperTimestampedASR + asr_cls = WhisperTimestampedASR -demo_audio_path = "cs-maji-2.16k.wav" -if os.path.exists(demo_audio_path): - # load the audio into the LRU cache before we start the timer - a = load_audio_chunk(demo_audio_path,0,1) + asr = asr_cls(modelsize=size, lan=language, cache_dir=args.model_cache_dir, model_dir=args.model_dir) - # TODO: it should be tested whether it's meaningful - # warm up the ASR, because the very first transcribe takes much more time than the other - asr.transcribe(a) -else: - print("Whisper is not warmed up",file=sys.stderr) + if args.task == "translate": + asr.set_translate_task() + tgt_language = "en" + else: + tgt_language = language + e = time.time() + print(f"done. It took {round(e-t,2)} seconds.",file=sys.stderr) + if args.vad: + print("setting VAD filter",file=sys.stderr) + asr.use_vad() + demo_audio_path = "cs-maji-2.16k.wav" + if os.path.exists(demo_audio_path): + # load the audio into the LRU cache before we start the timer + a = load_audio_chunk(demo_audio_path,0,1) + + # TODO: it should be tested whether it's meaningful + # warm up the ASR, because the very first transcribe takes much more time than the other + asr.transcribe(a) + else: + print("Whisper is not warmed up",file=sys.stderr) + + if args.buffer_trimming == "sentence": + tokenizer = create_tokenizer(tgt_language) + else: + tokenizer = None + online = OnlineASRProcessor(asr,tokenizer,buffer_trimming=(args.buffer_trimming, args.buffer_trimming_sec)) +else: # seamless backend: + print(f"Loading Seamless Streaming backend model",file=sys.stderr,flush=True) + + from seamless_integration import SeamlessProcessor + online = SeamlessProcessor(language, args.task, logfile=sys.stderr) ######### Server objects