[Question] Faster-Whisper Home Assistant GPU or Tensor (Coral) Suport?

[werddomain]

I host the Faster-Whisper Docker container on my server. I did not see if it can use other hardware to speed things up?

Thanks!

[Shulyaka]

Faster-whisper supports GPU execution:
https://github.com/guillaumekln/faster-whisper/#gpu-support

Coral TPU is not supported since it does not work well the recurring neural networks: SYSTRAN/faster-whisper#203

The last link however suggests TFLite version of https://github.com/usefulsensors/openai-whisper which could work, but no promises.

[Shulyaka]

Update: I doubt that a model of this size would fit into the edge TPU anyway. Your best bet is GPU. Probably even NVIDIA Jetson, but those are very expensive.

[werddomain]

Faster-whisper supports GPU execution: https://github.com/guillaumekln/faster-whisper/#gpu-support

Coral TPU is not supported since it does not work well the recurring neural networks: guillaumekln/faster-whisper#203

The last link however suggests TFLite version of https://github.com/usefulsensors/openai-whisper which could work, but no promises.

thanks for the fast response. I understand i need to create a docker image to support gpu?

[Shulyaka]

Don't know about docker. This line just suggests to set the device variable to cuda:

rhasspy3/rhasspy3/configuration.yaml

Line 492 in 11e8d30

device: "cpu" # or cuda

[werddomain]

do you know where i can find the makefile or the build file for the docker image rhasspy/wyoming-whisper ?

[Shulyaka]

Nope, sorry

[edurenye]

@Shulyaka you can find it here: https://github.com/rhasspy/wyoming-addons

And I added a PR to support GPU, and for the moment works for whisper but not for piper:
rhasspy/wyoming-addons#4

[jmorille]

The ticket SYSTRAN/faster-whisper#203 say that not be possible ti run on TPU

But the project whisper-jax has done a migration with TPU Coral

• https://github.com/sanchit-gandhi/whisper-jax

If the huggingface test is correct

• https://huggingface.co/spaces/sanchit-gandhi/whisper-jax
• The performance was pretty amazing on running on a TPU

A blog article made some benchmark also:

• https://nikolas.blog/making-openai-whisper-faster/
  

Any implementation on TPU Coral could be an great specially for cheat hardware ?

[krzakx]

I do not thing Iy would be possible to run Whisper-JAX on Google Coral TPU (consumer) since model weight to be loaded is to heavy at least this is what I had read on some other discussions