https://byby.dev/vector-databases

Also advise you to Google transfer learning and zero shot learning to learn more(sorry for the pun, but I couldn't resist).

I'm sure you'll be able to do that once the audio files are embedded into a vector database.

There are already some audio generating transformers available I think some are open source. You might want to look for that.

You can comment again if you still have doubts.