Comment by larme

5 hours ago

People who are interested in this application should check synplant[0]. It has a ML technology called "Genopatch" which gives you 2 functionality:

1. you can try to describe a sound with some tags and it will try to generate a sound to capture the feeling of these tags

2.you can feed it with a sound sample and it will try to re-synthesize the sound with its synth engine. Though the end result will usually be just a "re-imagined" version of your input sample.

My guess is the underlying model is not a "deep" model. The main benefit is that the end result is not a wave file, but a list of generated parameters that can be synthesized by the synthplant engine. And now it comes the interesting part: you can tweak these parameters to finetune the generated sound. These parameters have actual meanings (FM ratio, reverb etc.)

[0]: https://soniccharge.com/synplant

10 comments

larme

Reply
aw123  4 hours ago

How far are we from getting a general model that can resynthesize any instrumental audio sound without fiddling with any knobs, so that we can recreate instruments we hear from any song? Seems like it should exist by now?

  • larme  3 hours ago

    For me creating the exact sound is not very interesting from sound designing perspective. You can always sample the real instrument.

    Like physical modeling synthesis, the interesting part is to compress the sound to some parameters that you can tweak and generate new sounds

    Another approach is VAE, which also you give your some latent embedding, you can tweak the embedding to generate new sound. However the meaning of this embedding is not explicit.

    • lightedman  3 hours ago

      "You can always sample the real instrument."

      This doesn't really work on instruments like guitars. Open D sounds way different than fretted D on the E string. Timbre changes with position and it's one of the ways I determine where a player's hands are on the neck when I'm trying to play their song.

      4 replies →

  • zhinit  3 hours ago

    SUNO is pretty close. It still has some weird things going on with high frequency artifacts and phase between left and right channels but if you aren't listening on a good system (like a phone) most people probably wont notice.