Diffusion text noise could improve reasoning. A kind of overview of the problem, instead of trying to guess just the next token. If you make an oopsie at the start it can quickly compound later-on with autoregression. Being able to go back and forth must be a huge boost. I could see a model in the future where the question to an answer is put like "therefore answer to [question asked] must be" at the end of the noise to force it to answer. It's also a step into the direction of explainability.
That's a great call out, the benefits of infilling could definitely help with certain output formats. IE put the curly braces at the start and end of the sequence.
couldnt the diffusion pertubations happen on the embedding vector level - as suggested in one of the questions - and a nearest neighbor search be used to predict a vector that resembles an actual token?
Yes, I love this idea. I think someone should try it and see how well it works. We dived a little into the code in our next video as a jumping off point!
Diffusion text noise could improve reasoning. A kind of overview of the problem, instead of trying to guess just the next token. If you make an oopsie at the start it can quickly compound later-on with autoregression. Being able to go back and forth must be a huge boost. I could see a model in the future where the question to an answer is put like "therefore answer to [question asked] must be" at the end of the noise to force it to answer.
It's also a step into the direction of explainability.
Could this potentially improve function calling and adherence to certain output formats, e.g., JSON?
That's a great call out, the benefits of infilling could definitely help with certain output formats. IE put the curly braces at the start and end of the sequence.
Tesla diffusion model taught itself to read street signs.
Oh interesting, do you have a link?
couldnt the diffusion pertubations happen on the embedding vector level - as suggested in one of the questions - and a nearest neighbor search be used to predict a vector that resembles an actual token?
Yes, I love this idea. I think someone should try it and see how well it works. We dived a little into the code in our next video as a jumping off point!