Great video once again. May I check that: 1. For the final simulation, the shape of the posterior distribution is arbitrary, though in this case, it is only designed to be right-skewed and close to zero, so that the biased-ness of the rejection sampling becomes apparent. 2. I understand that the vanilla metropolis is unbiased, whereas the rejection sampling variant is biased. Though computationally speaking, I have trouble seeing how they are different, since in both cases you reject negative samples of sigma and repeat the sampling process until you get a positive one. In short, following @4.22, I would have coded both approaches the same way. 3. Practically speaking, is it safe to neglect concerns of biasedness of the rejection sampling variant if the distribution is say far from zero and has large positive values? Thank you for the work you're doing. I am a PhD student in statistics and machine learning who did his undergrad in Engineering, hence I am trying to pick up on graduate level statistics to bridge the gap. I am learning a lot (and quickly) from your resource!
2nd point is exactly my thoughts! Because in vanilla metropolis, while you do propose a negative value (illegal) of sigma, you can never accept it since the posterior probability of an illegal value is 0. So you anyway propose a new value, until that value is not negative (illegal). While in the rejection sampling case, if your proposal distribution proposes a negative (illegal) value for sigma, you reject it right away. So, again you propose a new value until it's not negative (illegal). I honestly don't see a difference! Did you find an answer to this? Thanks for asking this. EDIT: I think I get it. In vanilla metropolis, when you propose a negative (illegal) value of sigma, you then find the posterior probability ratio (which turns out to be 0), so you don't jump to the point and sample the current value. This can lead to a bit of oversampling of values very close to 0 (near illegal regions) While in rejection sampling, since you immediately reject the negative (illegal) values without calculating the ratio, you don't sample the current value when a negative value of sigma is proposed.
@@abhishekbhatia6092 the difference is that in the 1st case (vanilla) once you reject, you add the last observation again. While in the 2nd case (rejection) you always wait until a valid observation is reached. So if you run this for a 1000 iterations, in the 1st case you will have a lot of repeated values, while in 2nd you won't. But as mentioned, the 2nd will be biased. And in any case both will be less efficient than Hastings.
Hi, Thanks for these masterpiece videos. Please in what particular order can one watch your videos? I am very new to Bayesian and it hasn’t been easy for me
Hello Ben awesome video. I wonder if you might be willing to share some of your mathematica notebooks? I am starting to learn mathematica and it would be helpful.
Hi David, thanks for your comment. Yes, that’s what I was thinking of doing. Over the next week or so I’ll try to put them up; either in the caption or on my website. Will let you know when I have done so! Best, Ben
Hey Ben, big fan of your book and both econometrics and bayesian videos. I am also getting started with mathematica and would love if you could share some materials as well! Thanks for putting these out here to help truncate our learning.
This is a fantastic video. I have just one question: how do I obtain the likelihood and prior? I am guessing for likelihood you can calculate the product of the probability densities of each data point from N(mean, b_t). What about the prior?
Why does multiplying by jumping distribution fix the issue ?
Great video once again. May I check that:
1. For the final simulation, the shape of the posterior distribution is arbitrary, though in this case, it is only designed to be right-skewed and close to zero, so that the biased-ness of the rejection sampling becomes apparent.
2. I understand that the vanilla metropolis is unbiased, whereas the rejection sampling variant is biased. Though computationally speaking, I have trouble seeing how they are different, since in both cases you reject negative samples of sigma and repeat the sampling process until you get a positive one. In short, following @4.22, I would have coded both approaches the same way.
3. Practically speaking, is it safe to neglect concerns of biasedness of the rejection sampling variant if the distribution is say far from zero and has large positive values?
Thank you for the work you're doing. I am a PhD student in statistics and machine learning who did his undergrad in Engineering, hence I am trying to pick up on graduate level statistics to bridge the gap. I am learning a lot (and quickly) from your resource!
2nd point is exactly my thoughts!
Because in vanilla metropolis, while you do propose a negative value (illegal) of sigma, you can never accept it since the posterior probability of an illegal value is 0. So you anyway propose a new value, until that value is not negative (illegal).
While in the rejection sampling case, if your proposal distribution proposes a negative (illegal) value for sigma, you reject it right away. So, again you propose a new value until it's not negative (illegal).
I honestly don't see a difference!
Did you find an answer to this? Thanks for asking this.
EDIT: I think I get it.
In vanilla metropolis, when you propose a negative (illegal) value of sigma, you then find the posterior probability ratio (which turns out to be 0), so you don't jump to the point and sample the current value. This can lead to a bit of oversampling of values very close to 0 (near illegal regions)
While in rejection sampling, since you immediately reject the negative (illegal) values without calculating the ratio, you don't sample the current value when a negative value of sigma is proposed.
@@abhishekbhatia6092 the difference is that in the 1st case (vanilla) once you reject, you add the last observation again. While in the 2nd case (rejection) you always wait until a valid observation is reached. So if you run this for a 1000 iterations, in the 1st case you will have a lot of repeated values, while in 2nd you won't. But as mentioned, the 2nd will be biased. And in any case both will be less efficient than Hastings.
@yuan hu I agree. Don't see any difference between case1 and case2.
@yuan hu No. In the 1st case you reject the proposal, i.e. you keep the last point, x_(t+1) = x_(t).
Why does metroppolis also not have the same jumping distribution problem?
Would you mind sharing application in Mathematica? And even if R codes is avaliable?
Hi,
Thanks for these masterpiece videos. Please in what particular order can one watch your videos? I am very new to Bayesian and it hasn’t been easy for me
Hello Ben awesome video. I wonder if you might be willing to share some of your mathematica notebooks? I am starting to learn mathematica and it would be helpful.
Hi David, thanks for your comment. Yes, that’s what I was thinking of doing. Over the next week or so I’ll try to put them up; either in the caption or on my website. Will let you know when I have done so! Best, Ben
Hey Ben, big fan of your book and both econometrics and bayesian videos. I am also getting started with mathematica and would love if you could share some materials as well! Thanks for putting these out here to help truncate our learning.
What the hell is the difference between the first and second proposal for sigma-t-prime in the Metropolis algorithm? Both looked exactly the same.
They are the same. But, in the second one, the proposed value is sampled until it is greater than zero. Then, the sample is rejected/accepted.
This is a fantastic video.
I have just one question: how do I obtain the likelihood and prior?
I am guessing for likelihood you can calculate the product of the probability densities of each data point from N(mean, b_t).
What about the prior?
Worse: What if the data shows a difficult to model distribution?
Hi... what do you mean when you say "kernel"?
more example Ben10