ImageGPT (Generative Pre-training from Pixels)

This video will explore the exciting new 6.8 Billion parameter ImageGPT model! The researchers show that better and larger generative models learn better representations for tasks like ImageNet classification!
Thanks for watching! Please Subscribe!
Paper Links:
ImageGPT (Blog Post): openai.com/blog/image-gpt/
ImageGPT (Paper): cdn.openai.com/papers/Generat...
A Survey of Long-term Context in Transformers: www.pragmatic.ml/a-survey-of-...
Google TPUs: cloud.google.com/tpu/docs/tpus
The Illustrated Transformer: jalammar.github.io/illustrated...
PixelCNN: keras.io/examples/generative/...
PixelCNN (Paper): arxiv.org/pdf/1606.05328.pdf
Contrastive Predictive Coding: arxiv.org/pdf/1905.09272.pdf
Big BiGAN: arxiv.org/pdf/1907.02544.pdf
BERT: arxiv.org/pdf/1810.04805.pdf
Rethinking Pre-training and Self-Training: arxiv.org/pdf/2006.06882.pdf

[@connorshorten6311]

2:18 Auto-Regressive modeling of Pixels
4:18 Denoising Autoencoders: AR and BERT
5:40 GPT Architecture, No CNN Prior!
7:00 6.8 BILLION parameters!! Comparison with SimCLR, CPC, BigBiGAN
8:24 Generative Models and Representation Learning for Vision
10:30 Fine-Tuning with Linear Probes
11:50 Working around Quadratic Complexity of Self-Attention
12:50 Context Reduction
13:52 Results and Ablations
18:50 Promise of Longer Context Transformers and Visual Representation Learning

[@herp_derpingson]

Yannic Kilcher sent me here. Good channel. Subbed!

[@jeremykothe2847]

ditto

[@Schematical]

Awesome stuff. Have to watch it a couple times to wrap my head around it.

[@citiblocsMaster]

That imageGPT result is crazy. It seems that you can replace inductive biases (translation invariance via convolutions) with just more data and compute.

[@jeremykothe2847]

The resolution is so low though - not sure it would scale as well even if memory was available for a larger size.

[@airazure2050]

Awesome content! Thanks!

[@erikacardenas2964]

Awesome video!

[@Ritrixone]

Good job!

[@AbdennacerAyeb]

Great job. We need colab tutorials.

[@geekionizado]

😩 too awesome i can't even process

[@quadhd1121]

Can u use plain English please ,it still sounds complex for bigginners