Language Models Are Few-shot learners
In my talk, I’ll try to familiarize listeners with what a GPT-3 is, what it can and can’t do, and what, as a research community, we can learn from it. I’ll start by analyzing research that laid the foundations for developing GPT-3: generative pre-training and language model scaling laws. Afterward, I’ll spend some time discussing the engineering challenges that we needed to overcome to train a model of this size successfully. I’ll follow by briefly looking into the capabilities of a language model of this size and what memorization and meta-learning skills it has learned during training. The talk will conclude by considering future research directions (RL from human feedback, deployment) and summary of some lessons learned.
Bio
Jarosław (Jerry) Tworek – Jerry Tworek is a research scientist at OpenAI. He specializes in Deep Reinforcement Learning, specifically using curricula, memory and representation learning to solve hard reasoning and control problems. Graduated with MSc in Mathematics from University of Warsaw and spent first five years of his career in the Hedge Fund industry. There, Jerry used optimization theory and advanced techniques of signal extraction from noisy datasets to research and develop quantitative trading strategies in futures markets, which finally led him to study reinforcement learning. Jerry took part in the robotics project „Solving Rubik’s Cube with a Robot Hand” which he later presented at NeurIPS 2019 DeepRL workshop. Currently works on evaluating and training big language models (GPT-3) to solve reasoning and logic problems.