Reinforcement Learning: How the Method Behind Chatbots Is Teaching Drones and Robots

In short: Reinforcement learning (RL) trains a system by trial and reward: try an action, get scored, and adjust so better actions become more likely. The same algorithm, PPO, was used to fine-tune early instruction-following chatbots and is now the standard way to train robot and drone controllers in simulation. Robotics is also borrowing the full chatbot recipe: pretrain a model on examples, then improve it with RL on whether it succeeded. The big difference is cost. A chatbot can practice millions of times for very little money, while a real robot can break things, so most robot RL happens in simulation and then has to cross the "sim-to-real gap." The best-known drone result, a University of Zurich system called Swift, won 15 of 25 races against three human champions after training in simulation.

Key takeaways

• One idea, two worlds: PPO trains chatbot behavior from human preferences and trains drone controllers from flight rewards.
• Robots copy the chatbot recipe: learn from demonstrations first, then fine-tune with RL on task success.
• Simulation makes it practical: thousands of simulated robots in parallel can learn to walk in minutes on one GPU.
• The sim-to-real gap is the hard part: the winning drone system measured what its simulator got wrong and added that error back in.
• Learning plus classical engineering beats either alone: Swift paired a small learned controller with a traditional Kalman filter for state estimation.
• Students can start today with free simulators, Python and a small drone.

What reinforcement learning is

RL is learning by doing, guided by a score. Four terms cover most of it:

• Agent: the thing learning, such as a drone's flight controller or a language model.
• Action: what it does at each step: a motor command for a drone, the next word for a chatbot.
• Reward: a number that says how well it did, such as progress toward the next gate or a human rating of an answer.
• Policy: the agent's strategy, usually a neural network that maps what it observes to what it does.

Training repeats a simple loop: run the policy, collect rewards, and nudge the network so high-reward actions become more likely. Thousands or millions of loops later, the policy is good at the task, as long as the reward actually describes what you wanted.

Here's that loop on a laptop. We trained PPO three times on CartPole, a classic task where the policy balances a pole on a moving cart and scores a point for every step it stays up. Each run started out falling over within a few dozen steps, and each first hit the perfect score of 500 between about 17,000 and 19,000 training steps.

The same algorithm in chatbots and drones

PPO (Proximal Policy Optimization) is the common link. Its key idea is caution: each update is clipped so the policy can't change too much at once and collapse.

For language models, OpenAI's 2022 InstructGPT paper set the pattern. First, fine-tune on human-written example answers. Second, train a reward model from people ranking the model's answers. Third, use PPO to optimize against that reward model. The result: people preferred answers from a 1.3-billion-parameter InstructGPT model over the 175-billion-parameter GPT-3, despite it having about 100 times fewer parameters. RL after pretraining changed behavior more than size did.

In robotics, PPO trains the controllers for walking robots and racing drones, with the reward written as an equation instead of learned from people.

The methods keep moving between the two fields. In 2024, DeepSeek introduced GRPO (Group Relative Policy Optimization) for math reasoning. It drops PPO's separate "critic" network, which is usually as large as the policy itself, and instead scores each attempt against the average of a group of attempts at the same problem. Robotics researchers then applied GRPO to robot models: SimpleVLA-RL (ICLR 2026) fine-tunes robot policies with nothing but a success-or-fail reward, the robot version of checking whether a math answer is correct.

PPO wasn't new in 2023. The simulators were.

PPO dates from 2017, and the ideas behind it go back decades. What changed between 2017 and today is mostly how fast robots can practice in simulation, not the core algorithm.

Year	Milestone	Why it mattered
1989–1992	Q-learning (Watkins) and the REINFORCE policy gradient (Williams)	The two basic ways to learn from reward: value what actions are worth, or adjust the policy directly
2013–2015	DeepMind's DQN learns Atari games from screen pixels	Neural networks plus RL work at scale. "Deep RL" starts here
2015–2016	TRPO; AlphaGo beats a top professional Go player	TRPO kept each policy update a safe size, but it was complicated to implement
2017	PPO (Schulman et al., OpenAI)	The same safe-update idea, much simpler. It became the default
2018–2019	OpenAI uses PPO to train a robot hand entirely in simulation, with heavy randomization; ETH Zurich's ANYmal walking robot moves from sim to real	Sim-to-real for real robots becomes credible
2021	Isaac Gym (NVIDIA) runs physics and learning together on the GPU	2 to 3 orders of magnitude faster than CPU simulation. Robots learn to walk in minutes
2022	InstructGPT uses PPO on language models	The same algorithm crosses into chatbots
2023	Swift races champions; Orbit framework built on Isaac Sim	Drone RL reaches champion level. Orbit is the project that grew into Isaac Lab
2024	GRPO for reasoning models; Orbit becomes Isaac Lab	Methods flow back from language models to robots
2025	Isaac Lab paper; RL fine-tuning of robot foundation models	One open framework for RL, imitation learning, sensors and photorealistic rendering

Put Swift and Isaac Lab side by side and the shift is clear. Swift used a six-year-old algorithm and trained 100 simulated drones in parallel. Isaac Gym and Isaac Lab are built to run thousands of simulated robots at once on a single GPU, which turns a day of practice into minutes. The algorithm barely changed. The amount of practice a policy can get per hour changed enormously, and so did the know-how for writing rewards and crossing the sim-to-real gap.

The chatbot recipe, applied to robots

Robot foundation models now follow the same stages as language models.

Language model stage	Robotics version	Published result
Pretrain, then fine-tune on examples	Vision-language-action (VLA) models trained on images, language and recorded robot demonstrations	The starting point for the RL methods below
RL from feedback after fine-tuning	Physical Intelligence's π*0.6 learns from its own attempts plus human corrections	On some of the hardest tasks, including making espresso and folding laundry, it more than doubled throughput and roughly halved the failure rate
Human feedback in the loop	HIL-SERL (UC Berkeley): a person steps in to correct a real robot while it learns	Near-perfect success on precise manipulation tasks within 1 to 2.5 hours of training, about 2× the success rate of imitation learning
A language model as a helper	Eureka (NVIDIA and partners): an LLM writes and refines the reward function	Beat human-expert rewards on 83% of 29 tasks, with an average normalized improvement of 52%

Why robots are harder than chatbots

Practice is expensive, simulators are imperfect, and good rewards are hard to write.

• Cost of practice. A language model can attempt a problem millions of times on servers. A real robot gets one attempt per reset and can break itself or something nearby.
• Simulation helps a lot. In 2021, ETH Zurich researchers trained a four-legged robot to walk using thousands of simulated robots in parallel on a single GPU: under four minutes for flat ground and twenty minutes for uneven terrain. That work ran on Isaac Gym, the predecessor of today's Isaac Lab.
• But simulators are wrong in small ways. Real motors lag, air pushes back, cameras blur, and batteries sag. A policy that's perfect in simulation can wobble or crash on real hardware. This is the sim-to-real gap.
• Rewards are rarely obvious. "Was that a good grasp?" is harder to score than "Is the code correct?" A badly written reward gets exploited: the policy finds a way to score points without doing the task.

Case study: Swift, the drone that raced world champions

In 2023, a University of Zurich team published Swift in Nature: an autonomous racing drone, trained with RL in simulation, that beat human champions in head-to-head races using only onboard sensors. It's the clearest public example of every idea above working together, so here it is in detail.

The hardware and the two-part design

Swift is an 870-gram quadcopter with a thrust-to-weight ratio of 4.1, an Intel RealSense T265 tracking camera, an inertial measurement unit (IMU), and an NVIDIA Jetson TX2 computer on board. Its software has two parts:

• Perception (mostly classical): a visual-inertial odometry system estimates the drone's motion, and a small neural network detects the racing gates in the camera image. A Kalman filter fuses the two into one estimate of where the drone is.
• Control (learned with RL): a small neural network, two layers of 128 neurons each, takes that estimate and outputs a total thrust and three rotation rates for the motors.

The learned part is surprisingly small. The heavy lifting of knowing where you are is still done by well-understood engineering.

The reward, term by term

The policy was trained with PPO. Its reward at each step has four parts: progress + perception + smoothness − crash. Each one solves a specific problem:

• Progress rewards getting closer to the next gate on every step. A reward only for finishing a lap would be too rare for the policy to learn from at first; a reward on every step gives it a steady signal.
• Perception rewards keeping the camera pointed at the next gate. This is the subtle one. Swift's position estimate depends on seeing gates. A policy that flies fast while looking away would lose track of where it is. So the reward ties control to sensing: fly in a way that keeps your own perception working.
• Smoothness penalizes jerky commands. A simulator tolerates thrust that flips back and forth every few milliseconds; real motors and propellers don't.
• Crash is a penalty for hitting a gate or the ground, which ends the episode.

How the sim-to-real gap was closed

The first policy trained on 100 million simulated steps, run as 100 drones in parallel, in about 50 minutes on a workstation. Then the team flew the real drone and measured how reality differed from the simulator:

• Perception error: from just 50 seconds of real flight, they fitted a Gaussian process, a statistical model that predicts how far off the real position estimate tends to be in each situation.
• Dynamics error: from 800 to 1,000 real data points, they fitted a k-nearest-neighbors model (k = 5) of the forces the simulator missed, such as aerodynamic effects at high speed.

They added both error models to the simulator and fine-tuned the policy for another 20 million steps. The reasoning is worth noticing: instead of trying to build a perfect simulator, they measured what was wrong with theirs and taught the policy to expect it.

The results, and what the numbers do and don't show

Human opponent	Swift wins
Alex Vanover, 2019 Drone Racing League world champion	5 of 9
Thomas Bitmatta, two-time MultiGP international champion	4 of 7
Marvin Schaepper, three-time Swiss national champion	6 of 9
Total	15 of 25 (60%)

Swift also set the fastest race time of the study, about half a second ahead of the best human time.

Read the win count carefully, because 25 races is a small sample. Our own calculation from the published results: a 95% confidence interval for a 15-of-25 win rate runs from about 41% to 77%. And if the matchups were truly even (a 50% chance per race), a result of 15 or more wins would still happen about 21% of the time by luck alone. So the win count by itself doesn't prove Swift is the better racer. The stronger evidence is the lap time. A machine that can set the fastest lap repeatedly has a real speed advantage. Human champions won the other 10 races because a fast policy still has to be consistent, and any error at racing speed ends the race.

The limits the authors list

• Swift wasn't trained to recover after a crash. Human pilots can sometimes keep going.
• Its perception assumes the environment looks like it did during training. Changes in lighting or appearance can make it fail.
• It was trained for a specific track. Humans adapt to a new course far faster.

That's the honest picture of RL in robotics today: superhuman at a well-defined task in a controlled setting, and still brittle outside it.

Where RL meets computer vision

Most vision models are trained differently from Swift's controller: with supervised learning, where every image comes with the right answer. RL is for decisions you can only score after the fact. The line between them is starting to blur in four ways:

• Split by job. Swift's gate detector learned from labeled images; its controller learned from rewards. Most robots still divide perception and control this way.
• The same simulator for both. Isaac Sim, which Isaac Lab runs on, also generates labeled synthetic images with randomized lighting and textures for training detectors. Vision has its own sim-to-real gap, and the fixes look the same.
• RL tuning vision directly. Before PPO reached chatbots, Google's AutoAugment (2018) used it to search for image augmentations. In 2023, Google researchers used RL to tune vision models for detection, segmentation and captioning toward the scores they're judged by. In 2025, Visual-RFT applied GRPO to a vision-language model, rewarding how well its predicted boxes overlapped the true ones.
• One network, end to end. In vision-language-action models, a single network sees the camera image and outputs the action. When it's fine-tuned with RL on task success, as π*0.6 and SimpleVLA-RL are, the vision part learns from the outcome too. In the newest robot models, RL starts to shape what the robot learns to see.

Where it's heading

• Robot foundation models that keep learning. VLAs that improve from their own deployments, as π*0.6 does, instead of being frozen after training.
• Simpler rewards. Success-or-fail rewards, borrowed from math and code training, reduce the need to hand-tune reward equations.
• Better simulators, and models that act as simulators. Learned models of the world that a policy can practice inside.
• Hybrids. Learned policies for the decisions that are hard to write by hand, with classical control and estimation underneath for safety and predictability.

How students can try it

Everything below runs on a regular laptop with free, open-source tools.

1. Start with a classic. Train PPO on the CartPole balancing task with the Gymnasium and Stable-Baselines3 Python libraries. It trains in minutes and shows the whole loop.
2. Break the reward on purpose. Change the reward and watch the policy find loopholes. It's the fastest way to learn why reward design matters.
3. Move to a drone simulator. Open-source quadcopter environments let you train a hover or waypoint task with the same PPO code. Isaac Lab is the professional option, but it needs real hardware: Ubuntu 22.04 or Windows 11 (no macOS), 32 GB of RAM and an NVIDIA GPU with 16 GB of memory, per its installation guide. Start smaller and move up when you need thousands of robots in parallel.
4. Copy Swift's design, not its speed. On a small indoor drone, let a learned policy choose where to go and let a simple classical controller handle the actual flying, with a kill switch ready. Measure how its behavior differs from the simulator, which is the sim-to-real gap in miniature.

These are the skills drone and robotics teams hire for: Python, machine learning, control and testing ideas against real hardware. Fly safely and within the rules. Recreational flying outdoors needs the free TRUST certificate, and anything that isn't purely recreational needs a Part 107 remote pilot certificate.

Where Drone Edge fits

We build drone courses for schools that start with the rules and safety every operator needs and lead into hands-on flying and building. If you teach and want projects like this in your program, see how we work with schools. For the computer vision side of the same problem, see our explainer on how computer vision spots small drones.

Questions answered

Is the RL in chatbots the same as the RL in robots?

The core algorithms are shared. PPO and its relatives train both. What differs is the reward and the cost of practice: chatbots are scored by reward models or answer checkers, robots by equations or task success, and robots need simulation because real-world practice is slow and risky.

Why not train robots directly in the real world?

RL needs a huge number of attempts, and each real attempt takes time, needs a reset, and risks damage. Simulation provides millions of attempts quickly. Methods like HIL-SERL show that real-world RL can work in hours when a human helps, but most training still starts in simulation.

What is the sim-to-real gap?

It's the difference between how a simulator behaves and how the real world behaves: motor delays, aerodynamics, sensor noise. Policies trained only in simulation can fail on real hardware. Teams close the gap by randomizing the simulator, measuring real errors and adding them back in, or fine-tuning on real data.

Did an AI really beat world champion drone pilots?

Yes, in a controlled setting. Swift won 15 of 25 races against three champions and set the fastest race time, on a track it was trained for, under lighting it was trained for. It couldn't recover from crashes, and it wasn't tested on new tracks.

What do I need to start learning RL?

Python, a laptop and free libraries like Gymnasium and Stable-Baselines3. A GPU helps for large simulations but isn't needed for the first projects.

Last updated: October 2026. Results cited are from the papers linked above. The confidence interval and luck estimate for Swift's race record are our own calculations from the published win counts (Wilson 95% interval; one-sided binomial test against a 50% win rate).