MIT’s FLEX Platform Uses LLMs to Give Household Robots Common Sense

Telling a robot what to do usually requires tedious, step-by-step instructions because machines lack intuitive human common sense. MIT engineers have built a new platform called FLEX to bridge this gap by letting people command robots using natural conversational language. When a user warns a robot not to spill coffee on a laptop, older systems simply note the laptop and update the physical path.

That literal approach fails when environments change or new hazards appear without explicit user warnings. FLEX solves this problem by adding a generalization step powered by GPT-4o, a large language model trained on massive libraries of internet text and video transcriptions. Because these models have processed countless discussions about spilled liquids ruining electronics, the system connects liquids and devices to deduce a broader rule.

Our read: this turns internet text scraping into an intuitive safety filter for physical hardware. The robot uses this background knowledge to automatically avoid a nearby phone as well, even though the user never mentioned it. The platform operates through a three-step cycle of adding features, generalizing rules, and masking out-of-date objects, taking about five seconds in total.

The team tested the software in virtual scenarios where robotic arms carried items like a coffee cup, a candle, and a bowling ball past clutter.

With the new pipeline, the robots generalized warnings to steer clear of flammable objects and fragile items without requiring users to spell out every single rule.

This approach cuts down the massive amount of math and manual programming consumers would otherwise need to learn.

Yet the system relies entirely on simulated scenarios rather than messy, unpredictable real-world homes.

We still do not know how reliably computer vision systems will track and mask dynamic objects during physical execution.

If these generalization pipelines scale effectively, the difference will determine whether household robots become useful helpers or frustrating gadgets people refuse to use.