We’ve all pulled up Google Maps Street View to show our friends what our childhood home looked like, or dropped a little person icon on a Parisian street to see if we’ve booked a hotel in a nice neighborhood. Imagine doing that in a more immersive and interactive way. This allows you to actually simulate the street and its surroundings, adjust the weather, and even see what it would look like in a “Day After Tomorrow” scenario.
That’s one of the goals of Google’s latest integration. Starting today, Google DeepMind is connecting Street View to Project Genie, the company’s general-purpose world model that can generate diverse and interactive environments. This new feature was announced during the Google I/O developer conference.
“This is very powerful both for agent (and robotics) use cases and for humans playing together. That’s always been the theme of Genie,” Jack Parker-Holder, a research scientist on DeepMind’s open-endedness team, told TechCrunch.
He gave the example of a new robot being deployed in London that rarely sees the light of day. Parkerholder said Genie can simulate the rare situations where sunlight hits a Victorian house, so the robot can avoid being shocked by the rays when such an event occurs.
“At the same time, some people might say, ‘I’m going to New York, but not at this time of year,'” he continued. “‘It’s going to snow. I’d like to see what the blocks look like in the snow.'”
For two decades, Google has collected Street View data from cars equipped with cameras and individuals wearing “tracker backpacks.” The tech giant has collected more than 280 billion images across 110 countries and seven continents.
“Street View gives us tons of images from all over the world,” says Jack. “You can imagine how powerful it could be to combine rich sources of real-world information and data with the ability to simulate the world.”
Google released its latest world model, Genie 3, for research preview last August, and opened up access to the tool to Google AI Ultra subscribers in the US in January, allowing customers to create interactive game worlds from text prompts and images. The goal is to use Genie for educational experiences, games, and robot training.
The Genie 3 is already powering one of Waymo’s simulators to help train self-driving cars in “very rare events” like tornadoes or accidental encounters with elephants. Adding Street View data to this could help Waymo prepare to launch in more cities around the world.
Waymo has its own simulator, which it expanded to 11 U.S. cities and used to test its AI drivers in several more. The difference with the Genie is that this is all from the car’s perspective, Parkerholder says. In addition to simulating a world anchored in real-world locations, Street View also allows you to shift your perspective to other types of agents, such as humans or robots.
Google is launching Genie Street View for some Ultra users in the US today, and will slowly roll out access at scale. The company says global Ultra users will gain access in the coming weeks.
According to Diego Rivas, product manager at DeepMind, the researchers’ goal is to make this new feature available to as many people as possible. He cautioned that Street View in particular, and Genie in general, are still in the experimental stage and there is room for improvement in terms of accuracy.
In the samples the Google team showed me (including an underwater simulation of the area where I used to live), the results were impressive and recognizable, but still more video game quality than photorealistic. Also, the model is not yet physics aware. That is, cause and effect are not yet understood. For example, in a simulation of a woman running through snow-covered Joshua trees, she ran straight through cacti and bushes.
Compare this to, for example, Google’s image generation tool Nano Banana (which can now generate perfect text in infographics) or its video generation tool Veo. Veo understands that paper boats float on currents, smoke diffuses into the air, and fabric drapes over shapes.
These models do not have physics hardcoded into them. They learn it intuitively over time, through passive observation, just like living things.
“For these types of models, I think they’re probably six to 12 months behind video in terms of accuracy and quality, so I think that can be resolved,” Parkerholder said.
Google Maps director Jonathan Herbert, who joined the Street View team as an intern 12 years ago, said Genie still can’t faithfully recreate streets. He believes the real progress is in the spatial continuity of AI. When you rotate 360 degrees, the AI accurately remembers and simulates the environment behind you. From that point on, the model can build new environments on top of it.
“We’ve been thinking for a long time about how to build the world’s best and richest models based on Street View data,” said Herbert. “Using map data in new ways and for new kinds of AI research has definitely been an idea of ours for quite some time.”
If you buy through links in our articles, we may earn a small commission. This does not affect editorial independence.
