Gemini AI can see and talk to you about what’s on your screen – which could be more helpful than it sounds
- Google Gemini is releasing a feature for Pixel 9 that lets it discuss what’s on a screen in real-time
- Gemini Live can discuss images, files, and YouTube videos
- The feature makes Gemini Live more aware of the context of discussions
Google is giving its Gemini AI assistant a new perspective, your view of what’s on a smartphone screen. Specifically, the Pixel 9 series of devices, where Gemini Live is now able to “Talk Live about this” and have a real-time chat about images, files, and YouTube videos as you see them, as first spotted by 9to5Google. It’s ideal if you’re keen to ask about a cooking tutorial mid-recipe or demand an explanation about a meme you’re not hip enough to understand.
Gemini Live’s role until now has been much like any standard AI voice assistant, powered by the more conversational models in the mold of ChatGPT. But, now, it can peep at specific content on your screen and add that to its discussions. Pixel 9 owners can access the feature by launching the floating Gemini overlay, where a suggestion will appear suggesting the AI “Talk Live about video” on YouTube, “Talk Live about PDF” in Files by Google, and “Talk Live about this” for images on the screen. It gives Gemini context without you having to explain what’s there. That’s a lot faster than having to manually upload an image from your gallery.
Once you activate the feature, Gemini Live opens up with a preview of whatever’s on the screen you might want to discuss. The AI might offer destination ideas based on a YouTube travel video, summarize a contract PDF, or explain the allegory of a piece of Rennaisance art you’re examining on your phone. If you find it all a tad intrusive, you can stop Gemini Live from automatically ogling what you’re looking at. If you don’t have a Pixel 9, don’t worry. Google has said it will release the feature for Samsung Galaxy S24 and S25 smartphones soon, followed by other Android devices.
Gemini alive
The feature fits neatly into Google’s Gemini strategy. In case you haven’t noticed, Google dreams of cementing Gemini’s place at the center of people’s lives, especially on mobile devices, as evidenced by its ongoing integration with Android. And this feature won’t be the end of Gemini Live’s upgrades. Google is gearing up for Project Astra, a still evolving toolkit that should let users share their screen and stream video in real-time while conversing with Gemini Live.
Google is leaning hard into real-time, in-context assistance. Rather than just generating responses based on abstract queries, Gemini Live wants to be part of the moment, reacting to whatever’s on your screen with (hopefully) useful insights. To keep ahead of Apple‘s plans for Apple Intelligence and whatever OpenAI and Microsoft plan, Google wants Gemini to be as omnipresent as possible.
You might also like
Google Gemini is releasing a feature for Pixel 9 that lets it discuss what’s on a screen in real-time Gemini Live can discuss images, files, and YouTube videos The feature makes Gemini Live more aware of the context of discussions Google is giving its Gemini AI assistant a new perspective,…
