Google Lens can now answer questions about videos

Google has modernized the purpose of visual research, which has the potential to answer almost realistic questions about the environment. Android and iOS English users with the installed Google Applications can start shooting videos via lenses and ask questions about objects that show interest in videos. 

Lu Wang, director of product management at Lens, said the feature uses a "custom" Gemini model to understand relevant videos and questions. Gemini is a family of Google AI models that power many products in the company's portfolio. "Let's say you want to know more about some interesting fish," Wang said at a press conference. "[Lens] creates an overview that explains why they're swimming in circles and gives you additional resources and useful information."

To access the new Lens video analytics features, you'll need to sign up for the Google Search Labs program, as well as subscribe to the Labs' AI Review and other experimental features. In the Google app, pressing the shutter button on your phone activates video mode on Lens. When you ask a question while recording a video, Lens will directly access answers provided by AI Summary, a feature in Google Search that uses AI to summarize information from across the web. 

Wang said Lens uses AI to determine which frames of a video are most "interesting," salient and, above all, relevant to the question, and then uses that to "ground" the answers from the AI ​​summary. "This is all a result of observing how people are currently trying to use things like Lens," Wang said. "If you reduce barriers, ask these questions, and help people satisfy their curiosity, people will gather naturally."

Google has beaten both companies to the punch, it seems minus the fact that Lens is asynchronous (you can’t chat with it in real time), and assuming that the video feature works as advertised. We weren’t shown a live demo during the press briefing, and Google has a history of overpromising when it comes to its AI’s capabilities. 

In addition to video analytics, Lens can now search images and text at once. English-speaking users, including those not enrolled in Labs, can simply open the Google app, hold down the shutter button to take a photo, and ask a question out loud. Starting today, when Lens on Android or iOS recognizes a product, it’ll display information about it, including the price and deals, brand, reviews, and stock. Product ID works with uploaded or recently taken photos (not videos) and is currently limited to certain countries and certain purchase categories, such as electronics, toys, and cosmetics.