AI to Read Your Emotions on TV: The Future of Targeted Advertising?

As businesses experiment with embedding AI everywhere, one unexpected trend is companies turning to AI to help its many newfound bots better understand human emotion.

It’s an area called “emotion AI,” according to PitchBook’s new Enterprise Saas Emerging Tech Research report that predicts this tech is on the rise.

The reasoning goes something like this: If businesses deploy AI assistants to execs and employees, make AI chatbots be front-line salespeople and customer service reps, how can an AI perform well if it doesn’t understand the difference between an angry “What do you mean by that?” and a confused “What do you mean by that?”

Emotion AI claims to be the more sophisticated sibling of sentiment analysis, the pre-AI tech that attempts to distill human emotion from text-based interactions, particularly on social media. Emotion AI is what you might call multimodal, employing sensors for visual, audio, and other inputs combined with machine learning and psychology to attempt to detect human emotion during an interaction.

Major AI cloud providers offer services that give developers access to emotion AI capabilities such as Microsoft Azure cognitive services’ Emotion API or Amazon Web Services’ Rekognition service. (The latter has had its share of controversy over the years.)

While emotion AI, even offered as a cloud service, isn’t new, the sudden rise of bots in the workforce give it more of a future in the business world than it ever had before, according to PitchBook.

“With the proliferation of AI assistants and fully automated human-machine interactions, emotion AI promises to enable more human-like interpretations and responses,” writes PitchBook’s Derek Hernandez, senior analyst, emerging technology in the report.

“Cameras and microphones are integral parts of the hardware side of emotion AI. These can be on a laptop, phone, or individually located in a physical space. Additionally, wearable hardware will likely provide another avenue to employ emotion AI beyond these devices,” Hernandez tells TechCrunch. (So if that customer service chatbot asks for camera access, this may be why.)

To that end, a growing cadre of startups are being launched to make it so. This includes Uniphore (with $610 million total raised, including $400 million in 2022 led by NEA), as well as MorphCast, Voicesense, Superceed, Siena AI, audEERING, and Opsis, each of which also raised modest sums from various VCs, PitchBook estimates.

The Potential of Emotion AI in the Business World

Of course, emotion AI is a very Silicon Valley approach: Use technology to solve a problem caused by using technology with humans.

But even if most AI bots will eventually gain some form of automated empathy, that doesn’t mean this solution will really work.

In fact, the last time emotion AI became of hot interest in Silicon Valley — around the 2019 time frame when much of the AI/ML world was still focused on computer vision rather than on generative language and art — researchers threw a wrench in the idea. That year, a team of researchers published a meta-review of studies and concluded that human emotion cannot actually be determined by facial movements. In other words, this idea that we can teach an AI to detect a human’s feelings by having it mimic how other humans try to do so (reading faces, body language, tone of voice) is somewhat misguided in its assumption.

There’s also the possibility that AI regulation, such as the European Union’s AI Act, which bans computer-vision emotion detection systems for certain uses like education, may nip this idea in the bud. (Some state laws, like Illinois’ BIPA, also prohibit biometric readings from being collected without permission.)

All of which gives a broader glimpse into this AI-everywhere future that Silicon Valley is currently madly building. Either these AI bots are going to attempt emotional understanding in order to do jobs like customer service, sales and HR and all the other tasks humans hope to assign them, or maybe they won’t be very good at any task that really requires that capability. Maybe what we’re looking at is an office life filled with AI bots on the level of Siri circa 2023. Compared with a management-required bot guessing at everyone’s feelings in real time during meetings, who’s to say which is worse?

Emotion AI and the Future of Television

The advent of AI technology promises to accelerate this process. But given the unique nature of TV viewing – large screens, immersive shows – AI could theoretically take things much further. In fact, NBC Universal (NBCU) has begun experimenting with targeting ads within specific shows based on content themes – intense scenes, funny moments, and emotional content – all using AI.

It’s obviously quite early in this journey, and not only does NBCU have to prove that AI is capable of this type of targeting, but this sort of mindset targeting is effective. Still, it’s not hard to imagine the many directions this could go.

For example, it’s possible that mindset-targeting proves more potent than delivering ads based on old fashioned demographics or digital IDs. Maybe TV’s role as a mass marketing branding vehicle changes, or is even eclipsed. Perhaps some forms of programming, such as emotionally charged shows, provide more resonance with certain types of advertising.

At the same time, maybe other types of programming, for example less intense, more passive viewing, puts people in the right frame of mind to take actions or make purchases.

This could have a massive influence on performance advertising and its role in the TV ecosystem. For years, media companies have been on a quest to prove TV’s ability to drive actions, including purchases. That’s because there are massive amounts of ad budgets locked up in performance driven vehicles that TV would love to tap into.

The more that AI can help determine which kind of mindsets and programming lend themselves to driving consumers to respond to ads they encounter, the bigger the TV ad business can become over time.

This is all very theoretical, but if AI-driven, mindset-buying reveals multiple new revenue lanes for TV, it could radically impact programming decisions. TV networks may soon evaluate individual shows from a CTV ROI perspective. What if, say, action adventure shows drive not just high ratings, but more responsive mindsets?

Conversely, could certain lower rated shows become more valuable for specific types of brands?

Challenges and Opportunities of AI-Powered Emotion Targeting

For starters, if AI-driven mindset targeting is to pay dividends, the tools and technology used for such an endeavor must have enough data to learn from. Just how much information can be extracted from TV programming and viewers' responses to it is an open question.

And even if AI is able to train itself to measure mindset at scale, how easy will it be to prove its worth? There is likely a lot of experimenting ahead for networks and purveyors before they have full confidence in these tactics.

Plus, if AI targeting does work, will brands have the systems in place to act on it? For example, if media buyers can’t fully be optimised based on mindset in a timely fashion, its usage may be limited.

Then there’s the question of just where emotional targeting will fit in the TV ad process. Change in the media business is never easy. Witness the past few years, as ad buyers and networks have grappled with a slew of new measurement providers promising to reinvent TV currency. So far, progress has not kept up the enthusiasm fueling this crusade.

More recently, brands and media companies have explored employing a new set of attention metrics as a perhaps better way to gauge the impact of both media environments and individual creatives. This effort is still finding its footing.

So where does mindset targeting fit in? Will marketers view AI targeting as yet another option on the menu? Could it prove so lucrative that it replaces the current suite of options? In this surprising conservative industry, it’s often smart to bet on the former.

This would seem to be the right time for forward-thinking brands to experiment widely. Surely, NBCU won’t be the only player to make an emotion-driven ad push. Mastering this kind of media buying will take a significant commitment from ads agencies, along with a willingness to operate without establishing benchmarks, historical precedents, or fully ironed out processes for the time being. To help, expect a number of startups to emerge as either potential AI intermediaries or complementary vendors. This area is ripe for new contenders to emerge and help lead the industry forward. We might even see an arms race among third-party partners looking to create scalable solutions across pubs. Ideally, if mindset advertising shows promise and results, emotional-based targeting could become another strong option for advertisers looking for new ways to break through the noise and creative influence. Either way, the effectiveness of the tech will largely depend on the ability of the AI to understand which audiences are the most impactable Ultimately, if marketers can combine that information with behavioural and attitudinal data, we may finally be able to deliver relevant advertising to the right consumer at the right time and place.