2020: Human Conversations and Digital Humans ~ Not Just a Pretty Face

From 2020.

My article “Human Conversations and Digital Humans – Not Just a Pretty Face” published on Medium, 5 February 2020.

There is much hype about digital humans. But what are they and what do they do? How are they created?

And why the current prevalence of female personas? This article explores these questions and more through the human need for conversation, a journey I have been on for several decades.

In the creation of avatars, the gaming and movie industry have led animation and CGI with increasing realism and sophistication. But my interest has been in the gritty messy world of service delivery — in government and healthcare — where people often at their most vulnerable, are adversely impacted by systems and turgid bureaucratic language. This service delivery world is nothing like futuristic movies or gaming — and I ask, why not? Surely creativity and innovation are not at odds with questions of purpose, human experience and ethics.

So at this exciting intersection, here is a look back at my journey over 20 years to digital humans and a look forward to the next 20 years.

Digital Humans…Gradually then Suddenly

Having led the co-creation of “Nadia”, the world’s first AI powered digital human for service delivery for people with disability (2016) and the AI powered Digital Human Cardiac Coach (2019, following 14 years of research and lived experience), I know this landscape deeply.

Almost every other week now we read about a new one or see one ‘in the flesh’ on our devices, and with few exceptions universally pretty.

Over the past year, China’s Xinhua state news agency introduced the AI news reader.

Lil Miquela debuted in 2019 as a “19 year-old” virtual digital influencer on Instagram with 1.5 million followers.

At CES 2020, Samsung introduced “Neon”, the human-like life form.

Deepak Chopra has recently immortalised himself as a digital human.

And Microsoft is now experimenting with customised “social bots” as emotional companions — to create 999 virtual women with its AI bot XiaoIce.

But is this all hype about these apparent shiny new things. What do these digital humans actually do?

And what are they…actually?

Digital humans exist to converse with humans. To ask and answer questions, to provide information, even to give comfort. Not in the staccato short format of the current crop of personal assistants but in real conversations, nuanced with context and expression.

What is largely missing from the commentary is the central role of co-design and ethics. Also wanting, is a deep exploration of the previously unimaginable and unreachable human potential and the unique support for human agency that AI powered digital humans enable.

This is moon-shot stuff but are we brave enough?

Over many years, I have written and spoken about how AI powered empathetic digital humans will exponentially change servicing models across sectors. And this is because human-like interfaces bring back the human dimension to servicing, and scales it. In contrast, the automation race removes the human dimension in an efficiency paradigm that assumes the human dimension is unaffordable, and in fact, not needed.

We have not appreciated nor really understood the depth of the impact of the human need for conversations, empathy and trust, in industry and service delivery transformation. There is much talk of “life events”: a bureaucratic and marketing construct attempting to organise people’s lives and activities around service offers. You only have to look at the marketing efforts of banks in recent years: a divorce or death of a partner is just an opportunity to sell more stuff — and at a time when people are most vulnerable.

20 Years Ago…Information Brokers & Conversations

More than 20 years ago, I was undertaking research in my MBA program at the Melbourne Business School, on the coming disruption of financial services driven by the evolution of electronic “information brokers” and what I saw as common patterns of information rich “conversations”. These “conversations” by customers at the service delivery interface, were an expression of their needs and frustrations. The power would eventually move to the customer when they no longer needed to go into and out of websites. Understanding these conversations would change servicing models and people’s behaviour.

Conversations with Business

A number of years later in the early 2000’s, I was leading the Business Entry Point (www.business.gov.au), a UN Award winning initiative of the three levels of government in Australia. The Business Entry Point was established to be an information services platform with the objective of reducing the red tape impact of “information search” on small business.

Sitting with businesses, particularly small local businesses, provided insight to the actual conversations that businesses were wanting to have about their business, government regulation and taxation. These real-world conversations had very familiar common patterns, which did not neatly fit into the structured world of the web, the labyrinth of government websites, turgid bureaucratic language and FAQs. Small business came to rely on intermediaries to navigate and interpret the complexity of information — in a similar way that customers of financial services had come to rely on advisors and brokers.

The web at that stage was designed to connect documents, not empathetic human conversations. The web world of government, health care and financial services was focussed on structures, templates, and “web standards”. A find-it-yourself false efficiency model that deemed these conversations unaffordable. And yet as humans, we have a visceral urge to communicate and it is through conversations that meaning and understanding is conveyed and discerned. Enterprises responded by pushing away conversations in an outsourced model: the call centre industry mushroomed and wait times exploded. Busy tones, IVR, robo-calls, fixed hours and call-back were all about rationing to avoid the real conversation.

How could we bring conversations back?

This is an excerpt of my work in 2001 with the Business Entry Point, which was a break-away from the structured website paradigm, and described conversations and dialogues and how these would change servicing.

This work changed the model of the Business Entry Point to what a notable global commentator called a “reverse portal” model.

Instead of a “government portal”, this model of syndication provided information in context to wherever and whenever a business needed it: that is, wherever businesses were having conversations. This model was shaped by co-design with businesses, in particular small business. One small business operator famously said to me “…don’t expect businesses to go online simply for the pleasure of doing business with government…(you) need to understand the conversations that businesses need and want to have…”

My view has always been, don’t expect a person to come to a government (or any servicing) website and know what to look for, and that was the basis of my MBA research. Information should be available and accessible in the context of the customer, not the context of the organisation, its structures or products.

“Seamless Service” Vision at Microsoft

That required and involved, machine-to-machine communication in a “seamless service” strategy which I co-authored at Microsoft in the US in 2006. In the years since, some have interpreted “machine-to-machine” as “machine experience”, which is really a false equation as experience is unique to the human condition (some might also say other animals, but here I am talking humans).

But still the servicing model and technology capability were far behind the vision of the human experience of natural contextual conversations.

The structured web paradigm pushed chaos and complexity out onto the people, impacting disproportionately people with disability, people who are chronically ill, people who are illiterate, the vulnerable, and people who are economically and socially disadvantaged.

But bringing conversations to the web as text-based interactions would not be enough: chit-chat text based chatbots are inherently inaccessible and inequitable.

We had to push the paradigm further.

The real challenge for society was/is, how could we make the web itself more human? In fact, human-like and naturally conversational.

Indeed, many researchers (Dr Lucien Engelen), designers, scientists (notably Sir Tim Berners-Lee), gaming and sci-fi movie producers worldwide have been exploring this challenge for years and have really pushed at the intersection of science and creativity.

Nadia: World’s 1st Digital Human for Service Delivery

My own work was accelerated not by technology research and development, but by a most significant humanitarian challenge: the human rights of people with disability in accessing services.

Given this complexity and chaos, how could it be that people with disability could exercise choice and control in accessing services, and for their human agency to be supported in the exercise of their decision making.

The rigid world of structured websites, structured forms, structured channels, call centres, defined hours, and complex bureaucratic language are impenetrable barriers for people with disability in accessing information and services; and in particular, for people with intellectual disability and cognitive impairment.

The structured paradigm has been about pushing away conversations, forcing people least able to adapt — to adapt; to fit in with these structures, language, channels and technology.

As a mother with family members with disability, our lived experience is that this is a cruel and unjust paradigm.

I started the work on Nadia, the world’s first AI powered digital human for service delivery co-created by people with disability, as a human rights issue, well before any of the technology was brought together.

The UN Convention on the Rights of Persons with Disabilities is remarkable drafting. It calls out the right of freedom of expression and access to information by accepting and facilitating “augmentative and alternative communication” so that people with disability can “receive and impart information and ideas on an equal basis…”

Clearly we haven’t got this right. In healthcare and government, many of us are under-served and effectively excluded, not because of our own capabilities, but because the industrialised model of automation and the rigid online world doesn’t accommodate the individual and their human rights.

I am grateful to have had the opportunity to spend time with Sir Tim Berners-Lee at W3C in Boston to discuss the concept of Nadia and to understand his work on the human accessible web.

One clear goal in support of the human rights of people with disability: to achieve natural empathetic embodied conversations regardless of ability or language.

Instead of the industrialised online automation driven model where people had to adapt to channels, this was a vision to have the interface, the language, the channel and the technology adapt and respond to the person.

And we achieved this through radically inclusive co-design grounded in human rights and ethics. The emerging science of co-design has proven fundamental to the creation of systems that are like people, and I will talk further about that shortly.

What was started as a human rights challenge in Australia and driven by people with disability, would go on to benefit all people.

Digital Human Cardiac Coach

And one of these areas of global interest is the role of AI powered digital humans in healthcare: cardiac health care is a particular area of personal interest and work. I wrote about “Digital Humans in Healthcare” in a contribution to the book by the phenomenal global health influencer, Dr Lucien Engelen “Augmented Health(care): The End of the Beginning”.

Eighteen million people die each year from cardiovascular disease, classified as a pandemic by the WHO. Complicating this, sixty percent of populations are health illiterate. That means that people are unable to complete healthcare forms; do not understand basic health concepts; and cannot read basic text or instructions. A great majority of the population simply cannot comprehend the written material the health system gives them. Health illiteracy has been identified as one of the most significant factors impacting cardiac health outcomes.

Another way to think of this is that conversations — or the lack of access to conversation on cardiac health — adversely impacts peoples’ health and mortality.

From our 14 years of lived experience in cardiac health, Al Johnson (my husband) and I have created the AI powered digital human cardiac coach as a mechanism to provide access to cardiac health information via natural empathetic conversations.

Al Johnson is a heart patient: 4 heart surgeries, all up, 8 cardiac bypass grafts and 4 stents. But Al is also a former Royal Australian Air Force (RAAF) aeronautical engineering officer, with a research background in expert systems. He is a personal trainer, a Heart Foundation ambassador and volunteer, a cardiac rehab patient representative, a guest university speaker, and contributor to the development of the Australian heart and stroke national action plan.

And some days, lost in his own personal grief.

It is our experience that if we struggled, how could anyone survive physically and mentally? And it occurred to us, that our experience struggling with bureaucratic health information, not understanding it — and with no one to talk to when we were most vulnerable — was in many ways the same isolating traumatising experience suffered by people with disability in accessing services.

Again, one clear goal. What if we could find a way for people to access and understand cardiac health information — and ask questions without fear of judgment or stigma — in simple language, in their own words, at any time.

As with Nadia, this was an endeavour of co-design but we took it further.

We created the Digital Human Body of Knowledge — Health Variant © for the strategy, design, planning and implementation of AI powered digital humans in health care. From the human experience across the patient journey, we have documented common themes that only co-design can reveal, to build a corpus and conversation topics which is 80% common across health domains. And the explanation for this commonality, is that a heart patient is more than a heart patient: they are people seeking information and reassurance, and this is conveyed through repeated reinforcing empathetic conversations whenever the person needs it.

This story was shared at Singularity University Exponential Medicine in San Diego in 2019: the Digital Human Cardiac Coach “Hanna” project led by Chris Hillier at the New Hanover Regional Medical Center (NHRMC) in Wilmington, North Carolina.

The depth of community enthusiasm and need for Hanna was shown at the 2019 Cucalorus Independent Film Festival health futures stream, an intersection that brings together story-telling, exponential technology and heath care innovation. This is a community, led by Chris Hillier, driving innovation based on the need for healthcare conversations and minimising staff burnout.

With Accenture forecasting the global economic opportunity for “virtual nursing assistants” in healthcare to be $20 Billion in annual benefits by 2026, the digital human service delivery industry, sparked essentially by Nadia, has the potential to serve previously unmet need.

Defining Digital Humans through Co-Design

But there are two fundamental question that are asked: what is a digital human and how is it created?

In short, a digital human is composed of three capability domains: an avatar platform; a cognitive / AI platform; and the digital human co-design capability that determines the servicing model, corpus and context. A digital human requires all three to fulfil its purpose and potential.

A complete digital human is more than an avatar, a CGI replica, or a human-realistic face with digital nervous system. And it is more than AI. These are phenomenal technologies with different trajectories and approaches to R&D on emotionally responsive embodied systems. These technologies have existed for some time but separately are not sufficient to constitute a digital human.

The creation of the digital human is dependent on co-design. Many mistake consultation for co-design — it is not!

Co-design is fundamental to the articulation of the purpose and role of the digital human, the consideration of ethics, and development of the corpus and operating model: all dimensions necessary to transform the avatar to life as a digital human. Co-design creates the rich conversations and knowledge that breathe life into the otherwise dormant avatars and AI technologies.

And by this I don’t mean text chat-bot conversations which are inherently inaccessible. Co-design produces natural contextual human-like embodied conversations: the ebb-and-flow of natural conversational interactions that are impacted by illiteracy, disadvantage, and bureaucratic and technical language.

Some commentators have stated that it is a fairly straight-forward process to “point” an avatar at a brochure or FAQ, and that this can be done in a few days.

From my experience, this is not the case, and there is method and effort in how this should be done. A digital human is not a talking brochure … with a pretty face. Co-design is a strategic function that considers the relationship between an organisation’s brand, purpose and risk profile and how they will use the avatar and AI to converse/inform and emote/connect to influence behaviours. These behaviours might be as commercial as selling a product or service, or as altruistic as medication adherence and adopting a healthy lifestyle.

The co-design process also determines the purpose, role, personality and operating model supporting the particular digital human. The operating model is necessary to sustain operations, scale, agility and quality. Digital humans need to be able evolve to meet the societal changes that impact organisational to human conversations.

There is also much commentary regarding autonomously responsive emotional systems. Once again, my perspective is always “for what purpose”. Whilst the scientific and R&D efforts on these emotionally responsive systems is phenomenal, that does not mean that such systems are appropriate in every setting. Take for example in the service delivery setting of health or disability services. When dealing with people who are traumatised, illiterate, disadvantaged or with impaired cognitive function, emotional systems must respond and exhibit behaviours within a range defined by co-design, not left to an algorithm.

Even real humans involved in service delivery do not act “autonomously” as they like. If a customer or patient screams at them, the person delivering that service is trained to remain calm and neutral with managed escalation procedures, not reflect and react with a response that might otherwise be fight or flight. And so it is with AI digital humans in service delivery: they need to be designed and trained for context.

This doesn’t mean that emotional systems operating within co-designed boundaries are not powerful. Quite the opposite. Co-designed emotional systems achieve highly contextual intelligent conversations in specific domains such as cardiac health or disability. A careful distinction needs to be made between roles and relationships during co-design; this is why a psychologist was part of the co-design team for Nadia.

Co-Designing Digital Human Personas

Another contentious question is: why the prevalence of young female digital human personas?

Commentators have quite rightly asked questions about this, suggesting bias and even sexism. There might be some truth to this in some instances.

From my experience, the issues of gender selection, appearance, mannerisms, voice and so on are extremely complex, once again involving issues of ethics and co-design.

Take for example Nadia. As the first AI powered digital human, Nadia was grounded in human rights and inclusion. So, how is the Nadia female persona explained?

Nadia was grounded in human rights and we used that to prise open design processes that for a long time had effectively excluded people with disability. Nadia’s personality, look, mannerisms and role were all determined through co-design, led by people with disability.

A phenomenal entrepreneur, a quadriplegic, running an assistive technology business; a deafblind colleague doing a PhD in haptic communication; a disability advocate running a social media platform; a young man with cerebral palsy an Apple ambassador; people with intellectual disability supported by university psychology faculty. And many hundreds of other people with disability and their families. Nadia looks how they wanted her to look.

Nadia was not a curiosity being paraded by tech companies: Nadia did not exist as such a product. Nadia was first and foremost, the creation and imagination of people with disability. And significantly, the co-design process was not just about the embodiment of Nadia, but the experience that people with disability wanted and needed, with accessibility across all disabilities a key design determinant.

Co-design revealed insights not otherwise discernible. For example, her lip colour supports lip reading by those with hearing impairment. Her short hair reduces distractions that affect some with cognitive disability. Her calm manner and use of only positive expressions — smiles, head gestures etc — are designed to not startle. Even her voice was carefully designed. Her words had to be clearly enunciated and pace not too fast. A job for an actress and one that Cate Blanchett, who volunteered her time, fitted to perfection.

We explored and were inspired by what was happening in the gaming world and virtual worlds such as Second Life where virtual beings and avatars engaged in conversations. Many people with disability are avid gamers: they say they can be whoever they want to be and are not judged. From this insight, Nadia was to have a presence in Second Life to be where the people are. In addition to Nadia, it was also envisaged that in this Second Life environment, people would have their own avatars taking on roles such as mentor and connector. The power of these conversations and peer guidance was seen as an important and new way to build capacity.

But we all knew that as a young fair skinned female persona, Nadia would not reflect all the communities served. There were plans for a male and female Indigenous persona, co-design to be led by the Indigenous community as an authentic natural interface. Communicating men’s business and women’s business, passing knowledge through conversations and yarns.

Co-design and ethics go hand-in-hand. The principle “nothing about me, without me”, encapsulates the UN Convention on the Rights of Persons with Disabilities. In practice, this principle is universal: it’s fundamental to co-design and is a core principle underpinning the ethics of the design and operation of digital humans as emotionally responsive systems.

I am of the view that in the field of AI, much commentary about ethics is paternalistic and mostly by vested interests. Co-designed AI powered digital human conversations break through barriers of illiteracy and disadvantage unlocking immense human potential. Far more inclusive commentary and research is needed that encompasses AI and the ethics of opportunity enabling human agency in new ways.

So Nadia was a phenomenal example of co-design. Nadia, the female persona, was to be just the starting point.

Similarly, the work on Hanna, the Digital Human Cardiac Coach involves co-design and localisation reflecting the expressions and customs of the community of North Carolina served by NHRMC. At the Cucalorus Film Festival, we wondered if Michael Jordan would step up to be the persona or digital human cardiac coach ambassador for North Carolina to raise awareness through digital human conversations of the humanitarian cause of cardiac health amongst disadvantaged communities.

Clearly, sports brands and cosmetic brands for example, might want a digital human persona and personality to reflect their brand and values. Perhaps a new face not seen elsewhere: an “energetic” digital human influencer perhaps more fully utilising autonomous emotionally responsive systems as an engagement mechanism for their followers.

This goes to the purpose and role of the particular digital human, and the strategic importance of co-design.

And What of the Next Twenty Years?

The gaming and virtual world sectors will be major drivers of innovation in the design of services over the coming years. Digital humans (SimCoach) and virtual worlds (Mayo Clinic, US VA) are not curiosities and not creepy: it’s already happening and the applications are far-reaching into every domain.

With use cases ranging from health care, education, the preservation of icons, space travel and through to the individual consumer.

Nadia, the world’s first AI powered digital human for service delivery, co-created by people with disability, was a breakthrough. It was an imagination breakthrough.

The digital human cardiac coach, focussed on health illiteracy, establishes a new model for access to cardiac health information and services, and in fact for all health domains.

Digital humans such as SimCoach (referred to as a virtual human) have been used for a number of years in mental health settings, monitoring for signs of PTSD in veterans and service personnel.

Avatars operating within a virtual world such as Second Life, are being used in the rehabilitation of service veteran amputees.

Digital humans as companions for the elderly and supporting people with dementia.

In education, a digital human reading coach could support young learners to read in ways not otherwise possible: I spoke about this at the 2018 O’Reilly AI Conference in London.

In 2018, NASA introduced CIMON, its AI virtual assistant on the International Space Station. Currently with a cartoon face, imagine a future embodiment of CIMON as a trusted persona (Neil Armstrong perhaps) for the long trip to Mars.

And in the near future, we can expect AI powered digital humans will upend and augment the call centre industry and the online forms industry. Imagine, never having to physically fill in a form again, especially for people who are illiterate, or have functional or cognitive impairment.

At the individual level, we see the consumerisation of digital humans has started with the preservation of icons: the wisdom of Deepak Chopra will be conveyed in conversations by his digital human clone, long after he has passed. And over the coming years as the industry evolves, people more generally will be able to have a digital human clone of themselves to hold memories and conversations about their life experiences.

And while some commentators such as Fast Company are intrigued by the emergence of digital humans, the reality is that the journey to this point has taken decades. In fact, two of the digital human (avatar) companies referenced in the Fast Company article originated from my Nadia project in 2016.

But this has been far more than a journey of science and technology advancement and the world of start-ups. It has been the human experience, suffering and disadvantage with the yearning visceral human need for conversations, that helped forge this from imagination to reality.

The conversation services economy has started, shaped by the emerging field of co-design. The exponential factor here is that there is no “new” technology needed, no “new” discovery we are waiting on.

The vision for the next 20 years is the managed convergence of digital humans, virtual worlds, augmented reality and their ‘digital humanitarian kin’ and to dismantle the ‘industrialised legacy web’.