Every year, billions of dollars are spent on professional development and training across various industries.  Training ranges from the development and mastery of hard skills and subject matter expertise to soft skills that support better productivity and create stress-free environments in the workplace. While technical skills across industries are easily assessable, the impact and measurement of soft skills is a relatively hard topic to address. A recent surge in awareness around this area has resulted in research that shows the staggering impacts of soft skills training. Skills such as time-management, problem solving, teamwork, conflict resolution, and communication have a direct impact on a company’s bottom-line by improving teamwork, increasing employee engagement and reducing absenteeism to name a few. More importantly, soft skills are transferable across industries, making the return on investment (ROI) on such training very significant for both the company and the individual learners who invest in it.
 
Let’s begin with a popular example of a soft skill - active listening. Active listening involves paying close attention to a person, people, or event with a view to hearing, interpreting, and understanding the entirety of the message being communicated. Human parity in speech recognition is estimated to be roughly 95% (i.e. a native speakers understand 95 out of every 100 words they hear in that language). I plan to address more about this statistic and it’s impact in relation to Artificial Intelligence and Natural Language Processing  in future blogs. While the human parity number is remarkable, research suggests that we remember only about 25 - 50 percent of what we hear.  What if we were able to increase this percentage ? For a passive task such as watching a movie the additional information gleaned may allow us to better understand the underlying plot or director’s motives. For an active task such as one that involves a discussion among colleagues or teams, we would have more information to process, allowing us craft a better reciprocal response. By increasing the amount of information absorbed and disseminated thoughtfully, we create a feedback cycle that multiplies productivity.

Often, professional development or adult learning around soft skills tends to involve components that are rule-governed. A quick Google search will reveal that behavioral traits such as nodding or shaking your head in agreement, making eye contact, smiling etc. are all recommended to help establish rapport and engage with counterparts during conversation. There is a fundamental problem with this approach. We developed body language and gestures as a means of communication far before the power of speech evolved in us. On one of my visits to meet with Sandy Pentland at MIT (an advisor to us at Mursion), we began discussing how computers through machine learning / deep learning were capable of analyzing human conversation and predicting outcomes. If we think about the purpose of gestures and body language on an evolutionary timeline, we begin to see that they are an attempt by humans at turn-taking. They acknowledge to the other person(s) that the information presented is being absorbed, and specific gestures convey that we are ready to contribute to the conversation. Put in the context of active listening, it follows that these signals are subconscious in nature and are causal i.e. the body language or gestures occur because we are actively listening and not vice versa.
 
Sandy, in his book Honest Signals, presents very powerful scientific and data-driven evidence of how non-verbals can be analyzed to very successfully predict outcomes in human conversation. He goes on to explain how understanding and measuring these “signals” can help create a very successful and productive workforce. Understanding such signals however requires training of the conscious and subconscious mind. And it is only with continuous practice that we begin to build the relevant skills that are necessary for this task. Measuring performance while building soft skills and receiving feedback in the moment are critical aspects that allow positive behavioral change. Both of these are difficult to do in real life, since situations do not lend themselves to scientific measurement and providing constructive/tough feedback in the middle of a conversation (another soft skill !) is not easy to do without practice. It is here that we can harness the power of virtual reality and simulations - the premise of our work at Mursion. If you’re interested in soft skills development, pick up a copy of Honest Signals and come see us at Mursion to see how we create authentic and powerful Virtual Reality simulations for various soft skills and measure human performance to help professionals master their craft.

Within the last decade, we have seen the likes of Oculus and HTC wipe out the existence of head mounted displays (HMDs) such as the VST-2002 (COASTAR) and MREAL (HM-A1) by Canon or the xSightTM HMDs by Sensics. What was meant purely for research at Universities or at large companies with high end budgets has suddenly become a consumer device, with costs dropping from the tens of thousands into the hundreds. When the power and entertainment value of Virtual Reality (VR) technology is coupled with affordability, it seemed only natural to see a widespread adoption of the technology – one of Facebook’s rationale for acquiring Oculus at ~$2B. But 4 years on, despite promises and the belief that “the VR wave is really here”, we are yet to see applications beyond gaming in the consumer world, and a trickle of adoption for training in the business world. While experts and people will view the reasons for this through different lenses (most of which are likely true and hard to argue against), it remains a fact that we are social beings and any technology that targets mass adoption must yield an increased social value while preserving convenience. The radio, the television, the smart phones, the internet and a host of other technologies have consistently proven their value by evolving to increase social connectivity while preserving convenience. And hardware manufacturers of VR technologies are striving to do the same. The difference however lies in the fact that they are dependent on the effectiveness of the content that is best suited to and delivered via VR. There is an added complication - while most other technologies augment our senses, VR attempts to effectively replace some of them. And there-in lies the challenge.
I recall developing applications for VR during my academic days and studying their effectiveness over other mediums. Hardware advancements clearly helped create better experiences. But the term “better” and “effectiveness” only had meaning when the context of the application was taken into account. The effectiveness of a VR experience is quantified by the degree to which each of our senses (sight, sound, smell, touch & taste) are encompassed by the experience. The illusion of reality must be created, preserved, and fostered by the experience. When a VR experience attempts to replace any of our real-world senses, it is critical that this replacement be nearly as good as what we may experience in the real world. A personal favorite that captures this concept is Mel Slater’s work on Situational Plausibility and Place Illusion.
​
When designing a VR experience with an established purpose, ask yourself the following questions:

• Does your experience require at least one of the 5 senses to be fully replaced?
• Which remaining senses have a co-dependence on the replaced sense?
• Does the technology affordance allow you to effectively replace the primary and the co-dependent senses?

To best illustrate the above, I can’t help but refer to one of the earliest experiments in Virtual Reality termed “The Pit”. I have experienced several variations of this, but probably the best rendition I have ever witnessed is at the Stanford Virtual Human Interaction Lab run by friend and mentor, Jeremy Bailenson. The essence of “The Pit” is this – would you be willing to walk off a plank, at an unknown height, into a deep dark pit, knowing fully well that the experience is being created in virtual reality?

What makes this particular rendition so effective is the attention to detail that Jeremy and his colleagues have paid to the replacement of the senses. The technical barriers to achieving this are many including but not limited to sensor tracking rates, sensor spatial accuracy, latency, refresh rates of the rendering device, resolution, field of view and frame rate among others. At VHIL, all of these have not only been overcome, but also augmented by the addition of synchronized haptic floor shakers, auditory cues and stunning visuals using a variety of VR headsets. In essence, when you are experiencing “The Pit” in VR at VHIL, your sight, sound and touch have effectively been replaced. And there is synchrony and co-dependence between them in the context of the experience. You can see the pit, you can hear the floorboards under you creak as you get closer to the edge, and you may even feel the difference in haptics under your feet as you walk along the edge of the pit – all reinforcing the belief that there is a real pit only a few inches away from where you stand. This is what makes the experience seem nerve-wrecking and very real – in other words, situationally plausible while preserving place illusion.
At Mursion, we follow similar design principles when recreating the most challenging situations encountered on the job. Only, Mursion is not an isolated experience like “The Pit”. It is social, allowing people to interact with avatars, just as they would with other people – through spoken dialogue. Artificial Intelligence (AI) in general and Natural Language Processing in specific is simply not nuanced enough to mimic human conversation (I’ll write more about this in future blogs). We overcome this by augmenting AI with human reasoning captured via live human performance. We focus on the effective replacement of sight and sound in addition to adding a social component to create, foster and preserve the intricacy of human interaction. The term effective is associated with technological nuances beyond the scope of this blog. But, the end result is a highly engaging platform that places learners in different situations – unique events specified by location, time, and circumstance. This exposure to various situations (called multi-exemplar training) facilitates the acquisition of soft skills through intentional practice – or experiential learning. And we are beginning to study just how effective these simulations can be at helping enhance interpersonal skills across various industries.
​
Virtual Reality can be very exciting and powerful - provided we take the time to thoroughly understand its purpose, recognize the limitations and respect the design constraints given our natural ways of interaction with the world. When experiencing a situation in virtual reality, your conscious self is completely aware that the experience is not real. But is it strong enough to overcome your sub-conscious instincts, driven in part by the processing of sensory signals by your brain? If you’re a VR enthusiast and are in the Bay Area, do yourself a favor and sign up for a tour at VHIL to find out. And while you’re at it, come visit us at Mursion, so we can show you what we’re up to.