A Virtual Assistant built with conversational GenAI for optimized navigation of  retail services.

<p>We built a 3D Virtual Assistant at GatorHacks 2023. Although we didn't end up winning, I still think the project is too cool to not share. So here it is, a detailed post about how we, the amazing Team Swamp Jef@s built a 3D virtual companion of our own in 48 hours.</p>

<div class="work-info">
  <div class="work-info__el"><b>Project</b>
    <p>Hackathon</p>
  </div>
  <div class="work-info__el"><b>My Role</b>
    <p>Front End and a tad bit of the backend</p>
  </div>
  <div class="work-info__el"><b>Timeline</b>
    <p>48 hours! (phew)</p>
  </div>
</div>

<br/>

<h4>Overview</h4>
<h2>Background</h2>

<p style="margin-top: 2rem">
GatorHacks 2023 was sponsored by Verizon, and they presented us with their challenge statement which went something like this:</p>

<br/>

<blockquote>
The AI Day Verizon Hackathon is asking participants to rethink how they can leverage conversational GenAI and similar technologies to provide optimized navigation with tailored products, services, and/or educational experiences.
</blockquote>

<br/>

<p> 
Needless to say, we were up for the challenge. We tried to think of many interesting ways by which we could reduce the number of clicks, the time it would take to peruse an online website and the effort needed to sift through the details.</p>

<br/>

<p>It didn't matter what we thought of. Nothing could be easier than just visiting a store and talking to someone and asking them to find you a product that suits your needs. With this realization, we were set to try and emulate the same experience.</p>

<br/>

<p>We realized talking to a disembodied voice would just be boring and awkward. So we decided to build a 3D Virtual Avatar, something akin to a mascot who would speak to you from a position of credibility. </p>

<br/><br/><br/>

<h2>Design Process</h2>

<p style="margin-top: 2rem;">Say hello to Voxie,  an animated AI character who engages with customers directly.</p>

<br/><br/>

<img src="https://www.datocms-assets.com/92199/1697591568-screenshot-2023-10-15-141108.png" alt="Voxie" />

<br/><br/>

<p>Here's how she works.</p>

<ol>
  <li>A user clicks on the Mic icon at the top of the screen to start talking.</li>
  <li>Voxie listens to the user (and actually sends off the user's prompt to GPT-4.0).</li>
  <li>With a little help of Prompt Engineering, Voxie comes to life and answers the user just like a person would.</li>
  <li>Using a text query, Voxie searches our Elasticsearch database and pulls up relevant products and other recommendations.</li>
  <li>The end result: Voxie answers the query and shows all the relevant products she found from her database.</li>
</ol>

<h4 style="margin-top: 8rem;">The hack begins</h4>
<h2>Development</h2>

<br/>

<p>I was in charge of building the frontend. I knew I wanted to build a 3D Avatar, but I had absolutely no idea how to do it. After a lot of research online, I found <a href="readyplayer.me">readyplayer.me</a>. Ready Player Me specializes in creating customizable avatars and virtual identity solutions for various applications and platforms, including virtual reality (VR), video games, and social media. Their primary product is a platform that allows users to create and customize a 3D avatar that can be used in different virtual environments.
</p>
<br/>
<p>With their amazing character creator, I was able to rather quickly create Voxie and also give her some very brand centric clothing.</p>
<br/>
<p>Since we were building a web app, I needed some way to bring this 3D Avatar into a web environment and also animate it. For rendering the 3D character, I used Three.js. </p>

<br/>

<p>
While I was well aware of Three.js, this was my first time working with it. After many a tutorials and two sleepless nights, I was able to put together an avatar that was interactive and full of life, thanks to the animations on Mixamo.</p>

<p> There were no animations for making her talk though, and I couldn't find any solutions online. So I ended up cycling through a bunch of visemes and setting the morph targets on her head at varying intensity and with varying delays to give the illusion of proper pronunciation and mouth movement. The visemes I used are as follows.</p>

<br/>
<code>
const visemes = [ "viseme_kk", "viseme_kk", "viseme_O", "viseme_kk", "viseme_kk", "viseme_aa"];
</code>
<br/>

<br/><br/>

<img src="https://www.datocms-assets.com/92199/1697595940-new-project.png" alt="Tools" />

<br/><br/>

<p>For STT we used Chrome's WebSpeech API. To make Voxie speak we used the Tiktok lady voice... <i>We had fun with this project</i>.</p>

<h4 style="margin-top: 8rem;">Hardship</h4>
<h2>Challenges</h2>

<p style="margin-top: 2rem;">On the last night of the hackathon, with less than 20 hours until the deadline, we ran into trouble. Our logic for building the blocks that would be used for building the UI elements on the Web App (pictured below) stopped working. Or rather it didn't work well from the beginning and now it really started to show.</p>

<br/><br/>

<img src="https://www.datocms-assets.com/92199/1697596530-ui-element.png" alt="trouble" 
style="
    height: 50rem;
    width: auto;
    display: block;
    margin: auto;
"
/>

<br/><br/>

<p>We scrambled to find a solution. We gave a last-ditch effort and tried our luck with Elasticsearch. We uploaded all the data we compiled by scraping through Verizon's Product pages and tried to use the GPT-4.0 response in a complex query string query on Elasticsearch. And to our suprise, it worked and remarkably well.</p>

<br/><br/>

<img src="https://www.datocms-assets.com/92199/1697597528-screenshot-2023-10-17-225148.png" alt="Elasticsearch" />

<br/><br/>

<h4 style="margin-top: 6rem;">Conclusion</h4>
<h2>Lessons Learned</h2>

<p style="margin-top: 2rem;">
  And here's the live demo of our project:

<br/><br/>

<p><b>HEADPHONES WARNING: You might wanna turn the volume down.</b></p>

<br/><br/>

<iframe  src="https://www.youtube.com/embed/pe1lNC42cy8?si=kiDpxn4sLnlJhsu3" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

<br/><br/>

<b>So why didn't we win?</b>

<br/><br/>

<p>I think our project was great, at least deserving of some recognition. I think where we fell short was in our presentation and getting across what our project was actually about. We worked all the way until the deadline, with just 2 hours left until submission. We couldn't find the time to come up with a good presentation of our project.<p>

<br/>

<p>
But we know better than letting defeat bring us down. Failure isn't a side effect of trying to succeed, it's the price of admission.
</p>

<br/>

<p>Much love and huge shoutout to my fellow teammates: Zirui Cao, Xiaomei Hai, Diego Viteri and Rachana Gugale ❤️.</p>

<style>
  .work-info {
    display: flex;
    flex-direction: row;
    flex-wrap: wrap;
    justify-content: space-between;
    margin: 4rem 0;
  }

  .work-info__el {
    display: flex;
    flex-direction: column;
  }

  .work-info__el>b,
  .work-info__el>p {
    margin-bottom: .5rem;
    font-size: 1.7rem;
  }

  .actions-list {
    margin-top: 3rem !important;
    margin-bottom: 2rem !important;
    list-style-type: square;
  }

  .actions-list>li {
    margin-bottom: 1rem !important;
  }

code {
color: var(--accent-color);
    mix-blend-mode: color-dodge;
    background-color: rgba(128, 128, 128, 0.5);
    padding: 0.5rem 0.75rem;
    border-radius: 4px;
    font-size: 1.6rem;
    line-height: 2.8rem;
}

iframe {
    display: block;
    width: 100%;
    height: 100%;
    aspect-ratio: calc(16/9);
}
</style>

Building a Virtual Voice Retail Assistant