GoArt Metaverse

Challenge: Visitors can imagine how historic landmarks looked thousands of years ago with the assistance of tours and museums. How can augmented reality provide users with an even more immersive experience?

So you want to time travel

Bustling marketplaces, shining armories, a centuries-old castle towering over craggy rocks. I could elaborate on the Byzantine Empire’s vista in in detail, but nothing would quite be like seeing it for yourself. So what if you could?

My team and I aimed to create an augmented reality mobile application in which users can see how locations appeared in the past. Our first landmark was Harput Castle and the settlement within it.

I converted a height-scanned mesh terrain generated by the development team into an editable Unity terrain. The international art team provided further 3D assets. These include the castle walls, pieces of which I asked to have in one file so XZ positions around the existing topology would be preserved. I measured and meticulously fitted these pieces to their respective Y-position ground elevations alongside supporting boulders.

With the castle walls in place, I began adding the settlement buildings according to a scenic reference map of their locations. On-foot GPS tracking scans and drone photography aided in initial building placement and the avoidance of user pathway obstruction. I continued to refine building positions as the project progressed in accordance to further GPS scans and on-site video recordings of walkable user paths.

Asset Preparation

The international art team provided 2D references of the rich and lively environment we aimed to recreate, as well as individual 3D models of objects and buildings.

Beginning with the raw assets, I prepared the assets’ materials, lighting settings, and scaling. All models were then converted to prefabs and organized in Unity folders and in the scene, with consideration of efficient integration for both myself and future designers.

Character Preparation

I also designed the mesh hierarchy for quickly customizing character outfits, inspired by my system for VENVI. Character models went through three passes: material/lighting/scaling preparation in a similar manner to previous assets, animation setup and troubleshooting, and scene placement setup. For the final pass, I placed characters on color cubes corresponding to zones in which they should appear.

The Art of World Building

I iteratively continued refining Harput’s terrain and buildings as on-site team members sent in updated videos of present day pathways. Of particular importance were Byzantine column roads that hosted a variety of shops, and I took careful note to ensure the roads aligned with real-world measurements of where they once stood.

The decision to place objects first was made, as the environment would define how characters interacted with the world around them. I trained two junior developers to help me populate the large sprawl of buildings with lush plants, embellished homes, and eye-catching displays of fruits and merchandise for sale. As we worked, I took thorough notes and wrote detailed instructions and tutorials on level design for the team.

Giving It Character

With the city’s whimsical riches and roads in place, it was time to bring in the populace!

I placed interacting characters as close to each other as possible, playing animations to make sure their meshes did not intersect as characters’ arms and objects they held moved about. Several characters often turned their heads more than one way in the animations, so I made sure to have them speaking in groups so they did not seem to be nodding to thin air.

I had plenty of artistic freedom setting up character interactions. Perhaps users could relate to the scene of students trying to focus on music lessons, with parents chiming in on the performance. One could take a stroll through the marketplace, watching crowds haggle fiercely over the price of an imported artisan carpet or dance to the tune of a festival band. How visitors would smile, seeing that so many experiences weren’t so different from what they know at all!

I also collaborated with the Head of Engineering to create a hierarchy organization that was quick to parse at a glance. Characters and props were placed in groups under the IDs of their nearest buildings, allowing the development team to easily separate and test components as needed.

Adding Finishing Touches

I passed the completed scene with characters to the sound engineer to give it that extra layer of immersion, assisting in areas where sounds overlapped to avoid dissonance. The band’s soft but jovial tune could be heard throughout Harput, setting a cheery ambience as visitors weaved around the chatter of voices.

This was a good opportunity for me to move to painting the terrain. As we iteratively tested the scene, the development team and I quickly found that memory constraints on our mobile application only allowed the use of four textures at a time. Armed with my 2D illustration skills, I worked in passes to lay down the textures and blend them seamlessly within the technological constraints provided.

I worked with the Head of Engineering to apply lighting, baking it into the scene to finish the polish on the world.

The Result

GoArt Metaverse allows users to explore thirty-one sites from the past, visit historical artifacts, and interact with citizens living in the Byzantine Empire. Following is the application for download:

• GoArt: Virtual Guide, Travel, Culture on Android

• GoArt: Virtual Guide, Travel, Culture on iOS