Fig 1: Click photo to see the video: representing fluid knowledge using a dynamic system.
.
Reasoning is dynamic. Whether escaping a war zone or watching a suspenseful movie, a person’s understanding of the open world changes over time. If machines could adapt their knowledge as we do, perhaps they could detect unexpected causes of disease. Or flag hostile actions online. But they can’t. One problem is, their models of knowledge can’t evolve.
The Virginia Modeling and Simulation Center (VMASC) is building a narrative modeling platform that can do this, in collaboration with Griffith University and the Eastern Virginia Medical School (EVMS). The tool will enablers users to annotate how they comprehend events across contexts, times, scales and perspectives. To capture the needed dynamism and complexity, the tool is presented in a Unity 3D virtual reality environment. That’s me in the video above, making an animated interpretation of the short story Red Riding Hood as a Dictator Would Tell It.
Current formal knowledge models struggle to accommodate adaptive reasoning. Here’s why. Below is an example of how these knowledge structures are usually represented –
.
Fig 2: Ontology for a system that seats airplane passengers.
.
These methods use fixed definitions and type systems to classify inputs. This means they can only identify expected entities. As a result, many essential aspects of the world slip through their net.
We use narrative-based structures to make dynamic models. Here are some in Keynote, the predecessor to the current tool – Red Riding Hood as a Dictator Would Tell It and Game of Thrones.
The most recent visualization of that tool reimagines ontologies as dynamic, immersive structures:
Narrative structure enables inferences from many sources to be manipulated until they suit a new purpose. Rather than being halted by the unexpected, a story is fueled it, because it ensures the reader must keep turning pages to fully understand. Our method draws from an understanding of narrative from cognitive science (David Herman, Peter Brooks, Paul Thagard) to identify the structures that enable a progressing story to unfold. Our new taxonomy allows us to visualize how existing paradigms are disrupted, re-formed and influence each other. The above image is a still frame from a sequence of 55 frames, which change around the user as the structures evolve.
An immediate future goal is to use this format to represent higher-level devices such as novelty, metaphor, influence, tension and information transfer.
Examples will focus on social media interactions, post-traumatic stress disorder and refugee absorption into new communities. The models produced will inform autonomous reasoning systems and allow better communication of how systems interact with each other.
There are several challenges in developing representations of dynamic information structures. The most fundamental is the tension between using established tools to demonstrate new and evolving qualities. Some others are:
- Information architectures are more complex than can be captured by the simple node-link structures pictured in Figure 2. Instead of solely node-link dependencies, we add another tier that shows how channels of information form gateways, hubs and barriers. These need to be intuitively represented.
- Users will model these structures in a 3D, in a Unity-based VR environment. John Shull is currently adapting Unity to do things it wasn’t originally designed for, including manipulating text, zooming through scales and animating structures as they change.
- New physical user interface gestures are being developed by Alex Nielsen to ensure the actions we use to interact with the interface intuitively reinforce the modeler’s intent.
- Contextual influence needs to be visualized. This is tricky because it involves constellations of changing entities. A gallery of experimental visualizations by Niccolo Casas, Alessio Erioli and Ted Goranson can be found here.
This project was seed-funded by the National Academies Keck Futures Initiative from 2016-2018 and is now supported by a collaboration between the Virginia Modeling Analytics and Simulation Center (VMASC) at Old Dominion University, the Institute for Intelligent Integrated Systems (IIIS) at Griffith University, and researchers from the Eastern Virginia Medical School (EVMS).
*Collaborators:
Beth Cardier – Eastern Virginia Medical School
John Shull, Saikou Diallo, Alex C. Nielsen – Old Dominion University
Ted Goranson – Griffith University
Niccolo Casas – RISD Rhode Island School of Design; The Barlett College, University College of London
Larry D. Sanford, Patric Lundberg, Richard Ciavarra – Eastern Virginia Medical School.
Advisor:
Matthew Garcia – Brandeis University
Publications for this project can be found here.