Embodied Large Language Models
Designed at SensiLab↗By Aileen Ng, Rowan Page & Nina Rajcic
Housed within a reflective, orb-like body, it captures fragments of sound, image, and movement as it is carried through everyday life. From these partial perceptions, it produces short textual reflections using generative artificial intelligence, in the form of a large language model.
These ‘thoughts’ are offered not as answers to prompts or explanations, but as simple musings to whoever is curious enough to wonder what it is ‘experiencing’.
These ‘thoughts’ can be observed in real time, running across a screen embedded within the device as a stream of ‘consciousness’.
Machine Eye does not claim understanding or usefulness. It registers presence, assembles meaning imperfectly, and speaks from within the limits of its own perception.
Metaphor has proven central to technology development and interaction design. Metaphors allow designers to frame interactions and help facilitate user understanding of new technologies. Metaphors are used to link a new, potentially illegible, technology with something more familiar. The history of interface design can be read as a genealogy of such metaphors: the desktop as a suite of office furniture, the web as a library to be browsed, mobile applications represented with skeuomorphic icons, and most recently, AI as a conversational partner.
Unlike metaphors used in early interface design, which refer to externalised cognitive tools or systems, metaphors in LLM interface design draw mainly from human activities and roles; the personal assistant, a collaborator, a romantic partner, a therapist and so on.
LLMs are a general-purpose technology; in principle, their function is constrained according to what can and cannot be carried out in language. This task-agnostic generality opens up opportunities and, at the same time, challenges in designing for LLM embodiment. When engineers and designers embed a language model in a physical form, they typically choose a metaphor from which to design. That is, the prevailing design instinct is to reduce the inherent generality by prescribing a familiar role or function to the object.
Unlike metaphors used in early interface design, which refer to externalised cognitive tools or systems, metaphors in LLM interface design draw mainly from human activities and roles; the personal assistant, a collaborator, a romantic partner, a therapist and so on.
LLMs are a general-purpose technology; in principle, their function is constrained according to what can and cannot be carried out in language. This task-agnostic generality opens up opportunities and, at the same time, challenges in designing for LLM embodiment. When engineers and designers embed a language model in a physical form, they typically choose a metaphor from which to design. That is, the prevailing design instinct is to reduce the inherent generality by prescribing a familiar role or function to the object.
From here, the tension emerges: If LLMs are fundamentally open-ended, why should their embodied instantiations be reduced to narrow, predetermined roles? By reducing the generality of the underlying technology, we are limiting the space of possibilities from which new kinds of functions, roles, and relationships could emerge. Here, our primary approach is to leave the function and role open for the user to interpret.
If LLMs are fundamentally general, why should their physical embodiments be reduced to singular, predetermined functions?
What if the device's function were not fixed in advance but instead emerged relationally?
If LLMs are fundamentally general, why should their physical embodiments be reduced to singular, predetermined functions?
What if the device's function were not fixed in advance but instead emerged relationally?
Select Observations from Machine Eye
Visual Layouts by Sean Do
Read about Machine Eye in Detail
Aileen Ng
Dr. Nina Rajcic
We acknowledge and pay respect to the Traditional Owners and Elders—past, present and emerging—of the lands on which Monash University operates, and where this research was conducted. The Wurundjeri Woi Wurrung and Bunurong peoples of the Kulin Nation. We acknowledge Aboriginal connection to material and creative practice on these lands for more than 60,000 years.
We extend our thanks to Jian Shin See
We thank Jon McCormack
Additionally, we are grateful for all of the people who spent time with Machine Eye; we are grateful for their care and the thoughtful feedback that underpin this research.
This research was supported by the Australian Research Council through Rowan Page’s Discovery Early Career Award (DECRA) Fellowship (DE240100161) and the Monash University Faculty of Information Technology and Faculty of Art, Design & Architecture (MADA).