At the Vanguard of Interface Design

Implications for visualization from the 2022 UIST symposium

The ways in which we interact with data and visualization may look and feel very different in the years to come. Many of our current interfaces to visual representations of data can be described as WIMP (Windows, Icons, Menus, Pointer) systems, assuming indirect manipulation via a mouse and keyboard. Recent advances in artificial intelligence / machine learning, computer vision, environmental / physiological sensing, and material sciences may very well shape our future interactions with data.

Between October 29th and November 2nd 2022, the 35th annual ACM Symposium on User Interface Software and Technology (UIST) took place in Bend, Oregon. It was the first in-person research conference that I attended in over three years. I was there along with more than 480 attendees from 25 countries, including more than 270 graduate and undergraduate students. UIST is a highly selective conference, soliciting the best new research on interface design: of 372 research paper submissions, 98 (26%) were accepted to be presented at the symposium. UIST is also a great place to experience demonstrations of new interface prototypes, and this year didn’t disappoint with 64 demos. In addition to technical paper presentations and demos, there were two excellent keynote speakers (Ted Chiang and Marissa Mayer) as well as two shorter “vision” presentations by Mind in Motion author Barbara Tversky and Microsoft’s Chief Scientist Jaimie Teevan.

Unlike research conferences that are devoted to data visualization (such as IEEE VIS), UIST is broader in scope. While there was one session devoted specifically to information and visualization interfaces, I want to highlight a few interesting ideas and opportunities across the conference and comment on how they may be applicable to the future of data visualization. There were eighteen paper presentation sessions over the course of three days, each featuring between four and six presentations, with two sessions taking place concurrently. Given this schedule, I was only able to see about half of the presentations (all sessions were recorded and can be viewed on YouTube). Below, I include links to short video previews and open-access copies of the relevant articles where available. 

Given my role as a researcher specializing in data visualization, I decided to frame my UIST highlights as questions for future visualization research, design, and development to consider. 

• Can data visualization shape our thoughts like written language does?
• How might we talk about data in engaging ways?
• How can we make sense of relationships in text and network data?
• How might visualization manifest in generative design workflows?
• How can sonification and physicalization help people who are blind or visually impaired people understand sensor data?
• How might we visualize and interact with data in extended reality?
• Could reprogrammable “mixels” and shape-changing materials be used for dynamic data physicalization?
• How can we ensure our data visualization satisfies the 3 Ps of good graphics? 

---

Can data visualization shape our thoughts like written language does?

The first keynote speaker was award-winning science fiction writer Ted Chiang, whose “Story of Your Life” was the basis of the Academy Award nominated 2016 film Arrival. His keynote address reflected on how modern humans have collectively internalized the invention of the alphabet: once we learn to read and write, this technology shapes how we think. By evolving from an oral culture to a literate culture, we now longer had to memorize knowledge and therefore lost a dependency on rhyme and meter as mnemonics.

Modern technology has allowed us to further externalize our cognition, but we haven’t internalized these inventions in the same way that we have with the written word, leaving us with the feeling that modern technology is dehumanizing: we rely on artifacts and devices, and we feel paralyzed when we are without them. I prefer to see visualization as a humanizing technology, one that makes data more interpretable and accessible. Moreover, given that we are capable of visualizing data using everyday objects and without the aid of computers, have we internalized the process of data visualization in some capacity?

In rare cases, it would appear as though some exceptionally gifted individuals have internalized mathematical notations and the structure of computer code into their thinking, but unlike the alphabet, these technologies are not co-extensive with speech; thus, this internalization is not as broadly applicable to all human activities. These observations left me wondering about the ways by which we might internalize aspects of data visualization into our thinking, and the power of visualization as a communicative language. 

Finally, Chiang remarked that human alphabets are both easy to recognize and easy to write, and he asked whether there are shapes that are easy to recognize but difficult to generate without the aid of a computer? I would argue that many manifestations of data visualization fit this description, and particularly those that employ shape as a visual encoding channel. For instance, vision science research by Liqiang Huang (2020) would suggest that basic geometric shapes that are easy to draw may not be as perceptually separable as shapes that vary according to the dimensions of segmentability, compactness, and spikiness, which include shapes that may be difficult for humans (but easy for computers) to draw.

How might we talk about data in engaging ways?

My own contribution to UIST 2022 was a new approach for presenting data to remote audiences via augmented webcam video (preview), in which interactive and semi-transparent visualization overlays are composited with the speaker’s video. These overlays are also interactive and yet touchless, in that highlighting and selection are made possible with continuous bimanual hand-tracking. Former Tableau Research intern Brian Hall of the University of Michigan presented our paper, which received an honorable mention for best paper. We also invited attendees to try out our prototype at the symposium’s demo exhibit. 

In the same vein, Jian Liao of the University of Calgary presented RealityTalk (preview), in which webcam presentations of multimedia content could be augmented in real time with kinetic typography of keywords, and the invocation of images and media associated with those keywords, placed wherever a hand or tracked object happens to be within the video frame. I was intrigued by the prospect of combining this technique with ours, in which you could invoke dynamic visualization content and value or category annotations by uttering keywords associated with data.

How can we make sense of relationships in text and network data?

One of the highlights of the visualization session was the Scholastic project (preview) presented by Matt Hong of the University of North Carolina. For anyone who has ever undertaken a qualitative thematic analysis of a text corpus (e.g., documents, transcripts), the Scholastic tool visualizes this corpus at multiple reading levels, one that prioritizes human-in-the-loop semi-automated clustering that aims to remain transparent and trustworthy, aspects that are critical to interpretive analysis.

Moving beyond text corpora to large article collections and their related metadata, the FeedLens project (preview) presented by Harman Kaur of the University of Michigan is a new approach to faceted search that recognizes distinct entity types related to a search query. For example, consider the task of ranking cities to move to based on your cuisine preferences in your locale, which involves several entity types: cities, types of cuisine, and local restaurants. Kaur and colleagues implemented FeedLens as an extension to Semantic Scholar (a tool for searching the academic literature), visualizing the rank of related entities (papers, authors, institutions, journals) as small inline charts associated with different queries (or lenses).

Continuing with the theme of connecting entities through data, the next project I highlight has to do with relating disparate data sources across the web, often a tedious manual affair of rectifying differences across APIs and confirming relationships in the data. The Wikxhibit project (preview | wikxhibit.org) presented by Tarfah Alrashed of MIT CSAIL leverages WikiData as a universal join table and offers a low-code approach to building rich applications around the data linkages curated by the WikiData community. This work also suggests the possibility of visualizing these relationships explicitly as part of these applications without having to write too much code.

How might visualization manifest in generative design workflows?

Vivian Liu of Columbia University presented OPAL (preview), a tool that uses large language models and text-to-image prompting to generate illustrations for news articles, encompassing article keywords and tones. While the articles considered were primarily feature and opinion pieces, I was curious about the potential for semi-automated reporting about data: trends, before / after comparisons, and relationships. I recalled Gafni et al’s Make-a-Scene image-prompting tool (2022) that allows for shape input to steer image generation, as well as Coelho and Mueller’s Infomages project (2020), which introduced a technique that finds images within an image collection containing shapes that can be used to emphasize or draw attention to data trends. Combining all three techniques could be an interesting way of generating news illustrations for data stories.

Generative design can be a fruitful and serendipitous approach to design (including creative visualization design). However, it can be difficult for designers to maintain a sense of agency and control over the parameters. Yuki Koyama of the Japan’s National Institute of Advanced Industrial Science and Technology presented an approach that frames design as a Bayesian optimization problem (preview), wherein an intelligent assistant observes an iterative design process and makes peripheral suggestions based on the current state of the design, providing the ability to incrementally “blend” suggestions with the designers’ own creations. 

How can sonification and physicalization help people who are blind or visually impaired understand sensor data?

Interactive systems for monitoring streaming environmental and physiological sensor data are typically not accessible to people who are blind or visually-impaired. To address this gap, the University of Washington’s Venkatesh Potluri presented PSST (preview), a tool for specifying how sensor data could be sonified with continuous changes in pitch and amplitude. Beyond sonification, Potluri and colleagues demonstrated how PSST can physically print sensor data as punch cards for music boxes, thereby combining sonification and physicalization.

How might we visualize and interact with data in extended reality?

Four of the presentation sessions were explicitly devoted to various aspects of extended reality (or XR, encompassing both virtual and augmented reality interfaces). However, XR-related projects also appeared in many of the other sessions. Several projects were affiliated with Meta Reality Labs, who have invested heavily into XR. Given Reality Labs’ platinum-level sponsorship of the conference (disclosure: my employer Tableau was a Bronze-level sponsor of UIST), the recurring XR theme was not unexpected. 

Beginning with visual perception in XR, Zhipeng Li of Tsinghua University showed how color saturation and color value depth cues could help people understand the distance to 3D positions of content in AR applications (preview), complementing other depth cues such as contrast and blur while reserving color hue for encoding other information. Consider how when you look at a distant landscape, points that are farther away seem a little more blue and a little blurrier. One of the examples Li showed was a 3D scatterplot, thereby illustrating the implications for visualization in extended reality.

Moving on from perception to interaction in XR, Hiroki Kaimoto of the University of Tokyo presented a bidirectional interaction between mobile AR and robotic objects (preview), in which small robots obey spatial constraints imposed upon them via a mobile augmented reality application.Conversely, the movement and positions of the small robots can also update graphical elements in the AR display. Seeing this, I could imagine controlling a physicalized unit chart, where each robot represents an entity in the data (as demonstrated by Mathieu Le Goc and colleagues in 2018) which can be repositioned and recategorized manually or through interactive controls in an AR interface.

Being able to interact naturally at fine scales in XR is challenging. Ideally, we should be able to reach out, point, gesture, and make selections with our fingers, without requiring clumsy hand-held paddle devices that are currently used with many XR headsets. Nathan DeVrio and Daehwa Kim (both of Carnegie Mellon University working with Chris Harrison) respectively presented DiscoBand (preview) and EtherPose (preview), wrist-worn devices that track arm, hand, and finger movements. The former uses two arrays of tiny depth cameras, while the latter uses two antennas measuring dielectric loading emanating from the motion of the hand. If we want to make fine selections and perform precise filtering, sorting, and aggregation operations with 3D data visualization, sensing devices such as these prototypes could be very helpful.

As an alternative to hand-held and head-mounted AR displays, acoustic levitation techniques can suspend luminous particle-based holograms in mid-air. However, until recently it was impossible to interact directly with the holograms without distorting or destroying them. Diego Martinez Plasencia of University College London demonstrated TipTrap (preview), an approach that accommodates direct manual selection and deselection of hologram particles. This could be a boon for interacting directly with 3D point cloud visualization, particularly as the 3D resolution of these acoustic levitation displays improves.

Could reprogrammable magnetic “mixels” and heat-triggered shape-changing materials be used for dynamic data physicalization?

The UIST community also offers new ideas that could be applicable to the implementation of dynamic data physicalization. Shape-changing physical displays are often complicated electronic and mechanical devices that are energy-intensive and have limited expressivity. I want to highlight two projects that illustrate promising alternative approaches. 

First, Martin Nisser of MIT CSAIL presented Mixels (preview), short for magnetic pixels. These small cubes have tiles that can selectively and dynamically couple with other tiles, allowing them to assemble into varying 3D configurations when reprogrammed with new magnetic signatures. I could imagine mixels reassembling into physical unit charts, sorting and resorting themselves according to different categorizations of the data. 

Second, Justin Moon of KAIST presented ShrinkCells (preview), an efficient way to change the shape of a 3D-printed object by propagating heat across the individually-printed filaments of the object. Seeing this, I envisioned the implications for continuous or cyclical animated transitions in data physicalization, such as one that represents daily or annual climate patterns. 

How can we ensure our data visualization satisfies the 3 Ps of good graphics?

I’ll conclude my recap with Barbara Tversky’s vision talk to the UIST 2022 attendees. Tversky is a professor emerita of psychology at Stanford and Columbia Universities, and her 2019 book Mind in Motion: How Action Shapes Thought should be required reading for all visualization practitioners. Her vision talk extended what she had written in her book about how we think in terms of spatial and sequential relations, and how visual tools can help or hinder this thinking. She emphasized the 3 Ps of good graphics: Production, Preference, and Performance, which prompted me to reflect on the visualization tools, the charts, and dashboards we make for others to use and read. When we make visualization tools or visualize data for others, we should consider if and how they can use these tools to produce new representations and new knowledge, if they prefer these tools to other approaches (including those delivered in other modalities), and how they use these tools to perform a task or make a decision.