The Past and Present Future

Thin and lightweight tablets naturally afford a variety of laid-back interactions through touch, as well as digital pen input to capture freeform thoughts. But status-quo pen and touch interfaces often force people to reach for distant toolbars at inopportune times and awkward, fixed locations.

But what if tablets could recognize how you actually hold and use the device, and adapt the interface to suit?

We propose sensing techniques for tablet devices that transition on-the-go between mobile and stationary use, via Postural Awareness. Postural Awareness reflects the ability of a tablet to sense how it is being used, held, and oriented in a variety of situations, and then adapt to the particular context of use.

Our resulting Posture-Aware Interface responds to nuances (including those shown in A-G above) such as shifting hand grips, varying screen angles, planting the palm while writing or sketching with a digital pen, and detecting your hand even as it reaches onto the screen – allowing the system to detect left vs. right-handed use, for example, or to know you’re about to touch the screen before your meaty little finger even lands on the glass.

It even makes it super easy to discover how to use our tablet’s Surface Pro Pen simply by laying it down on the glass: the pen is recognized, and simple callouts let you customize the button, eraser-click functions, and ink style to your liking.

To realize these capabilities, Posture-Aware devices combine three sensing modalities chosen specifically for plausible integration with tablets:

1. the raw capacitance image from the touchscreen itself,
2. the tilt and motion of the device, and
3. electric field sensors around the screen bezel for grasp and hand proximity detection.

Our resulting Posture-Aware Interface responds to nuances (including those shown in A-G above) such as shifting hand grips, varying screen angles, planting the palm while writing or sketching with a digital pen, and detecting your hand even as it reaches onto the screen – allowing the system to detect left vs. right-handed use, for example, or to know you’re about to touch the screen before your meaty little finger even lands on the glass.

It even makes it super easy to discover how to use our tablet’s Surface Pro Pen simply by laying it down on the glass: the pen is recognized, and simple callouts let you customize the button, eraser-click functions, and ink style to your liking.

The project then demonstrates how these sensors enable Posture Awareness that adapts interaction and morphs user interface elements to suit fine-grained particulars of the user’s natural arm, hand, and grip placement relative to the device.

Watch Sensing Posture-Aware Pen+Touch Interaction on Tablets video on YouTube

Yang Zhang, Michel Pahud, Christian Holz, Haijun Xia, Gierad Laput, Michael McGuffin, Xiao Tu, Andrew Mittereder, Fei Su, William Buxton, and Ken Hinckley. Sensing Posture-Aware Pen+Touch Interaction on Tablets. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19). ACM, New York, NY, USA, 14 pages. Glasgow, Scotland, UK, May 4-9, 2019. Honorable Mention Award (top 5% of papers).
https://doi.org/10.1145/3290605.3300285. 
[PDF] [30 second preview – MP4] [full video – mp4] [Watch on YouTube]