The visual understanding of actions, activities and higher-level behaviour such as daily routines is one of the most challenges problems in computer vision. This is because we, humans, perform tasks in very different ways depending on how busy or focused we are.

Consider, for example, the many ways that coffee can be prepared in the morning, the sorts of interruptions that might occur during the process and how this process might differ from one person to another. BVI researchers led by Dima Damen in the Visual Information Laboratory have created an analytical framework that can perform this type of analysis in industrial settings and in smart homes of the future from both static and wearable cameras.

One particular aspect of the work is focused on the visual analysis of routine - defined as the frequent and regular activity patterns over a specified timescale. Long-term unscripted routine patterns were captured using silhouette and depth imagery and characterised using a Dynamic Bayesian Network processing spatial location and pose alongside time envelops to encode durations where activities are common. Unlike traditional supervised models, the work automatically selects the number of hidden states for fully unsupervised discovery of a single person’s routine. This work has many applications including more automated monitoring and assistance of our aging population in smart home environments.