TL;DR: We provide a novel perspective to tackle the Temporal Repetition Counting problem using a simple yet effective representation for action cycles, reducing the computational complexity from quadratic to linear with SOTA performance.

Abstract

Most existing methods rely on the similarity correlation matrix to characterize the repetitiveness of actions, but their scalability is hindered due to the quadratic computational complexity. In this work, we introduce a novel approach that employs an action query representation to localize class-agnostic repeated action cycles with linear computational complexity. Based on this representation, we develop two key components to tackle the essential challenges of temporal repetition counting. Firstly, to tackle open-set action counting, we define two action classes: "repetitive actions" and "others". Instead of manually defining the repetitive action class, we propose a dynamic action query strategy. Here, each action query directly represents an extracted video feature, allowing the repetitive actions of interest to be dynamically defined based on the video content itself. Secondly, to distinguish these repetitive action queries from others, we propose inter-query contrastive learning. This performs contrastive clustering over the queries, pulling similar action patterns together while pushing apart those related to background or unrelated movements. As a result, queries classified as repetitive actions are considered as repetitive cycles, which are then used for counting. Thanks to the query-based representation and contrastive learning strategy, our method significantly outperforms previous works on accuracy while being more lightweight and time-efficient. On the challenging RepCountA benchmark, we outperform the state-of-the-art method TransRAC by 26.5% in OBO accuracy, with a 22.7% mean error decrease and 94.1% computational burden reduction.

Video

Framework

Results

We visualize the predictions of our approach. Each colored block denotes the estimated action interval. The counting result is shown above the video.

Comparison to SOTA

We compare our method with SOTA method TransRAC. TransRAC represents the results by density map, and the final count value is obtained by summing the values in the density map.

Citation

Template courtesy of Jon Barron.