A Sequential Topic Model for Mining Recurrent Activities from Long Term Video Logs

This paper introduces a novel probabilistic activity modeling approach that mines recurrent sequential patterns called motifs from documents given as word \(\times \) time count matrices (e.g., videos). In this model, documents are represented as a mixture of sequential activity patterns (our motifs) where the mixing weights are defined by the motif starting time occurrences. The novelties are multi fold. First, unlike previous approaches where topics modeled only the co-occurrence of words at a given time instant, our motifs model the co-occurrence and temporal order in which the words occur within a temporal window. Second, unlike traditional Dynamic Bayesian networks (DBN), our model accounts for the important case where activities occur concurrently in the video (but not necessarily in synchrony), i.e., the advent of activity motifs can overlap. The learning of the motifs in these difficult situations is made possible thanks to the introduction of latent variables representing the activity starting times, enabling us to implicitly align the occurrences of the same pattern during the joint inference of the motifs and their starting times. As a third novelty, we propose a general method that favors the recovery of sparse distributions, a highly desirable property in many topic model applications, by adding simple regularization constraints on the searched distributions to the data likelihood optimization criteria. We substantiate our claims with experiments on synthetic data to demonstrate the algorithm behavior, and on four video datasets with significant variations in their activity content obtained from static cameras. We observe that using low-level motion features from videos, our algorithm is able to capture sequential patterns that implicitly represent typical trajectories of scene objects.

1. The starting time \({t_s}\) can range over different intervals, depending on hypotheses. In the experiments, we assumed that all words generated by a motif starting at time \({t_s}\) occur within a temporal document; hence \({t_s}\) takes values between 1 and \({T_{ds}}\), where \({T_{ds}}= {T_d}-{T_z}+1\). However, we can also assume that motifs are partially observed (beginning or end are missing). In this case \({t_s}\) ranges between \(2-{T_z}\) and \({T_d}\).

2. Note that \(P({ d}) \propto {N_d}\) and is thus not unknown.

3. The correspondence between the ground truth motifs and the estimated ones is made by optimizing the normalized cross-correlation measure between the learned motifs \(\hat{P}({{t_r}},{ w}|{ z})\) and the true motifs \(P({{t_r}},{ w}|{ z})\).

4. Note that the topic distributions contain more information than the location probability: for each location, we know what types of motion are present as well. This explains the location overlap between several topics, e.g., between those of Fig. 13b, e, which have different dominant motion directions.

5. Note that we did not set any prior on the motif occurrences within the document, i.e., we set \({{\alpha _{z,d}}}=0\).

6. In the the additional material an exhaustive set of results are provided with motifs rendered in animated-GIF.

7. Waiting activities are characterized by the same word(s) repeated over time in the motif. Thus the successive time color-coded images overwrite the previous ones in the collapsed representation as explained in Sect.. 5.3, leaving visible only the last (orange, red) time instant.

8. Note however that longer motifs increase the chance of observing some random co-occurrences, as the amount of overlap with other activities, potentially unexplained by current motifs, increases as well. This is particularly true when the amount of data is not very large like in the Traffic junction case.

9. To perform the temporal association we allowed a constant offset (either \(-\)1, 0 or 1) between the annotated instants of the event in the ground truth and the starting time of a learned motif.

10. Note that rather than simply using \(P({ z})\) as the prior for a topic to start at time \(t\), we could have further exploited the past informations available in the past motif occurrences \(\hat{P}({t_s},{ z}|{ d})\) (e.g., the motif of Fig. 15e is often followed by that of Fig. 15d several seconds later). However, as this is not part of our model, we preferred to go for the simpler case.

11. By global behavior state, we mean a state that control the set of activities that can occur in the whole scene i.e., at a global scene level. For example, in the MIT video, the traffic lights turning red or green determines which activities can occur in the scene as a whole (vehicles will stop in one side of the road, while stopped vehicles start to move on the other side).

12. Also taking into account that probabilities need to be positive. As a technical detail, a minimum value \(\varepsilon \) is required for \(p({t_s}|{ z},{ d})\) to avoid undefined log-likelihoods in the objective function.

\(\mathcal{D }\) :

Dataset, count matrices of the form \({ n}({ w},{{t_a}},{ d})\)

\({ z}\) :

Motif index

\({ w}\) :

Word index (SLA patterns for real data. See Sect. 5.)

\({ d}\) :

Temporal document index

\({{t_a}}\) :

Absolute time index in temporal documents

\({t_s}\) :

Start time of a motif

\({{t_r}}\) :

Relative time from the start of the motif

\(\varTheta \) :

Model parameters {\(P({ z}|{ d}), P({t_s}|{ z},{ d}), P({ w},{{t_r}}|{ z})\)}

\({N_z}\) :

Number of motifs

\({N_w}\) :

Vocabulary size (number of different words)

\({ D}\) :

Number of temporal documents

\({T_z}\) :

Maximum duration of a motif

\({T_d}\) :

Duration of the temporal document

\({T_{ds}}\) :

Number of motif start time indices in a temporal document

\(\lambda _{z,d}\) :

Sparsity constraint weight

\(\lambda _{bic}\) :

Penalty term weight in BIC equation

Authors gratefully acknowledge the support of the Swiss National Science Foundation (Project: FNS-198, HAI) and of the European Union through its 7th framework program (Integrated project VANAHEIM (248907, and Network of Excellence PASCAL2).

Authors and Affiliations

1. Idiap Research Institute, Martigny, Switzerland

   Jagannadan Varadarajan, Rémi Emonet & Jean-Marc Odobez

2. École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland

   Jagannadan Varadarajan, Rémi Emonet & Jean-Marc Odobez

Corresponding author

Correspondence to Jagannadan Varadarajan.

Appendix: Detailed EM Equation Derivation

In this section we detail the derivation of equations involved in PLSM inference. Our data \(\mathcal{D } = {{ n}({ w},{{t_a}},{ d})}\) is a matrix of counts, where each triplet \(({ w},{{t_a}},{ d})\) denotes the number of times a word \({ w}\), appears at time \({{t_a}}\) in document \({ d}\). The probability of observing this data is given by:

Taking the log on both sides we obtain our objective log-likelihood function as

Since \(P({ w},{{t_a}},{ d}) = \sum _{{t_s}=1}^{{T_{ds}}} \sum _{{ z}=1}^{{N_z}} P({ w},{{t_a}},{ d},{ z},{t_s})\), the log-likelihood equation conditioned on the model parameters \(\varTheta \) i.e., the probability distributions \(P({ z}|{ d}), P({t_s}|{ z},{ d})\), and \(P({ w},{{t_r}}|{ z})\) is written as

We want to infer the model parameters with a constraint that the distribution on motif starting times \(P({t_s}|{ z},{ d})\) is sparse and peaky. To achieve this goal as said in Sect. 3.4, we want to maximize \(D_{KL}(U||P({t_s}|{ z},{ d}))\), the KL divergence between Uniform distribution and \(P({t_s}|{ z},{ d})\). So the constrained log-likelihood equation is given by:

By removing the constant factor we obtain Eq. 5 given by

But the above equation cannot be solved directly due to summation terms inside the log making it intractable. Instead, the EM approach works by optimizing the expected log-likelihood of the complete data with respect to the hidden variables keeping the constraint unchanged, which gives

Notice that now, the log operates over the joint probability over all the variables and not just the observed variables. An optimized way to compute this is by using Eq. 1 and indices \({{t_r}}\) and \({t_s}\) instead of \({{t_a}}\). The joint distribution can be split into its constituent distributions using Eq. 2 So, the expected log-likelihood equation is re-written as:

The goal is thus to optimize this expression with constraints so that the distributions sum to one. Therefore such constraints are enforced using lagrangian multipliers. Finally, the constrained objective function that is optimized is given by:

where \(\gamma _{{ z}}, \delta _{{ z},{ d}}\) and \(\tau _{{ d}}\) are the lagrangian multipliers. The EM algorithm works by iterating through the following E-step and M-step:

E-step: In the Expectation step, the posterior distribution of hidden variables \(({ z},{t_s})\) is computed where the parameters come from the previous iteration’s M-step,

M-step: In the Maximization step, maximizing \(\mathcal{H }(\varTheta )\) w.r.t to the parameters which are the probability mass functions results in the following set of equations.

by eliminating the lagrangian multipliers ^{Footnote 12}, we obtain the following expressions that were presented in (Eqs. 8–10)

Reprints and permissions