AU2005252714B2 - Effective audio segmentation and classification - Google Patents

Description

WO 2005/122141 PCT/AU2005/000808 EFFECTIVE AUDIO SEGMENTATION AND CLASSIFICATION Field of the Invention The present invention relates generally to audio signal processing add, in particular, to efficient methods of segmenting and classifying audio streams. Background 5 The ability to subdivide an audio stream into segments containing samples from a source having constant acoustic characteristic, such as from a particular human speaker, a type of background noise, or a type of music, and then to classify each homogeneous segment into one of a number of categories lends itself to many applications. Such applications include listing and indexing of audio libraries in order to assist in effective 10 searching and retrieval, speech and silence detection in telephony and other modes of audio transmission, and automatic processing of video in which some level of understanding of the content of the video is aided by identification of the audio content contained in the video. Past work in this area has focused on indexing audio databases, where 15 performance and memory constraints are relaxed. Real-time methods are most commonly specific to speech detection and speech recognition, and are not designed to work with arbitrary audio models. Model-based segmentation methods, such as those using Hidden Markov Models (HMMs), efficiently segment and classify audio, but have difficulties dealing with audio that does not match any predefined model. In addition, segmentation 20 boundaries are limited to boundaries between regions of different classification. It is WO 2005/122141 PCT/AU2005/000808 -2 desirable to separate segmentation and classification, but doing so using known methods results in an unacceptable delay in reporting classifications for detected segments. A successful approach for segmenting audio that has been used is the Bayesian Information Criterion (BIC). The BIC is a classical statistical approach for assessing the 5 suitability of a distribution for a set of sample data. When applied to audio segmentation, the BIC is used to determine whether a segment of audio is better described by one distribution or two (different) distributions, hence allowing a segmentation decision to be made. It is possible to perform a second BIC-based segmentation pass over the resulting segmentation in order to eliminate segment boundaries that are not deemed statistically 10 significant. A disadvantage of such an approach is that the second BIC-based segmentation pass needs the original data on which the first segmentation was based, requiring storage for data of indefinite length. Summary of the Invention It is an object of the present invention to substantially overcome, or at least 15 ameliorate, one or more disadvantages of existing arrangements. According to an aspect of the invention, there is provided a method of classifying a signal segment, said method comprising the steps of: (a) receiving a sequence of frame feature data, each of said frame feature data characterising a frame of data along said signal segment; 20 (b) in response to receipt of each of said frame feature data, updating statistical data, characterising said signal segment, with the received frame feature data; (c) receiving a notification of an end boundary of said signal segment; and (d) classifying said signal segment based on said statistical data.

WO 2005/122141 PCT/AU2005/000808 -3 According to another aspect of the invention there is provided a method of classifying segments of a signal, said method comprising the steps of: (a) receiving a sequence of segmentation frame feature data, each of said frame feature data characterising a frame of data along said signal; 5 (b) in response to receipt of each of said frame feature data of a current segment, updating current statistical data, characterising said current segment, with the received frame feature data; (c) receiving a notification of an end boundary of said current segment; (d) in response to receipt of said notification, comparing said current statistical 10 data with statistical data characterising a preceding segment; and (e) merging current and preceding segments, or classifying said preceding signal segment based on said statistical data characterising said preceding segment, based upon the difference between said current statistical data and said statistical data characterising said preceding segment. 15 According to yet another aspect of the invention there is provided a method for processing an audio signal comprising the steps of: (a) providing a plurality of predetermined, pre-trained models; (b) providing an audio signal for processing in accordance with said models; (c) segmenting said audio signal into homogeneous portions whose length is 20 not limited by a predetermined constant; and (d) classifying at least one of said portions with reference to at least one of said models; WO 2005/122141 PCT/AU2005/000808 -4 wherein said segmenting step is performed independently of said classifying step, and step of classifying a homogeneous portion begins before segmenting step has identified the end of said portion. According to another aspect of the invention there is provided a method of 5 segmenting an audio signal into a series of homogeneous portions comprising the steps of: receiving input consisting of a sequence of frames, each frame consisting of a sequence of signal samples; calculating feature data for each said frame, forming a sequence of calculated feature data each corresponding to one of said frames; 10 in response to receipt of each said calculated feature data of a current segment, updating current statistical data with the received frame feature vector, said current statistical data characterising said current segment; determining a potential end boundary for the current segment; in response to determining a potential end boundary, comparing said current 15 statistical data with statistical data characterising a preceding segment; and merging said current and preceding segments or accepting said preceding segment as a completed segment, based upon the difference between said current statistical data and said statistical data characterising said preceding segment. According to another aspect of the invention there is provided a method of 20 segmenting an audio signal into a series of homogeneous portions comprising the steps of: receiving input consisting of a sequence of frames, each frame consisting of a sequence of signal samples; calculating a feature for each said frame, forming a sequence of calculated features each corresponding to one of said frames, wherein said feature is the product of the energy WO 2005/122141 PCT/AU2005/000808 -5 value of a frame with a weighted sum of the bandwidth and the frequency centroid of a frame; and detecting transition points in the sequence of calculated features using BIC over subsequences of calculated features, said transition points delineating homogeneous 5 segments. Other aspects of the invention are also disclosed. Brief Description of the Drawings One or more embodiments of the present invention will now be described with reference to the drawings, in which: 10 Fig. 1 shows a schematic block diagram of a single-pass segmentation and classification system; Fig. 2 shows a schematic block diagram of a general-purpose computer, upon which the segmentation and classification systems described herein may be practiced; Fig. 3 shows a schematic flow diagram of a process performed by the single-pass 15 segmentation and classification system of Fig. 1; Fig. 4 shows a schematic flow diagram of the sub-steps of a step for extracting frame features performed in the process of Fig. 3; Fig. 5A illustrates a distribution of example frame features and the distribution of i Gaussian event model that best fits the set of frame features obtained from a segment of 20 speech; Fig. 5B illustrates a distribution of the example frame features of Fig. 5A and the distribution of a Laplacian event model that best fits the set of frame features; WO 2005/122141 PCT/AU2005/000808 -6 Fig. 6A illustrates a distribution of example frame features and the distribution of a Gaussian event model that best fits the set of frame features obtained from a segment of music; Fig. 6B illustrates a distribution of the example frame features of Fig. 6A and the 5 distribution of a Laplacian event model that best fits the set of frame features; Fig. 7 shows a schematic flow diagram of the sub-steps of a step for segmenting frames into homogeneous segments performed in the process of Fig. 3; Fig. 8 shows a plot of the distribution of a clip feature vector comprising two clip features; 10 Fig. 9 illustrates the classification of the segment against 4 known classes A, B. C and D; Fig. 10 shows an example five-mixture Gaussian mixture model for a sample of two-dimensional speech features; and Fig. 11 shows a schematic block diagram of a two-pass segmentation and 15 classification system. Detailed Description Some portions of the description which follow are explicitly or implicitly presented in terms of algorithms and symbolic representations of operations on data within a computer memory. These algorithmic descriptions and representations are the 20 means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not WO 2005/122141 PCT/AU2005/000808 -7 necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It should be borne in mind, however, that the above and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels. 5 Unless specifically stated otherwise, and as apparent from the following, it will be appreciated that throughout the present specification, discussions refer to the action and processes of a computer system, or similar electronic device, that manipulates and transforms data represented as physical (electronic) quantities within the registers and memories of the computer system into other data similarly represented as physical 10 quantities within the computer system memories or registers or other such information storage, transmission or display devices. Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the 15 contrary intention appears. Fig. 1 shows a schematic block diagram of a single-pass segmentation and classification system 200 for segmenting an audio stream in the form of a sequence x(n) of sampled audio from unknown origin into homogeneous segments, and then classifying those homogeneous segments to thereby assign to each homogeneous segment an object 20 label describing the sound contained within the segment. Segmentation may be described as the process of finding transitions in an audio stream such that data contained between two transitions is substantially homogeneous. Such transitions may also be termed boundaries, with two successive boundaries respectively define the start and end points of WO 2005/122141 PCT/AU2005/000808 -8 a homogeneous segment. Accordingly, a homogeneous segment is a segment only containing samples from a source having constant acoustic characteristics. Fig. 2 shows a schematic block diagram of a general-purpose computer 100 upon which the single-pass segmentation and classification system 200 may be practiced. The 5 computer 100 comprises a computer module 101, input devices including a keyboard 102, pointing device 103 and a microphone 115, and output devices including a display device 114 and one or more loudspeakers 116. The computer module 101 typically includes at least one processor unit 105, a memory unit 106, input/output (1/0) interfaces including a video interface 107 for the 10 video display 114, an 1/0 interface 113 for the keyboard 102, the pointing device 103 and interfacing the computer module 101 with a network 118, such as the Internet, and an audio interface 108 for the microphone 115 and the loudspeakers 116. A storage device 109 is provided and typically includes a hard disk drive and a floppy disk drive. A CD-ROM or DVD drive 112 is typically provided as a non-volatile source of data. The 15 components 105 to 113 of the computer module 101, typically communicate via an interconnected bus 104 and in a manner which results in a conventional mode of operation of the computer module 101 known to those in the relevant art. One or more of the modules of the single-pass segmentation and classification system 200 may alternatively be implemented using an embedded device having dedicated 20 hardware or a digital signal processing (DSP) chip(s). Audio data for processing by the single-pass segmentation and classification system 200 may be derived from a compact disk or video disk inserted into the CD-ROM or DVD drive 112 and may be received by the processor 105 as a data stream encoded in a particular format. Audio data may alternatively be derived from downloading audio data WO 2005/122141 PCT/AU2005/000808 -9 from the network 118. Yet another source of audio data may be recording audio using the microphone 115 in which case the audio interface 108 samples an analog signal received from the microphone 115 and provides the audio data to the processor 105 in a particular format for processing and/or storage on the storage device 109. 5 The audio data may also be provided to the audio interface 108 for conversion into an analog signal suitable for output to the loudspeakers 116. The single-pass segmentation and classification system 200 is implemented in the general-purpose computer 100 by a software program executed by the processor 105 of the general-purpose computer 100. It is assumed that the audio stream is appropriately 10 digitised at a sampling rate F. Those skilled in the art would understand the steps required to convert an analog audio stream into the sequence x(n) of sampled audio. In a preferred implementation the audio stream is sampled at a sampling rate F of 16 kHz and the sequence x(n) of sampled audio is stored on the storage device 109. Fig. 3 shows a schematic flow diagram of a process 400 performed by the single 15 pass segmentation and classification system 200, and reference is made jointly to Figs. I and 3 during the description of the single-pass segmentation and classification system 200. Process 400 starts in step 402 where the sequence x(n) of sampled audio is read from the storage device 109 by a streamer 210 and divided into frames. Each frame 20 contains K audio samples x(n). K is preferably a power of 2, allowing the most efficient Fast Fourier Transform (FFT) to be used on the frame in later processing. In the preferred implementation each frame is 16ms long, which means that each frame contains 256 audio samples x(n) at the sampling rate F of 16 kHz. Further, the frames are overlapping, with the start position of the next frame positioned 8 ms, or 128 samples, later. The WO 2005/122141 PCT/AU2005/000808 - 10 streamer 210 is configured to produce one audio frame at a time to a feature calculator 220, or to indicate that not enough audio data is available to complete a next frame. The feature calculator 220 receives and processes one frame at a time to extract frame features in step 404 for each frame, that is from the K audio samples x(n) of the 5 frame being processed by the feature calculator 220. Once the feature calculator 220 has extracted the frame features, the audio samples x(n) of that frame is no longer required, and may be discarded. The frame features are used in the steps that follow to segment the audio sequence and to classify the segments. Fig. 4 shows a schematic flow diagram of step 404 in more detail. Step 404 10 starts in sub-step 502 where the feature calculator 220 applies a Hamming window function to the sequence samples x(n) in the frame i being processed, with the length of the Hamming window function being the same as that of the frame, i.e. K samples long, to give a modified set of windowed audio samples s(i,k) for frame i, with ke 1,...,K. The purpose of applying the Hamming window is to reduce the side-lobes created when 15 applying the Fast Fourier Transform (FFT) in subsequent operations. In sub-step 504 the feature calculator 220 extracts the frequency centroid fc of the modified set of windowed audio samples s(i,k) of the i'th frame, with the frequency centroidfc being defined as: fc(i)= )dco f S;(m) dco( 20 where co is a signal frequency variable for the purposes of calculation and IS (o is the power spectrum of the modified windowed audio samples s(i,k) of the i'th frame. The Simpson's Rule of integration is used to evaluate the integrals. The Fast WO 2005/122141 PCT/AU2005/000808 - 11 Fourier Transform is used to calculate the power spectrum |S;( 2 whereby the samples s(i,k), having length K, are zero padded until the next highest power of 2 is reached. In the preferred implementation where the length K of the samples s(i,k) is 256, no padding is needed. 5 Next in sub-step 506 the feature calculator 220 extracts the bandwidth bw(i) of the modified set of windowed audio samples s(i,k) of the i'th frame as follows: (C0 - FC(i))2|IS; (Cyd bw(i) = 1- F)) dco (2) IS;(i d ~S ( dco In sub-step 508 the feature calculator 220 extracts the energy E(i) of the modified set of windowed audio samples s(i,k) of the i'th frame as follows: 10 E(i)= s 2 (i,k) (3) kI A segmentation frame feature fs(i) for the i-th frame is calculated by the feature calculator 220 in sub-step 510 by multiplying the weighted sum of frame bandwidth bw(i) and frequency centroid fc(i) by the frame energy E(i). This forces a bias in the measurement of bandwidth bw(i) and frequency centroidfc(i) in those frames that exhibit 15 a higher energy E(i), and are thus more likely to come from an event of interest, rather than just background noise. The segmentation frame featurefs(i) is thus calculated as: f, (i) = E(i)(( - a) x bw(i) + a x fc(i)) (4) where a is a configurable parameter, preferably 0.4. Step 404 ends in sub-step 512 where the feature calculator 220 extracts the zero 20 crossing rate (ZCR) of the windowed audio samples s(i,k) within frame i. The ZCR within WO 2005/122141 PCT/AU2005/000808 -12 a frame i represents the rate at which the windowed audio samples s(i,k) cross the expected value of the windowed audio samples s(i,k). When the windowed audio samples s(i,k) have a mean of zero, then the ZCR represents the rate at which the signal samples cross the zero signal line. Thus for the ith frame the ZCR(i) is calculated as: K 5 ZCR(i) = jjsign(s(i,k)- ,u)-sign(s(i,k -1)- ,u, (5) k=1 wherein ps is the mean of the K windowed audio samples s(i,k) within frame i. Referring again to Figs. 1 and 3, the frame features extracted by the feature calculator 220, which comprise the frame energy E(i), frame bandwidth bw(i), frequency centroidfc(i), segmentation frame featurefs(i) and zero crossing rate ZCR(i), are received 10 by a segmenter 230 which segments the frames into homogeneous segments in step 408. In particular, the segmenter 230 utilises the Bayesian Information Criterion (BIC) applied to the segmentation frame features fs(i) for segmenting the frames into a number of homogeneous segments. Most previous BIC systems have used multi-dimensional features, such as mel-cepstral vectors or linear predictive coefficients, which are 15 computational expensive due to costly computations involving full-covariance matrices and mean-vectors. The segmentation frame feature fs(i) used by the segmenter 230 is a one-dimensional feature. The BIC provides a value which is a statistical measure for how well a chosen model represents a set of segmentation frame featuresfs(i), and is calculated as: D 20 BIC = log(L) - D log(N) (6) 2 where L is the maximum-likelihood probability for the chosen model to represent the set of segmentation frame featuresfs(i), D is the dimension of the model which is 1 when the WO 2005/122141 PCT/AU2005/000808 - 13 segmentation frame features fs(i) of Equation (4) are used, and N is the number of segmentation frame featuresfs(i) being tested against the model. The maximum-likelihood L is calculated by finding parameters 0 of the model that maximise the probability of the segmentation frame features fs(i) being from that 5 model. Thus, for a set of parameters 0, the maximum-likelihood L is: L = max P(f,(i)|o) (7) Segmentation using the BIC operates by testing whether the sequence of segmentation frame features fs(i) is better described by a single-distribution event model, or a twin-distribution event model, where the first m number of frames, those being 10 frames [1,...,m], are from a first source and the remainder of the N frames, those being frames [m+1,...,N], are from a second source. The frame m is termed the change-point. To allow a comparison, a criterion difference ABIC is calculated between the BIC: using the twin-distribution event model and that using the single-distribution event-model. As the change-point m approaches a transition in acoustic characteristics, the criterion 15 difference ABIC typically increases, reaching a maximum at the transition, and reducing again towards the end of the N frames under consideration. If the maximum criterion difference ABIC is above a predefined threshold, then the two-distribution event model is deemed a more suitable choice, indicating a significant transition in acoustic characteristics at the transition-point m where the criterion difference ABIC reached a 20 maximum. A range of different statistical event models can be used with the BIC method. The most commonly used event model is a Gaussian event model. Most BIC segmentation systems assume that D-dimensional segmentation frame features fs(i) are WO 2005/122141 PCT/AU2005/000808 - 14 best represented by a Gaussian event model having a probability density function of the form: g(f,(i, 2) ) exp{- (f,(i)- )TE~'(f,(i)- L) (8) (2;r)|l 2 where t is the mean vector of the segmentation frame features fs(i), and E is the 5 covariance matrix. The segmentation frame featurefs(i) of the preferred implementation is one-dimensional and as calculated in Equation (4). The maximum log likelihood of N segmentation features f,(i) fitting the probability density function shown in Equation (8) is: log(L) log - (f(i)-p) 2 (9) 2 2a- ,i 10 Fig. 5A illustrates a distribution 500 of segmentation frame features fs(i), where the segmentation frame features fs(i) were obtained from an audio stream of duration'i second containing voice. Also illustrated is the distribution of a Gaussian event model 502 that best fits the set of segmentation frame featuresfs(i). It is proposed that segmentation frame features fs(i) representing the 15 characteristics of audio signals such as a particular speaker or block of music, is better represented by a leptokurtic distribution, particularly where the number N of frame features fs(i) being tested against the model is small. A leptokurtic distribution is a distribution that is more peaked than a Gaussian distribution. An example of a leptokurtic distribution is a Laplacian distribution. Fig. 5B illustrates the distribution 500 of the same 20 segmentation frame features fs(i) as those of Fig. 5A, together with the distribution of a Laplacian event model 505 that best fits the set of segmentation frame featuresfs(i). It can WO 2005/122141 PCT/AU2005/000808 - 15 be seen that the Laplacian event model gives a much better characterisation of the feature distribution 500 than the Gaussian event model. This proposition is further illustrated in Figs. 6A and 6B wherein a distribution 600 of segmentation frame features fs(i) obtained from an audio stream of duration 1 5 second containing music is shown. The distribution of a Gaussian event model 602 that best fits the set of segmentation frame features fs(i) is shown in Fig. 6A, and the distribution of a Laplacian event model 605 is illustrated in Fig. 6B. A quantitative measure to substantiate that the Laplacian distribution provides a better description of the distribution characteristics of the segmentation frame featuresfs(i) 10 for short events rather than the Gaussian model is the Kurtosis statistical measure IC, which provides a measure of the "peakiness" of a distribution and may be calculated for a sample set X as: E(X - E(X))4 K (= -- 3 (10) (var(X)) 2 For a true Gaussian distribution, the Kurtosis measure K is 0, whilst for a true 15 Laplacian distribution the Kurtosis measure K is 3. In the case of the distributions 500 and 600 shown in Figs. 5A and 6A, the Kurtosis measures K are 2.33 and 2.29 respectively. Hence the distributions 500 and 600 are more Laplacian in nature than Gaussian. The Laplacian probability density function in one dimension is: 1-2 { lf,(i)-p} 20 g(f, (i), P, O exp - (1) -a a- WO 2005/122141 PCT/AU2005/000808 -16 where p is the mean of the segmentation frame features fs(i) and a is their standard deviation. In a higher order feature space with segmentation frame features fs(i), each having dimension D, the feature distribution is represented as: g(f{ (i), P, )) = () Kv (2(f, (i)- p)E-'(f,(i)- p) 5 (12) where v = (2- D) /2 and K (.) is the modified Bessel function of the third kind. Whilst the segmentation performed in step 408 may be performed using multi dimensional segmentation features fQ(i), as noted above, the preferred implementation uses the one-dimensional segmentation frame feature fi(i) shown in Equation (4). 10 Accordingly, given N segmentation frame features fs(i), the maximum likelihood L-for the set of segmentation frame featuresfs(i) falling under a single Laplacian distribution is: N J L =1 H(2.2)-2 ex( _- f,(i)- p (13) where o is the standard deviation of the segmentation frame features fs(i) and p is the mean of the segmentation frame featuresfs(i). Equation (13) may be simplified in order to 15 provide: N F L = (2U)2 exp -ZZfi-iI (14) The maximum log-likelihood log(L), assuming natural logs, for all N frame featuresf(i) to fall under a single Laplacian event model is thus: log(L)= N (22)_ 1 -I (15) 2 0 .

WO 2005/122141 PCT/AU2005/000808 - 17 A log-likelihood ratio R(m) provides a measure of the frames belonging to a twin-Laplacian distribution event model rather than a single Laplacian distribution event model, with the division in the twin-Laplacian distribution event model being at frame m, is: 5 R(m) = log(LI) + log(L 2 )- log(L) (16) where: log(L)= - ( 20 )-I f ,(Ij )-pli (17) 2 and log(L2 =-(N-m)(2I - f,(i)-p21 (18) 2 C2 i=m+1 10 wherein { p., a } and { p 2

,*

2 } are the means and standard-deviations of the segmentation frame featuresfs(i) before and after the change point m. The criterion difference ABIC for the Laplacian case having a change point m is calculated as: ABIC(m) = R(m) - I logm( m) (19) 2N 15 In the preferred implementation of the BIC, a segmentation window is filled with a sequence of N segmentation frame features fs(i). It is then determined by the segmenter 230 whether the centre of the segmentation window defines a transition. In the case where the centre does not define a transition, the segmentation window is advanced by a predetermined number of frames before the centre is again tested. 20 Fig. 7 shows a schematic flow diagram of the sub-steps of step 408 (Fig. 3). Step 408 starts in sub-step 702 where the segmenter 230 buffers segmentation frame features WO 2005/122141 PCT/AU2005/000808 -18 fs(i) until the segmentation window is filled with N segmentation frame features fs(i). Preferably the segmentation window is N=80 frames long. Since it is assumed that the frames in the first half of the segmentation window do belong to the current segment being formed, the frame features of the frames in the 5 first half of the segmentation window are passed to a classifier 240 in sub-step 703 for further processing. The segmenter 230 then, in sub-step 704, calculates the log-likelihood ratio R(m) by first calculating the means and standard deviations {u , oI} and {p 2 ,o, I of the segmentation frame features fs(i) in the first and second halves of the segmentation window respectively. Sub-step 706 follows where the segmenter 230 calculates the 10 criterion difference ABIC(m) using Equation (19). Then, in sub-step 708, the segmenter 230 determines whether the centre of the segmentation window is a transition between two homogeneous segments by determining whether the criterion difference ABIC(m) is greater than a predetermined threshold, which is set to 0 in the preferred implementation. 15 .1 If it is determined in sub-step 708 that the centre of the segmentation window is not a transition between two homogeneous segments, then the segmenter 230 in sub-step 710 shifts the segmentation window forward in time by removing a predetermined number of the oldest segmentation frame featuresfs(i) from the segmentation window and adding the same number of new segmentation frame featuresfs(i) thereto. In the preferred 20 implementation the predetermined number of frames is 10. As soon as a segmentation frame feature fs(i) passes the centre of the segmentation window, it is known that the frame i represented by the segmentation frame featurefs(i) is part of a current segment being formed. Accordingly, the frame features of WO 2005/122141 PCT/AU2005/000808 -19 the frames that shifted past the centre of the segmentation window are passed to a classifier 240 in sub-step 712 for further processing before step 408 returns to sub-step 704 from where the segmenter 230 again determines whether the centre of the shifted segmentation window defines a transition. 5 The segmentation window may be easily implemented using a data structure known as a circular buffer, allowing frame feature data to be shifted as more data becomes available, and allowing old data to be removed once the data moved through the circular buffer. Sub-steps 704 to 712 continue until the segmenter 230 finds a transition. Step 10 408 then continues to sub-step 714 where the frame number i of the frame where the transition occurred is also passed to the classifier 240. The frame number i of the frame where the transition point occurred may optionally also be reported to a user interface for display on the video display 114 (Fig. 2). In sub-step 716 all the segmentation frame features fs(i) that have been 15 determined to belong to the current segment are flushed from the segmentation window. The operation of the segmenter 230 then returns to sub-step 702 where the segmenter 230 again buffers segmentation frame features fs(i) until the segmentation window is filled with N segmentation frame features fs(i) before the segmenter 230 starts to search for the next transition between segments. 20 Referring again to Figs. 1 and 3, as is described above with reference to the segmentation step 408, the classifier 240 receives from the segmenter 230 the frame features, calculated using Equations (1) to (5), of all the frames belonging to the current segment, even while a transition has not as yet been found. When the transition is located the classifier 240 receives the frame number of the transition, or last frame in the current WO 2005/122141 PCT/AU2005/000808 - 20 segment. This allows the classifier 240 to build up statistics of the current segment in order to make a classification decision as soon as the classifier 240 receives notification that a transition has been found, in other words, that the boundary of the current segment has been found. The classification decision is delayed by only half of the segmentation 5 window length, which is 40 frames in the preferred implementation. Since the classifier 240 does not add any delay to the system 200, and a delay of 40 frames is a relatively small delay, system 200 is extremely responsive. In order to classify the homogeneous segment, the classifier 240 extracts a number of statistical features from the segment. However, whilst previous systems 10 extract a feature vector from the segment and then classify the segment based on the feature vector, the classifier 240 divides each homogenous segment into a number of smaller sub-segments, or clips, with each clip large enough to extract a meaningful clip feature vector f from the clip. The clip feature vectors f are then used to classify the associated segment based on the characteristics of the distribution of the clip feature 15 vectors f. The key advantage of extracting a number of clip feature vectors f from a series of smaller clips rather than a single feature vector for a whole segment is that the characteristics of the distribution of the feature vectors f over the segment of interest may be examined. Thus, whilst the signal characteristics over the length of the segment are expected to be reasonably consistent, some important characteristics in the distribution of 20 the clip feature vectors f over the segment of interest is significant for classification purpose. Each clip comprises B frames. In the preferred implementation where each frame is 16 ms long and overlapping with a shift-time of 8 ms, each clip is defined to be at least 0.64 seconds long. The clip thus comprises at least 79 frames.

WO 2005/122141 PCT/AU2005/000808 -21 The classifier 240 then extracts a clip feature vector f for each clip from the frame features received from the segmenter 230, and in particular the frame energy E(i), frame bandwidth bw(i), frequency centroid fc(i), and zero crossing rate ZCR(i) of each frame within the clip. In the preferred implementation, the clip feature vector f for each 5 clip consists of six different clip features, which are: (i) volume standard deviation; (ii) volume dynamic range; (iii) zero-crossing rate standard deviation; (iv) bandwidth; 10 (v) frequency centroid; and (vi) frequency centroid standard deviation. The volume standard deviation (VSTD) is a measure of the variation characteristics of the root means square (RMS) energy contour of the frames within the clip. The VSTD is calculated over the B frames of the clip as: B 15 VSTD = F'B ,(20) wherein E(i) is the energy of the modified set of windowed audio samples s(i,k) of the i'th frame calculated in sub-step 508 (Fig. 4) using Equation (3) and pE is the mean of the B frame energies E(i). The volume dynamic range (VDR) is similar to the VSTD. However the VDR 20 measures the range of deviation of the energy values E(i) only, and as such is calculated as: WO 2005/122141 PCT/AU2005/000808 - 22 VDR = ax(E(i)) , where ie[1, 2,..., B] (21) The zero-crossing rate standard deviation (ZSTD) clip feature examines the standard deviation of the zero-crossing rate (ZCR) over all frames in the clip of interest. The ZSTD clip feature is then calculated over B frames as: B Z (ZCR(i)-zc )2 5 ZSTD= B-1 (22) B-I wherein pzcR is the mean of the ZCR values calculated using Equation (5). The dominant frequency range of the signal is estimated by the signal bandwidth. In order to calculate a long-term estimate of bandwidth BW over a clip, the frame bandwidths bw(i) (calculated using Equation (2)) are weighted by their respective frame 10 energies E(i) (calculated using Equation (3)), and summed over the entire clip. Thus the clip bandwidth BWis calculated as: BW- B 1 E(ibw(i) (23) E(i)' i=1 The fundamental frequency of the signal is estimated by the signal frequency centroid. In order to calculate a long-term estimate of frequency centroid (FC) over a clip, 15 the frame frequency centroids fc(i) (calculated using Equation (1)) are weighted by their respective frame energies E(i) (calculated using Equation (3)), and summed over the entire clip. Thus the clip frequency centroid FC is calculated as: FC- 1 ZE(i)fc(i) (24) E(i) j=I WO 2005/122141 PCT/AU2005/000808 -23 The frequency centroid standard deviation (FCSTD) attempts to measure the characteristics of the frequency centroid variation over the clip of interest. Frequency centroid is an approximate measure of the fundamental frequency of a section of signal; hence a section of music or voiced speech will tend to have a smoother frequency centroid 5 contour than a section of silence or background noise. With the clip features calculated, the clip feature vector f is formed by assigning each of the six clip features as an element of the clip feature vector f as follows: VSTD VDR ZSTD f= (25) BW FC FCSTD To illustrate the nature of the distribution of the clip features over a homogenous 10 segment, Fig. 8 shows a plot of the distribution of two particular clip features, namely the volume dynamic range (VDR) and volume standard deviation (VSTD), over a set of segments containing speech, and a set of segments containing background noise. The distributions of clip feature vectors, as shown in this example, are clearly multi-modal in nature. 15 With the clip feature vectors f extracted, the classifier 240 operates to solve what is known in pattern recognition literature as an open-set identification problem. The open-set identification may be considered as a combination between a standard closed-set identification scenario and a verification scenario. In a standard closed-set identification scenario, a set of test features from unknown origin are classified against features from a 20 finite set of classes, with the most probable class being allocated as the identity label for WO 2005/122141 PCT/AU2005/000808 - 24 the object associated with the set of test features. In a verification scenario, again a set of test features from an unknown origin is presented. However, after determining the most probable class, it is then determined whether the test features match the features of the class closely enough in order to verify its identity. If the match is not close enough, the 5 identity is labelled as "unknown". Hence, the classifier 240 classifies the current segment in step 410 (Fig. 3) as either belonging to one of a number of pre-trained models, or as unknown. The open-set identification problem is well suited to classification in an audio stream, as it is not possible to adequately model every type of event that may occur in an 10 audio sample of unknown origin. It is therefore far better to label an event, which is dissimilar to any of the trained models, as "unknown", rather than falsely labelling that event as another class. Fig. 9 illustrates the classification of the segment, characterised by its extracted clip feature vectors f, against 4 known classes A, B, C and D, with each class being 15 defined by an object model. The extracted clip feature vectors f are "matched" against the object models by determining a model score between the clip feature vectors f of the segment and each of the object models. An empirically determined threshold is applied to the best model score. If the best model score is above the threshold, then the label of the class A, B, C or D to which the segment was more closely matched is assigned as the 20 object label. However, if the best model score is below the threshold, then the segment does not match any of the object models closely enough, and the segment is assigned the label "unknown". Given that the distribution of clip features is multi-modal, a simple distance measure, such as Euclidean or Mahalanobis, will not suffice for calculating a score for the WO 2005/122141 PCT/AU2005/000808 - 25 classification. The classifier 240 is therefore based on a continuous distribution function defining the distribution of the clip feature vectors f. In the preferred implementation a mixture of Gaussians, or Gaussian Mixture Model (GMM) is used as the continuous distribution function. A Gaussian mixture 5 density is defined as a weighted sum of M component densities, expressed as: p(x I A) = pib,(x) (26) j=1 where x is a D dimensional random sample vector, b;(x) are the component density functions, and pi are the mixture weights. Each density function b, is a D dimensional Gaussian function of the form: 10 b, (x) = ,1 exp 1 X- III T-' (x - p,) (27) (27c)r2 ,|i 2 where L, is the covariance matrix and pthe mean vector for the density function b,. The Gaussian mixture model A,, with c =1,2,..., C where C is the number of class models, is then defined by the covariance matrix L, and mean vector I, for each 15 density function b 1 , and the mixture weights p;, collectively expressed as: AC = {pi, p,, j, : i =,.,M} (28) The characteristics of the probability distribution function p(x.z) of the GMM can be more clearly visualized when using two-dimensional sample data x. Fig. 10 shows an example five-mixture GMM for a sample of two-dimensional speech features x, and 20 X2, where x=[xi,x 2

].

WO 2005/122141 PCT/AU2005/000808 - 26 The GMM A, is formed from a set of labelled training data via the expectation maximization (EM) algorithm known in the art. The labelled training data is clip feature vectors f extracted from clips with known origin. The EM algorithm is an iterative algorithm that, after each pass, updates the estimates of the mean vector pt, covariance 5 matrix I, and mixture weights pi. Around 20 iterations are usually satisfactory for convergence. In a preferred implementation GMM's with 6 mixtures and diagonal covariance matrices L, are used. The preference for diagonal covariance matrices L, over full covariance matrices is based on the observation that GMM's with diagonal covariance 10 matrices E, are more robust to mismatches between training and test data. With the segment being classified comprising T clips, and hence. being characterised by T clip feature vectors f,, the model score between the clip feature vectors f, of the segment and one of the C object models is calculated by summing the log statistical likelihoods of each of T feature vectors f, as follows: T 15 =Jlogp(f'.Z ) (29) t=1 where the model likelihoods p(f, je) are determined by evaluating Equation (26). The log of the model likelihoods p(fllkZ) is taken to ensure no computational underflows occur due to very small likelihood values. Equation (29) may be evaluated by storing the clip feature vectors f, of all the 20 clips of the current segment in a memory buffer, and calculating the model scores se only when the end of the segment has been found. The amount of memory required for such a WO 2005/122141 PCT/AU2005/000808 - 27 buffer is determined by the length of the segment. For segments of arbitrary length, this memory requirement is unbounded. To alleviate the above noted problem, an incremental method is implemented in the preferred implementation. It is noted that Equation (29) is just a simple summation of 5 the logs of the model likelihood of each individual clip, independent of other clips. This enables the algorithm to accumulate the frame features of the current segment until enough frame features have been accumulated to form a clip. The clip feature vector f, for that clip is then extracted, and the newly calculated clip feature vector f, to update the model scores s, by using the equation: 10 se = se +log p(f, IA") (30) The memory buffer used to store the clip feature vector f,, as well as a certain number of the feature vectors of the frames making up the clip, may then be cleared as that data is no longer required. In the preferred implementation where the clips are overlapping by half the length of the clip, half of the feature vectors of the frames making 15 up the clip may be discarded. Once the boundary (end) of the current segment is detected, the model scores s are used by the classifier 240 to classify the current segment. The classification of the current segment, along with the boundaries thereof, may then be reported to the user via the user interface, typically through the video display 114. The classifier 240 then 20 empties its buffers from frame feature and clip feature vector f, data, and resets the model scores s, to zero before starting with the classification of a next segment. In an alternative implementation of the single-pass segmentation _ and classification system 200 the intermediate values of the model scores s, calculated using WO 2005/122141 PCT/AU2005/000808 -28 Equation (30) are used to determine a preliminary classification for the current segment, even before the boundary of the current segment has been found by the segmenter 230. The preliminary classification serves as an indication of what the final classification for the current segment is most likely to be. While the preliminary classification may not be 5 as accurate as the final classification, reporting the preliminary classification has advantages, which are explored later in the description. As is described in relation to Fig. 9, an adaptive algorithm is employed by the classifier 240 to determine whether the model corresponding to a best model score s, truly represents the segment under consideration, or whether the segment should rather be 10 classified as "unknown". The best model score s, is defined as: s, = max(s ) (31) The adaptive algorithm is based upon a distance measure Dig between object models of the classes to which the test segment may belong. Fig. 10 illustrates four classes and the inter-class distances Dij between each object model i andj. As the object 15 models are made up of a mixture of Gaussians, the distance measure Dij is based on a weighted sum of the Mahalanobis distance between the mixtures of the models i and jas follows: M N Dy= pi pjiA'l (32) mn=I n=1 where M and N are the number of mixtures in class models 20 i and respectively, p' and p, are the mixture weights within each model, and A' is the Mahalanobis distance between mixture mof class i and mixture n of class j. The WO 2005/122141 PCT/AU2005/000808 - 29 inter-class distances Dij may be predetermined from the set of labelled training data, and stored in memory 106. The Mahalanobis distance between two mixtures is calculated as: 5 Because diagonal covariance matrices are used, the two covariance matrices E' and I-, may simply be added in the manner shown. It is noted that the Mahalanobis distance S., is not strictly speaking a correct measure of distance between two distributions. When the distributions are the same, the distance should be zero. However, this is not the case for the Mahalanobis distance S1 defined in Equation (33). For this to 10 be achieved, various constraints have to be placed on Equation (32). This adds a huge amount of computation to the process and is not necessary for the classification, as a relative measure of class distances is all that is needed. In order to decide whether the segment should be assigned the label of the class with the highest score, or labelled as "unknown", a confidence score is calculated. This is 15 achieved by taking the difference of the top two model scores S, and sq, and normalizing that difference by the distance measure Dpq between their class models p and q. This is based on the premise that an easily identifiable segment should be a lot closer to.the model it belongs to than the next closest model. With further apart models, the model scores s, should also be well separated before the segment is assigned the class label of 20 the class with the highest score. More formally, the confidence score may be defined as: s -s CD=1000 D q (34) Dq WO 2005/122141 PCT/AU2005/000808 -30 The additional constant of 1000 is used to bring the confidence score D into a more sensible range. A threshold t is applied to the confidence score cD. In the preferred implementation a threshold T of 5 is used. If the confidence score (D is equal or above the threshold r, then the segment is given the class label of the highest model score S,, else 5 the segment is given the label "unknown". Certain aspects of the single-pass segmentation and classification system 200 ensure operation in real time and fixed memory. "Real time" is defined as an application which requires a program to respond to stimuli within some small upper limit of response time. More loosely the term "real time" is used to describe an application or a system that 10 gives the impression of being responsive to events as the events happen. One aspect that ensures the operation of system 200 in fixed memory is that audio samples are discarded early in process 400 (Fig. 3). In particular, audio samples are discarded in step 404, which is as soon as the frame features are extracted therefrom. This also eliminates movement of large blocks of data between modules 220, 230 and 240, and 15 aids in making the implementation faster. The segmenter 230 uses a sliding segmentation window for making segmentation decisions, again allowing feature vectors of frames that moved through the segmentation window to be discarded. Classification only requires a running model score s, for each model for the current segment. All modules 210, 220, 230 and 240 keep only a small or minimal buffer of data necessary to calculate features, 20 and keep on recycling these buffers by using well-known techniques such as utilising circular buffers. System 200 may be said to be operating in real time if a classification decision is produced as soon as the end boundary of a segment is found. By updating the model WO 2005/122141 PCT/AU2005/000808 -31 scores s, continuously, very little processing is necessary when the boundary is found. Also, in the implementation where the preliminary classification is provided, the system 200 produces a classification decision even before the boundary of the segment has been found. 5 The above describes the single-pass segmentation and classification system 200. Fig. 11 shows a schematic block diagram of a two-pass segmentation and classification system 290 for segmenting an audio stream from unknown origin into homogeneous segments, and then classifying those homogeneous segments. Two-pass segmentation differs from single-pass segmentation in that, after two adjacent homogeneous segments 10 have been determined, the boundary between those adjacent segments is reconsidered, testing whether the two adjacent segments should be merged. The two-pass segmentation and classification system 290 may also be practiced on the general-purpose computer 100 shown in Fig. 2 by executing a software program in the processor 105 of the general-purpose computer 100. 15 The two-pass segmentation and classification system 290 is similar to the single pass segmentation and classification system 200 and also comprises a streamer 210, feature calculator 220, and segmenter 230, each of which operating in the manner described with reference to Figs. 1 and 3. The two-pass segmentation and classification system 290 further includes a controller 250, a merger 260 and a classifier 270. 20 In system 290 the controller 250 receives the frame features from the segmenter 230, and then passes the frame features to both the merger 260 and the classifier 270. The merger 260 extracts statistics, referred to as current segment statistics, from the frame features of the current segment. The classifier 270 uses the frame features to build up WO 2005/122141 PCT/AU2005/000808 - 32 model scores sc,,,,,, for the current segment in order to make a classification decision in the manner described with reference to classifier 240. The controller 250 also notifies the merger 260 and classifier 270 when a boundary of the current segment has been found by the segmenter 230. The first time the 5 merger 260 receives notification that the boundary of the current (first) segment has been found, the merger 260 saves the current segment statistics as potential segment statistics, and clears the current segment statistics. The merger 260 then notifies the controller 250 that a potential segment has been found. The controller 250, upon receipt that a potential segment has been found, notifies the same to the classifier 270. The classifier 270 10 responds to the notification from the controller 250 by saving the model scores S. ,C of the current segment into model scores s,,,,,a, of the potential segment. The classifier 270 also clears the model scores su,,,n,, of the current segment. When the merger 260 receives notification that the boundary of a subsequent - current segment has been found, the merger 260 determines whether the then, current 15 segment should be merged with the preceding segment characterised by the potential segment statistics. In other words, the validity of the end boundary of the preceding segment is verified by the merger 260. In the case where a Laplacian event model is used by the merger 260 the frame features for all frames of the current and preceding segments have to be stored in memory. 20 However, if a Gaussian event model is used, the merger 260 only needs to maintain the number N of frames in the current and preceding segments and the covariance o- of the segmentation features f,(i) for the current and preceding segments, which may be calculated incrementally within fixed memory.

WO 2005/122141 PCT/AU2005/000808 - 33 Starting off with Equation (9), the maximum log likelihood may be rewritten in terms of the number N of frames in the respective segment and the covariance a of the segmentation features fs(i) of that segment, without referring to individual segmentation featuresfs(i) as follows: 5 log(L)- Nlog(2.2)_N (35) 2 2 The covariance o is calculated incrementally using: .2 Z f,(s) 2 (Zf,(s)) 2 (6 o- = - ,S) (36) N N 2 The three terms If, (i) 2 , f,(i) and N are updated each time segmentation features fs(i) of frames are received by the merger 260. Initially each of the variables 10 sumX, sumXSquare and N are set to zero. Each time segmentation featuresfi(i) of frames are received by the merger 260, these variables sumX, sumXSquare and N are updated as follows: sumX = sumX + f, (i) sumXSquare = sumXSquare + f, (i) 2 (37) N = N +1 The covariance a is then calculated as: 15 .2 sumXSquare sumX (38) N N 2 which provides a complete set of variables to evaluate Equation (35). When a new boundary is detected, the criterion difference ABIC is calculated by first calculating: WO 2005/122141 PCT/AU2005/000808 - 34 2 sumXSquaree,,,,, sumXu current 2 Current current 2 sumXSquarepenaa .sumX,,ennal ~potential - N N 2 (39) potential Npotential 2 sumXSquarepotenti. + sumXSquare,,,potential + s current Coverall N poeta Nurn 2

*+

2 N,,, + Nur,,,, Npotential + current and substituting these values into Equation (35), to get: log(Lr,.,,,, )= - N'urre" og(2 ,, ) Ncurrent 2 current 2 log(L, ) = - Npotential o(22 N potential 40) lg(potential 2 lg2 ptnil 2 (0 log(L,,,a = - Ncurrent + Npotentia l(2 - Ncurrent + Npotential 2 2g\ 0 overall) 2 The log-likelihood ratio R(m) is then calculated as: 5 R(m) = log(Lcre,,) + log(Lpotential ) - log(Lvrau) (41) and the criterion difference ABIC as: 1( N N ABIC(m)= R(m)-log current potential (42) 2 ( Nu,,ent + Npotetia) The criterion difference ABIC is then compared with a significant threshold hnerge. The significant threshold merge is a parameter that can be adjusted to change the 10 sensitivity of the determination of whether the segments should be merged. In the preferred implementation the significant threshold hmerge has a value of 30. In the case where the merger 260 determines that the current and preceding segments should be merged based on the current and potential segment statistics, the merger 260 merges the current and preceding segments into the preceding segment by 15 merging the current and potential segment statistics into the potential segment statistics as follows,: WO 2005/122141 PCT/AU2005/000808 - 35 sumX,,,, = sumX,,,, + sumX S m potential - S m potential +S M current sumXSquareofnfa, = sumXSquarepoenial + sumXSquarecu,,,,, (43) Npoenia, = Npotenia + Ncurrent and clears the current segment statistics. The merger 260 additionally notifies the controller 250 that the current and preceding segments have been merged. Upon receipt of a notification by the controller 250 from the merger 260 that the 5 current and preceding segments have been merged, the controller 250 notifies the same to the classifier 270. The classifier 270 in turn, upon receipt of the notification from the controller 250, merges the model scores scurrentC of the current segment with the model scores potential, of the potential segment and saves the result as the model scores spofenialc of the potential segment through: 10 spoteniac = scurren,c +spoenial,c (44) The model scores surrentc of the current segment are also cleared by the classifier 270. No classification decision is produced by the classifier 270 upon merging of the preceding and current segments. Alternatively, in the case where the merger 260 determines that the current and 15 preceding segments should not be merged based on the current and potential segment statistics, the merger 260 saves the current segment statistics into the preceding segment statistics as follows: sumXptential = sUM)current sumXSquarepotential = sumXSquarec,,,,,t (45) Npoentia = Acurrent and clears the current segment statistics. The merger 260 additionally notifies 20 the controller 250 that the current and preceding segments have not been merged. -- . .. ~ - In f I WO 2005/122141 PCT/AU2005/000808 -36 Upon receipt of a notification by the controller 250 from the merger 260 that the current and preceding segments have not been merged, the controller 250 notifies the same to the classifier 270. The classifier 270 in turn, upon receipt of the notification from the controller 250, classifies the preceding segment based on the potential segment model 5 scores spoentia,c and passes the classification decision to the user interface in the manner described with reference to classifier 240 in Fig. 1. Additionally the classifier 270 saves the model scores s.e,t,, of the current segment as the model scores s,,,,,,,, of the potential segment, and clears the model scores s,,,.,.e,,, of the current segment. It is noted that the two-pass segmentation and classification system 290 10 introduces an unbounded delay between when a segment boundary is detected and when the classification of the segment is reported. This is because, when a segment boundary is detected, the system 290 still has to decide whether the segment defined by the segment boundary is a finalized segment. This decision is delayed until a subsequent segment has been detected, and the merger 260 has unsuccessfully tried to merge the two segments. In 15 the case where the two segments are merged, the preceding segment is expanded to include the newest segment and no classification is reported. Since segments have arbitrary length, it is not possible to predict when the system 290 will detect the following segment and be able to test whether the two segments need to be merged. In cases where the unbounded delay between when a segment boundary is 20 detected and when the segment classification is reported is undesirable, the unbounded delay may be avoided by specifying a maximum length for any segment. This would place an upper bound on the latency.

WO 2005/122141 PCT/AU2005/000808 - 37 Applications of the segmentation and classification systems 200 and 290 will now be described. In a first application the segmentation and classification systems 200 and 290 form part of an improved security apparatus. Most simple security systems today record all data that is received thereby. This approach is very costly in terms of the 5 storage space requirements. When the need arises to go through the data, the massive amounts of data recorded makes the exercise prohibitive. Accordingly, the improved security apparatus discards data considered uninteresting. The proposed improved security system receives audio/visual data (AV data) through connected capture devices. Each of the audio and video data is then analysed 10 separately for "interesting" events. For example, motion detection may be performed on the video data. The audio data received by the improved security system is further processed by either of the segmentation and classification systems 200 and 290 to segment the audio data into segments and to classify each segment into one of the available classes, or as 15 unknown, where some of the available classes have been marked as interesting for capture. The interesting segments of the AV data are then written to permanent storage. The improved security system uses a buffer, called an unclassified buffer, to store the current segment while that segment is being classified. Since segments can be potentially arbitrarily long, and the final classification is not reported until the segment is 20 completed, the size of the buffer is substantial. The size of the unclassified buffer may be reduced with the use of the preliminary classification. The preliminary classification gives the improved security system an indication of what the classification is most likely to be, and this information may be utilised in a variety of ways, some of which are explored below: WO 2005/122141 PCT/AU2005/000808 - 38 1) The improved security system may discard all data until it receives at least a preliminary classification. If this preliminary classification is consistently interesting, there is a fair chance that the entire segment will be classified as interesting. In this case the system writes the data directly to permanent storage, thereby avoiding buffering the 5 data. 2) The improved security system may store the audio/video data using a varying level of data loss, with the level of data loss depending on what percentage of the portion had an interesting classification. 3) Depending on the length of segments, the improved security system may save 10 only interesting portions of segments, i.e. portions having a preliminary classification of interesting. The most suitable option will depend on a trade-off between the cost of buffering data, and how much data loss can be safely tolerated. Another application of the segmentation and classification systems 200 and 290 15 is filtering of an input to a speech recognition system. Most simple speech recognition systems treat all input as potential speech. Such systems then try to recognise all types of audio data as speech and this causes mis-recognition in many cases. The speech recognition system using either of systems 200 and 290 first classifies all received sound as either speech or non-speech. All non-speech data is discarded, and recognition 20 algorithms are run on portions of audio classified as speech, resulting in better results. This is especially useful in speech to text systems. The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiment(s) being illustrative and not restrictive.

WO 2005/122141 PCT/AU2005/000808 -39 In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have corresponding meanings.

Claims (12)

1. 2. The method as claimed in claim I comprising the further step of: determining a preliminary classification for said signal segment from said statistical data before receipt of said notification of said end boundary of said signal segment. 15
2. 3. The method as claimed in claim I or 2 wherein said signal segment is classified as matching one of a plurality of classification categories, with each classification category being defined by a predefined model, or as not matching any one of said classification categories. 20
3. 4. The method as claimed in claim 3 wherein said statistical data comprises the accumulated logarithms of the statistical likelihood of each of said predefined models with respect to said frame feature vectors. 25689532 657341AUamend -41 5. A method of classifying segments of a signal, said method comprising the steps of: (a) receiving a sequence of segmentation frame feature data, each of said frame feature data characterising a frame of data along said signal; (b) in response to receipt of each of said frame feature data of a current 5 segment, updating current statistical data, characterising said current segment, with the received frame feature data; (c) receiving a notification of an end boundary of said current segment; (d) in response to receipt of said notification, comparing said current statistical data with statistical data characterising a preceding segment; and 10 (e) merging said current and preceding segments, or classifying said preceding signal segment based on said statistical data characterising said preceding segment, based upon the difference between said current statistical data and said statistical data characterising said preceding segment. 15 6. The method as claimed in claim 5 wherein said merging step additionally merges said current statistical data and said statistical data characterising said preceding segment.
4. 7. The method as claimed in claim 5 or 6 wherein said preceding segment is classified as matching one of a plurality of classification categories, with each 20 classification category being defined by a predefined model, or as not matching any one of said classification categories.
5. 8. The method as claimed in any one of claims I to 7 wherein said frame feature data is discarded once said statistical data has been updated. 2568953_2 657341AUamend - 42 9. The method as claimed in any one of claims 5 to 8 wherein said statistical data used for said comparing step is updated from a function of the energy value of a component frame, the bandwidth of said component frame, and the frequency centroid of 5 said component frame.
6. 10. The method as claimed in claim 9 wherein said function is the product of said energy value with a weighted sum of said bandwidth and said frequency centroid. 10 11. A method of classifying an audio segment, said method comprising the steps of: (a) receiving a sequence of audio frame feature data, each of said frame feature data characterising an audio frame along said audio segment; (b) in response to receipt of each of said audio frame feature data, updating statistical data, characterising said audio segment, with the received frame feature data; 15 (c) discarding said received frame feature data; (d) determining a preliminary classification for said audio segment from said statistical data; and (e) upon receipt of a notification of an end boundary of said audio segment, classifying said audio segment based on said statistical data. 20
7. 12. A method of classifying segments of an audio signal, said method comprising the steps of: (a) receiving a sequence of segmentation frame feature data, each of said frame feature data characterising a frame of data along said audio signal; 2568953_2 657341AUamend -43 (b) in response to receipt of euch of said frame feature data of a current segment, updating current statistical data, characterising said current segment, with the received frame feature data; (c) discarding said received frame feature data; 5 (d) receiving a notification of an end boundary of said current segment; (e) in response to receipt of said notification, comparing said current statistical data with statistical data characterising a preceding segment; and (f) merging said current and preceding segments, or classifying said preceding signal segment based on said statistical data characterising said preceding segment, based 10 upon the difference between said current statistical data and said statistical data characterising said preceding segment.
8. 13. A method for processing an audio signal comprising the steps of: (a) providing a plurality of predetennined, pre-trained models; 15 (b) providing an audio signal for processing in accordance with said models; (c) segmenting said audio signal into homogeneous portions whose length is not limited by a predetermined constant; and (d) classifying at least one of said portions with reference to at least one of said models; 20 wherein said segmenting step is performed independently of said classifying step, and said step of classifying a homogeneous portion begins before said segmenting step has identified the end of said portion. 2568953_2 657341AUamend - 44 14. The method according to claim 13 wherein the classification of a homogeneous portion completes within a fixed time after the end of said portion has been determined.
9. 15. The method according to claims 13 or 14 wherein said classifying step further 5 reports at least one preliminary classification of a homogeneous portion prior to the end of said portion has been determined.
10. 16. The method according to claims 13, 14 or 15 wherein said classifying step classifies a homogeneous portion either as consistent with one of said models or as not 10 consistent with any of said models.
11. 17. A method of segmenting an audio signal into a series of homogeneous portions comprising the steps of: receiving input consisting of a sequence of frames, each frame consisting of a 15 sequence of signal samples; calculating feature data for each said frame, forming a sequence of calculated feature data each corresponding to one of said frames; in response to receipt of each said calculated feature data of a current segment, updating current statistical data with the received frame feature data, said current 20 statistical data characterising said current segment; determining a potential end boundary for the current segment; in response to determining a potential end boundary, comparing said current statistical data with statistical data characterising a preceding segment; and 2568953_2 657341AUamend -45 merging said current and preceding segments, or accepting said preceding segment as a completed segment, based upon the difference between said current statistical data and said statistical data characterising said preceding segment. 5 18. A method of segmenting and classifying an audio signal into a series of homogeneous portions comprising the steps of: receiving input consisting of a sequence of frames, each frame consisting of a sequence of signal samples; calculating feature data for each said frame, forming a sequence of calculated 10 feature data each corresponding to one of said frames; in response to receipt of each said calculated feature data of a current segment, updating current statistical data with the received frame feature data, said current statistical data characterising said current segment; determining a potential end boundary for the current segment; 15 in response to determining a potential end boundary, comparing said current statistical data with statistical data characterising a preceding segment; merging said current and said preceding segment, or accepting said preceding segment as a completed segment and classifying said completed segment, based on the difference between said current statistical data and said statistical data characterising said 20 preceding segment.
12. 19. Computer apparatus configured for performing the method according to any one of the preceding claims. 2568953_2 657341AUamend -46 20. A computer readable medium, having a program recorded thereon, where the program is configured to make a computer execute a classification procedure according to the method of any one of claims I to 18. 5 DATED this 19th Day of April, 2010 CANON KABUSHIKI KAISHA Patent Attorneys for the Applicant Spruson & Ferguson 10 2568953_2 657341AUamend