CN113170228B

CN113170228B - Audio processing for extracting disjoint segments of variable length from audiovisual content

Info

Publication number: CN113170228B
Application number: CN201980058718.7A
Authority: CN
Inventors: M·斯托扬契奇; W·帕卡德
Original assignee: Stez Ltd
Current assignee: Stez Ltd
Priority date: 2018-07-30
Filing date: 2019-07-18
Publication date: 2023-07-14
Anticipated expiration: 2039-07-18
Also published as: JP2021533405A; EP3831083A4; AU2019314223A1; JP2024133486A; CA3108129A1; WO2020028057A1; CN113170228A; EP3831083A1; CN117041659A; AU2019314223B2; US20200037022A1; JP7541972B2; AU2024203420A1

Abstract

Boundaries of highlight segments of audiovisual content depicting an event are identified. The audiovisual content may be a broadcast, such as a television broadcast of a sporting event. The highlight segments may be segments of the audiovisual content that are considered to be of particular interest. Audio data for the audiovisual content is stored and automatically analyzed to detect soft entry points identified as low spectral activity points and/or low volume points in the analyzed audio data. A time index within the audiovisual content corresponding to the soft entry point may be designated as the boundary, which may be a start or end of the highlight.

Description

Audio processing for extracting disjoint segments of variable length from audiovisual content

Cross Reference to Related Applications

The present application claims the benefit of U.S. provisional application No. 62/712,041 entitled "audio processing for extracting variable length disjoint segments from television signals (Audio Processing for Extraction of Variable Length Disjoint Segments from Television Signal)" (attorney docket No. tha 006-pro), filed on at month 7 and 30 of 2018, which is incorporated herein by reference in its entirety.

The present application claims priority from U.S. provisional application No. 62/746,454 entitled "audio processing for detecting the occurrence of loud sounds characterized by short bursts of energy (Audio Processing for Detecting Occurrences of Loud Sound Characterized by Short-Time Energy Bursts)" (attorney docket No. THU 016-PROV), filed on 10/16 in 2018, which is incorporated herein by reference in its entirety.

The present application claims priority to U.S. utility application Ser. No. 16/440,229 entitled "Audio processing for extracting variable length disjoint segments from audiovisual content (Audio Processing for Extraction of Variable Length Disjoint Segments from Audiovisual Content)" (attorney docket number: THU 006), filed on publication No. 6/13, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application No. 13/601,915 entitled "excitement level to generate live performance (Generating Excitement Levels for Live Performances)" filed on month 8/31 2012 and issued as U.S. patent No. 9,060,210 on month 16 2015, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application No. 13/601,927 entitled "generating alert for live performance (Generating Alerts for Live Performances)" filed on month 8, 31 in 2012 and issued as U.S. patent No. 8,842,007 on month 9, 23 in 2014, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application No. 13/601,933 entitled "generating leading advertisement for live performance (Generating Teasers for Live Performances)" filed on month 8 and 31 in 2012 and issued as U.S. patent No. 8,595,763 on month 11 and 26 in 2013, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application Ser. No. 14/510,481 (attorney docket number: THU 001), entitled "generating custom highlight sequences for depicted events" (Generating a Customized Highlight Sequence Depicting an Event), filed on 10/9 at 2014, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application Ser. No. 14/710,438 (attorney docket number: THU 002), entitled "generating custom highlight sequences depicting multiple events" (Generating a Customized Highlight Sequence Depicting Multiple Events) filed on day 5, month 12 2015, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application Ser. No. 14/877,691 (attorney docket number: THU 004), entitled "custom generation of highlight showing with narrative composition" (Customized Generation of Highlight Show with Narrative Component), filed on 7, 10, 2015, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application Ser. No. 15/264,928 (attorney docket number: THU 005), entitled "user interface (User Interface for Interaction with Customized Highlight Shows) for interacting with custom highlight showing," filed on day 2016, 9 and 14, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application Ser. No. 16/411,704 entitled "video processing for achieving sports highlight Generation (Video Processing for Enabling Sports Highlights Generation)", filed on 5.14.2019 (attorney docket number: THU 009), which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application Ser. No. 16/411,710 entitled "machine learning (Machine Learning for Recognizing and Interpreting Embedded Information Card Content) for identifying and interpreting embedded information card content" (attorney docket number: THU 010), filed on 5.14 in 2019, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application Ser. No. 16/411,713 entitled video processing (Video Processing for Embedded Information Card Localization and Content Extraction) for embedded information card localization and content extraction (attorney docket number: THU 012), filed on 5.14 in 2019, which is incorporated herein by reference in its entirety.

The present application relates to U.S. utility application Ser. No. 16/421,391 entitled "Audio processing (Audio Processing for Detecting Occurrences of Crowd Noise in Sporting Event Television Programming) for detecting the occurrence of crowd noise in a sports event television program" (attorney docket number: THU 007), filed on 5/23 in 2019, which is incorporated herein by reference in its entirety.

Technical Field

This document relates to techniques for identifying multimedia content and associated information on a television device or video server delivering the multimedia content and having embedded software applications utilize the multimedia content to provide content and services in synchronization with delivery of the multimedia content. Various embodiments relate to methods and systems for providing automated audio analysis to segment program content depicting a sporting event in order to create video highlights for viewing during and after the game.

Background

Enhanced television applications such as interactive advertising and enhanced program guides with pre-game, in-game and post-game interactive applications have long been contemplated. Existing cable systems originally designed for broadcast television are being invoked to support many new applications and services, including interactive television services and enhanced (interactive) program guides.

Some frameworks for implementing enhanced television applications have been standardized. Examples include OpenCable ^TM Enhanced television application messaging specifications, and Tru2way specifications, which refer to interactive digital cable services delivered over a cable television network, including features such as interactive program guides, interactive advertisements, games, and the like. In addition, the cable operator "OCAP" program also provides interactive services, such as e-commerce shopping,Online banking, electronic program guides, and digital video recordings. These efforts have enabled first generation video synchronization applications to synchronize with video content delivered by programmers/broadcasters and provide additional data and interactivity for television programs.

The recent development of video/audio content analysis technology and powerful mobile devices opens up a new range of possibilities for developing complex applications that run synchronously with live television program events. These new technologies and advances in audio signal processing and computer vision, as well as improvements in the computational power of modern processors, enable the generation of complex program content highlights in real-time, accompanied by metadata that is currently lacking in television and other media environments.

Disclosure of Invention

A system and method are presented that enable automatic real-time segmentation of sporting event television programming and/or other audiovisual content based on audio processing for detecting low spectral activity and/or low volume entry points in an audio stream, such as the end of syllables, the end of words or phrases, and/or the end of sentences (soft entry points). A list of detected soft entry points (with minimal spectral activity and/or low volume) may be used to extract segments of audiovisual content according to criteria for video highlight generation.

In at least one embodiment, a spectrogram is constructed for the audio signal, allowing time-frequency analysis using a sliding 2D region window. For each analysis window position, a spectrum qualifier may be generated that represents spectrum activity within the analysis window. A vector of spectrum qualifiers with associated time locations may be formed and may be further divided into a set of consecutive one-second intervals. The internal qualifiers may be classified for each one-second interval, and non-maximal suppression may be performed to form a first vector of qualifier/position pairs with maximized qualifiers for each one-second interval. In a subsequent step, the detected entry points at one second intervals may be processed to define a desired average time interval and/or to define a further selection of qualifier/position pairs.

After the initial anchor element is established in the first vector (first element) of the qualifier/place pair, the next element of the first vector of the qualifier/place pair may be selected at a minimum desired distance (e.g., two seconds). Next, the immediate left and right neighbors of the next element may be examined to generate a new anchor element with the largest qualifier for the local neighborhood. This process may continue until all elements of the first vector of qualifier/position pairs are exhausted, resulting in a new set of entry points (soft entries) with variable mutual distances of two to three seconds on average, and with maximized spectrum qualifiers for each local neighborhood.

The soft-entry vector may then be converted to a list of best entry points, which is then applied to the generation of the video highlight. The video segment boundaries of the highlight segments may be modified according to the best available soft entry in the neighborhood of their original boundaries, and the highlight segments may then be extracted for further processing by the video highlight segment generation application.

In at least one embodiment, the extracted video highlight segments may also be processed by deploying an optional decay function, wherein mirrored leading and trailing curvatures are applied to segment boundaries, allowing further smoothing of transitions between extracted disjoint video segments.

In a sense, the methods presented herein may be used for video segmentation in any application that requires smooth reassembly of segmented video with minimal disturbing audio transitions.

In at least one embodiment, a method for identifying boundaries of highlight segments of audiovisual content depicting an event may comprise: audio data depicting at least a portion of the event is stored, the audio data is automatically analyzed to detect soft entry points of the audio data, and a time index corresponding to the soft entry points is designated as a boundary within the audiovisual content, the boundary comprising a beginning or an end of a highlight. The audiovisual content may be a television broadcast of a sporting event, for example.

The audiovisual content may comprise an audiovisual stream. The method may further comprise: audio data is extracted from the audiovisual stream before storing the audio data depicting at least part of the event.

The audiovisual content may comprise stored audiovisual content and/or real-time content. The method may further comprise: audio data is extracted from stored audio-visual content and/or real-time content prior to storing the audio data depicting at least part of the event.

The event may be a sporting event. The highlight reel may depict portions of a sporting event that are believed to be of particular interest to one or more users.

The method may further comprise: one of the audiovisual content and the highlight is played on the output device during detection of the soft entry point.

The method may further comprise: the audio data is pre-processed by resampling the audio data to a desired sample rate before the soft entry point is detected.

The method may further comprise: the audio data is preprocessed by filtering the audio data to reduce or remove noise before the soft entry point is detected.

The method may further comprise: the audio data is processed to generate a spectrogram for at least a portion of the audio data prior to detecting the soft entry point.

Detecting the soft entry point may include applying a sliding two-dimensional time-frequency analysis window for the spectrogram.

Detecting the soft entry point may include calculating an average spectral amplitude indicator for each position of a sliding two-dimensional time-frequency analysis window of the spectrogram, and forming a vector of spectral amplitude indicator/position pairs for the spectrogram using the average spectral amplitude indicator.

Detecting the soft entry point may further include converting the average spectral amplitude indicator for each vector element to an integer qualifier Q, and generating an initial vector having a Q/position pair.

Detecting the soft entry point may further include traversing elements of the initial vector having Q/position pairs step by step and maximizing Q for each one second interval by non-maximally suppressing the Q qualifier in each one second interval and forming a first vector having maximized Q qualifier.

Detecting the soft entry point may further include traversing each of the entered time components of the first vector having the maximized Q qualifier step by step; for each time position, comparing the time component of the current position with the previous time component of the previous position to obtain a distance; for each element of the first vector for which the distance is greater than the threshold, finding a maximum Q in the immediate neighborhood of the element location; and populating the new soft entry vector with the Q/position pair having the largest Q in each neighborhood.

The method may further comprise: the highlight segments are identified with tentative boundaries before the time index is designated as the boundary. Designating the time index as a boundary may include replacing the tentative boundary with a boundary obtained from the list of available soft entries.

Further details and variations are described herein.

Drawings

The accompanying drawings illustrate several embodiments along with the description. Those skilled in the art will recognize that the particular embodiments illustrated in the figures are merely exemplary and are not intended to limit the scope.

Fig. 1A is a block diagram depicting a hardware architecture in accordance with a client/server embodiment in which event content is provided by a content provider connected via a network.

Fig. 1B is a block diagram depicting a hardware architecture in accordance with another client/server embodiment in which event content is stored at a client-based storage device.

Fig. 1C is a block diagram depicting a hardware architecture according to an independent embodiment.

FIG. 1D is a block diagram depicting an overview of a system architecture according to one embodiment.

Fig. 2 is a schematic block diagram depicting an example of a data structure that may be incorporated into the audio data, user data, and highlight data of fig. 1A, B and 1C, according to one embodiment.

Fig. 3A depicts an example of an audio waveform diagram showing the occurrence of soft entry points in an audio stream extracted from sports event television program content in the time domain, according to one embodiment.

Fig. 3B depicts an example of a spectrogram corresponding to the audio waveform diagram of fig. 3A in the time-frequency domain, according to one embodiment.

Fig. 4 is a flow chart depicting a method for audio preprocessing by resampling, filtering, and spectrogram construction, in accordance with one embodiment.

Fig. 5 is a flow chart depicting a method for analyzing audio data, such as an audio stream, in the time-frequency domain and generating an initial vector of qualifiers, in accordance with one embodiment.

Fig. 6 is a flow chart depicting a method for generating a vector with maximized qualifiers within each one-second interval, in accordance with one embodiment.

FIG. 7 is a flow diagram depicting a method for further selecting soft entry points with variable spacing and maximizing local neighborhood qualifiers, in accordance with one embodiment.

FIG. 8 is a flow chart depicting a method for optionally modifying detected entry points of sparse segments, in accordance with one embodiment.

FIG. 9 is a flow diagram depicting a method for assembling an adjusted highlight clip based on a list of available soft entry points, according to one embodiment.

Detailed Description

Definition of the definition

The following definitions are presented for illustrative purposes only and are not intended to limit the scope.

Event: for purposes of discussion herein, the term "event" refers to a game, session, contest, series, show, program, concert, etc., or portion thereof (e.g., an action, period, quarter, half, office, scene, chapter, etc.). An event may be a sporting event, an entertainment event, a specific performance of a single person or subset of persons within a larger group of participants in an event, and so forth. Examples of non-sporting events include television programming, news headlines, political events, natural disasters, movies, shows, radio programming, podcasts, audio books, online content, musical performances, and the like. Events may have any length. For illustrative purposes, techniques are often described herein in terms of sporting events; however, those skilled in the art will recognize that the techniques may also be used in other contexts, including highlight presentations for any audiovisual, visual, graphics-based, interactive, non-interactive, or text-based content. Thus, the use of the term "sporting event" and any other sports-specific term in the description is intended to illustrate one possible embodiment, but is not intended to limit the scope of the described technology to that one embodiment. Rather, these terms should be construed as extending to any suitable non-sports context when technically appropriate. For ease of description, the term "event" is also used to refer to a story or representation of an event, such as an audiovisual recording of an event, or any other content item that contains a story, description, or depiction of an event.

Highlight clip: an excerpt or portion of an event, or excerpt or portion of content associated with an event that is considered to be of particular interest to one or more users. The highlight reel may have any length. In general, the techniques described herein provide a mechanism for identifying and presenting a customized set of highlight clips (selectable based on the particular characteristics and/or preferences of a user) for any suitable event. "highlight" may also be used to refer to a story or representation of a highlight, such as an audiovisual recording of the highlight, or any other content item that contains a story, description, or depiction of the highlight. The highlight reel need not be limited to a depiction of the event itself, but may include other content associated with the event. For example, for a sporting event, a highlight can include audio/video in the game, as well as other content such as interviews, analyses, commentary, etc. before, during, and after the game. Such content may be recorded from a linear television (e.g., as part of an audiovisual stream depicting the event itself), or retrieved from any number of other sources. Different types of highlight clips may be provided, including for example occurrences (manifestations in a game), strings, controls and sequences, all of which are defined below. The highlight segments need not have a fixed duration, but may incorporate a start offset and/or an end offset, as described below.

Clip: a portion of an audio, visual or audiovisual representation of an event. The clip may correspond to or represent a highlight. In many contexts herein, the term "clip" may be used interchangeably with "clip". The clip may be part of an audio stream, a video stream or an audiovisual stream, or it may be part of stored audio, video or audiovisual content.

Content delineator: one or more video frames indicating the beginning or end of a highlight.

Occurrence of: something that happens during an event. Examples include: goal, scene, opportunity to attack, hit, rescue, shoot, basket, break, catch or attempt to catch, gain-away, fight, start or end of play, quarter, half, period, or play, pitch, penalty, injury, dramatic things in an entertainment event, song, solo, etc. The occurrences may also be exceptions such as power outages, fan non-compliance accidents, and the like. The detection of such occurrences may be used as a basis for determining whether to designate a particular portion of the audiovisual stream as a highlight. For ease of naming, what will also occur is referred to herein as "performance in a race", but such usage should not be considered as limiting in scope. The occurrences may be of any length, and the representations of the occurrences may also be of varying lengths. For example, as mentioned above, an extended representation of an occurrence may include footage depicting a period of time just before and just after the occurrence, while a short representation may include only the occurrence itself. Any intermediate representation may also be provided. In at least one embodiment, the selection of the duration of the representation for the occurrence may depend on user preferences, available time, determined excitement level of the occurrence, importance of the occurrence, and/or any other factors.

Offset amount: the adjustment amount of the highlight length. In at least one embodiment, a start offset and/or an end offset may be provided for adjusting the start time and/or end time, respectively, of the highlight reel. For example, if a highlight reel is depicted as a goal, the highlight reel may be lengthened (via an ending offset) for a few seconds to include celebration and/or fan reactions after the goal. The offset may be configured to change automatically or manually, for example, based on the amount of time available for the highlight reel, the importance and/or excitement level of the highlight reel, and/or any other suitable factor.

Character string: a series of occurrences that are related or correlated to one another to some extent. This may occur within one control right (defined below), or may span multiple control rights. This may occur within one sequence (defined below), or may span multiple sequences. What happens may be linked or related due to some sort of thematic or descriptive connection with each other or because one causes the other, or for any other reason. One example of a string is a set of pass that results in a goal or basket. It is not to be confused with a "text string" which has the meaning commonly assigned in the art of computer programming.

Control right: any time-separated portion of the event. The demarcation of the start/end times of the control right may depend on the event type. For certain sporting events where one team may be in attack and another team in defense (e.g., basketball or football), control may be defined as the time period during which one team has a ball. In sports such as hockey or football, where the ball mass or ball control is more fluid, control may be considered to extend to a period of time where one team has substantial control of the ball mass or ball, while ignoring momentary contact by another team (e.g., blocked shooting or saving). For baseball, control rights are defined as half a game. For football, control may include several sequences in which the same team has a ball. For other types of sporting events as well as non-sporting events, the term "control" may be somewhat misshapen, but is still used herein for illustrative purposes. Examples in a non-sports context may include chapters, scenes, actions, and the like. For example, in the context of a concert, control may be equivalent to playing a single song. Control may include any number of occurrences.

Sequence: a time-separated portion of the event that contains one continuous period of action. For example, in a sporting event, the sequence may begin when an action begins (e.g., a tee, a jump, etc.), and may end when a whistle blows to indicate that the action is aborted. In sports such as baseball or rugby, the sequence may be equivalent to a scene in the form of an occurrence. The sequence may contain any number of controls, or may be part of a control.

Highlight clip presentation: a set of highlight clips arranged to be presented to a user. The highlight presentation may be presented linearly (e.g., audiovisual stream) or in a manner that allows the user to select which highlight to view and the order of viewing (e.g., by clicking on a link or thumbnail). The presentation of the highlight reel presentation may be non-interactive or interactive, e.g., allowing the user to pause, rewind, skip, fast forward, convey a preference or dislike, etc. The highlight reel may be, for example, a concentrated game. The highlight reel display may include any number of contiguous or non-contiguous highlight reels from a single event or multiple events, and may even include highlight reels from different types of events (e.g., different sports, and/or a combination of highlight reels from sports and non-sports events).

User/viewer: the term "user" or "viewer" interchangeably refers to a person, group, or other entity that views, listens to, or otherwise experiences an event, one or more highlights of an event, or a presentation of highlights. The term "user" or "viewer" may also refer to a person, group, or other entity that may view, listen to, or otherwise experience an event, one or more highlights of an event, or a presentation of highlights at some future time. The term "viewer" may be used for descriptive purposes, but the event need not have a video component, so that the "viewer" may instead be a listener of the content or any other consumer.

Level of excitement: an event or highlight is expected to be a measure of the level of excitement or interest for a particular user or, in general, a user. The level of excitement may also be determined relative to a particular occurrence or player. Various techniques for measuring or assessing the level of excitement are discussed in the related applications cited above. As discussed, the level of excitement may depend on what happens within the event, as well as other factors such as the overall context or importance of the event (post-season, corner-flag implications, contention, etc.). In at least one embodiment, the excitement level may be associated with each occurrence, concatenation, control, or sequence within an event. For example, the level of excitement of a control right may be determined based on what happens within the control right. The level of excitement measured may vary from user to user (e.g., a team's fans versus a neutral fan), and may depend on the personal characteristics of each user.

Metadata: data related to and stored in association with other data. The primary data may be media such as sports programming or highlights.

Video data. The length of the video, which may be in digital form or analog form. The video data may be stored on a local storage device or may be received in real-time from a source such as a television broadcast antenna, a cable television network, or a computer server, in which case the video data may also be referred to as a "video stream". The video data may or may not contain an audio component; if it contains an audio component, it may be referred to as "audiovisual data" or "audiovisual stream".

Audio data. The length of the audio, which may be in digital form or in analog form. The audio data may be audio-visual data or audio components of an audio-visual stream and may be isolated by extracting the audio data from the audio-visual data. The audio data may be stored in a local storage device or may be received in real-time from a source such as a television broadcast antenna, a cable television network, or a computer server, in which case the audio data may also be referred to as an "audio stream".

Stream. An audio stream, a video stream, or an audiovisual stream.

Time index. An indicator of a time within audio data, video data, or audiovisual data at which an event occurred or is otherwise related to a specified segment, such as a highlight segment.

Spectrograms. Such as a visual representation of the frequency spectrum of a signal, such as an audio stream, over time. The spectrogram may be a two-dimensional time-frequency representation of the audio signal derived by applying a short-time fourier transform (Short Time Fourier Transform; STFT) to the audio signal.

Analysis window. Video data, audio data, audiovisual data, a spectrogram, a stream or a specified subset of the otherwise processed version of the stream or data at which an analysis step is to be focused. Audio data, video data, audiovisual data or spectrograms may be analyzed, for example, in fragments using a moving analysis window and/or a series of analysis windows covering different fragments of the data or spectrograms.

Boundary. Demarcating one audio, video and/or audiovisual clip from another. The boundary may be the beginning or end of a segment, such as a highlight of audiovisual content (e.g., a television broadcast). The boundaries may be tentative (i.e., preliminary and/or intended for subsequent replacement) or final. In some embodiments, a highlight reel may be identified first with a tentative boundary. Audio analysis may be performed to identify soft entry points, which are then used to (timely) locate the final boundaries of the highlight.

Soft entry point. Portions of an audio, video or audiovisual stream represent possible or preferred transition points between different scenes, content types, games, etc. The soft entry point may be a quieter portion of the audio signal, in the context of a sporting event, representing a comment in the game or a pause between games.

Overview

In at least one embodiment, the systems and methods described herein perform automatic real-time, variable length segmentation of an audiovisual program, such as a television program, based on audio processing for detecting low activity entry points ("soft entry points") (e.g., syllables, sentences, and/or ends of phrases). These entry points may be used as guides when extracting segments of an audiovisual program to facilitate highlight generation by improved transition from one highlight to the next, thereby avoiding interruption of conversations or other sounds, and avoiding abrupt transitions.

In at least one embodiment, the automated video highlight and associated metadata generation application may receive live broadcast audiovisual streams or digital audiovisual streams received via a computer server. The application may then process the audio data (e.g., an audio stream extracted from an audio-visual stream), for example, using digital signal processing techniques, to detect soft entry points.

In alternative embodiments, the techniques described herein may be applied to other types of source content. For example, there is no need to extract audio data from an audiovisual stream; more specifically, the audio data may be a radio broadcast or other audio depiction of a sporting event or other event. Alternatively, the techniques described herein may be applied to stored audio data that depicts events; such data may or may not be extracted from the stored audiovisual data.

The interactive television application enables timely and relevant presentation of television program content with highlights to a user viewing the television program on a primary television display or on an auxiliary display such as a tablet, laptop, or smart phone. In at least one embodiment, a set of clips representing highlight clips of television broadcast content are generated and/or stored in real-time along with a database containing time-based metadata describing events presented by the highlight clips in more detail. As described in more detail herein, the start time and/or end time of such clips may be determined based at least in part on analysis of the extracted audio data.

In various embodiments, the metadata accompanying the clip may be any information, such as text information, images, and/or any type of audiovisual data. One type of metadata associated with video content during and after a game highlights current events detected by processing audio data extracted from a sports event television program or other audiovisual content in real-time. In various embodiments, the systems and methods described herein enable automatic metadata generation and video highlight processing, wherein the start and/or end times of highlight segments may be detected and/or refined by analyzing digital audio data, such as an audio stream.

For example, highlight segments may be identified by analyzing such audio data to detect cheering crowd noise after certain exciting events, audio announcements, music, and so forth. Additionally or alternatively, highlight clips may be detected in any of the other ways presented in any of the above-listed parent applications. The identification of soft entry points may be used to identify or refine the start and/or end times of the highlight segments such that the highlight segments start and/or end at natural breaks in the audiovisual content, thereby avoiding unnecessary abrupt transitions. In at least one embodiment, real-time processing is performed on audio data, which may be an audio stream extracted from sports event television program content, in order to detect, select, and track such soft entry points.

In at least one embodiment, a method for automatically processing in real-time audio signals extracted from audiovisual content, such as sports event television program content, detects a soft entry point. The method may comprise: capturing, decoding and preprocessing an audio signal; generating a time-frequency audio spectrogram for combining the time-frequency analysis to detect a low-frequency active region; generating a spectrum indicator for the overlapping spectrogram region; and forming a vector of spectrum qualifiers having associated time locations. The method may further comprise: dividing the vector into successive one second intervals; classifying the internal spectrum qualifiers for each one-second interval; and performing non-maximal suppression to form a first vector of qualifier/position pairs. Furthermore, the method may include processing the first vector for each qualifier/position pair to specify a desired average time interval and to specify a further selection of qualifier/position pairs. Still further, the method may comprise: forming a list of best entry points; and applying the list of best entry points to the video highlight generation based on the best available soft entry in the neighborhood of the original boundary, for example by modifying the video segment boundary with the highlight.

In at least one embodiment, the system and method receives compressed audio data and reads, decodes and resamples the compressed audio data to a desired sample rate. Pre-filtering may be performed to reduce noise, remove rattles, and select a frequency band of interest; any of a number of interchangeable digital filtering stages may be used.

The overlapping spectrogram region may be analyzed by a 2D time-frequency window of the sub-second time range. In at least one embodiment, the analyzed time-frequency windows are slid along the spectrogram time coordinates and a normalized average amplitude of the windows is calculated at each overlapping window position.

The average amplitude may be a spectral indicator generated at each analysis window position. An initial vector of spectrum indicators with associated time positions may be formed and further divided into successive one second intervals. The internal qualifiers may be classified for each one-second interval, and non-maximum suppression may be performed to form a first vector of qualifier/position pairs. Detected entry points at one second intervals may be processed to define a desired average time interval and to define further choices of qualifier/position pairs. For illustrative purposes, the description herein refers to one second intervals; however, one of ordinary skill in the art will recognize that any suitable length of spacing may be used.

After the initial anchor element (first element) is established, the next element may be selected at a minimum desired distance (e.g., a length of two seconds). Other lengths may also be used. Elements in the immediate vicinity of the next element to the left and right may be examined to generate a new anchor element with a maximization qualifier for the local neighborhood. All first vectors of qualifier/position pairs may be processed in successive steps to produce a new set of soft entry points with variable mutual distances averaged, for example, two to three seconds, and with maximized spectrum qualifiers for each local neighborhood.

The soft-entered vector may be converted to a list of best entry points, which is then applied to the generation of video highlight. The video segment boundaries of the highlight can be modified based on the best available soft entry in the neighborhood of its original boundary. Video segments with highlight segments having modified boundaries may be extracted for further processing by a video highlight generation application. The extracted video highlight segments may be further processed by deploying an optional decay function, wherein mirrored front and tail curvatures are applied to the segment boundaries.

System architecture

According to various embodiments, the system may be implemented on any electronic device or group of electronic devices equipped to receive, store, and present information. Such an electronic device may be, for example, a desktop computer, a laptop computer, a television, a smart phone, a tablet computer, a music player, an audio device, an all-in-one machine, a set-top box (STB), a gaming system, a wearable device, a consumer electronic device, and so forth.

Although the system is described herein in connection with implementations in a particular type of computing device, those skilled in the art will recognize that the techniques described herein may be implemented in other contexts and indeed in any suitable device capable of receiving and/or processing user input and presenting output to a user. Accordingly, the following description is intended to illustrate various embodiments by way of example, and not to limit the scope.

Referring now to fig. 1A, a block diagram depicting a hardware architecture of a system 100 for automatically analyzing audio data to detect soft entry points to specify boundaries of highlight segments is presented in accordance with a client/server embodiment. Event content, such as an audiovisual stream containing audio content, may be provided via a network-connected content provider 124. Examples of such client/server embodiments are network-based implementations in which each of the one or more client devices 106 runs a browser or application that provides a user interface for interacting with content from the

various servers

102, 114, 116, including the data provider server 122 and/or the content provider server 124, via the communication network 104. In response to a request from client device 106, the transmission of content and/or data may be performed using any known protocol and language, such as hypertext markup language (Hypertext Markup Language; HTML), java, objective C, python, javaScript, and the like.

The client device 106 may be any electronic device, such as a desktop computer, a laptop computer, a television, a smart phone, a tablet computer, a music player, an audio device, an all-in-one, a set-top box, a gaming system, a wearable device, a consumer electronic device, and so forth. In at least one embodiment, the client device 106 has several hardware components that are well known to those of skill in the art. Input device 151 may be any component that receives input from user 150, including, for example, a keyboard, mouse, stylus, touch sensitive screen (touch screen), touch pad, gesture receptor, trackball, accelerometer, five-way switch, microphone, and the like. The input may be provided via any suitable mode, including, for example, one or more of: pointing, tapping, typing, dragging, pointing, tilting, panning and/or voice. The display screen 152 may be any component that graphically displays information, video, content, etc. (including depictions of events, highlights, etc.). Such output may also include, for example, audiovisual content, data visualizations, navigation elements, graphical elements, queries requesting information and/or parameters for selecting content, metadata, and so forth. In at least one embodiment where only some of the desired outputs are presented at a time, dynamic controls, such as scrolling mechanisms, may be available via the input device 151 to select which information is currently displayed, and/or to alter the manner in which the information is displayed.

The processor 157 may be a conventional microprocessor for performing operations on data under the direction of software according to well known techniques. Memory 156 may be random access memory having a structure and architecture known in the art for use by processor 157 in running software to perform the operations described herein. Client device 106 may also include a local storage device (not shown), which may be a hard disk drive, flash drive, optical or magnetic storage device, network-based (cloud-based) storage device, or the like.

Any suitable type of communication network 104, such as the internet, a television network, a cable television network, a cellular network, etc., may be used as a mechanism for transmitting data between the client device 106 and the plurality of

servers

102, 114, 116 and/or content provider 124 and/or data provider 122 according to any suitable protocols and techniques. Other examples include, in addition to the internet, cellular telephone networks, EDGE, 3G, 4G, long Term Evolution (LTE), session Initiation Protocol (SIP), short message end-to-end protocol (SMPP), SS7, wi-Fi, bluetooth, zigBee, hypertext transfer protocol (HTTP), secure hypertext transfer protocol (SHTTP), transmission control protocol/internet protocol (TCP/IP), and the like, and/or any combination thereof. In at least one embodiment, the client device 106 transmits requests for data and/or content via the communication network 104 and receives responses from the

servers

102, 114, 116 containing the requested data and/or content.

In at least one embodiment, the system of FIG. 1A operates in conjunction with a sporting event; however, the teachings herein also apply to non-sporting events, and it should be appreciated that the techniques described herein are not limited to application to sporting events. For example, the techniques described herein may be used to operate in connection with television programming, movies, news events, gaming programming, political behavior, commercials, drama, and/or other episode content, or for more than one such event.

In at least one embodiment, the system 100 identifies highlight segments of audiovisual content depicting an event (e.g., a broadcast of a sporting event) by analyzing audio content representing the event. This analysis can be performed in real time. In at least one embodiment, the system 100 includes one or more web servers 102 coupled to one or more client devices 106 via a communication network 104. The communication network 104 may be a public network, a private network, or a combination of public and private networks, such as the internet. The communication network 104 may be a LAN, WAN, wired, wireless, and/or combinations thereof. In at least one embodiment, the client device 106 is capable of connecting to the communication network 104 via a wired or wireless connection. In at least one embodiment, the client device may also include a recording device capable of receiving and recording events, such as a DVR, PVR, or other media recording device. This recording device may be part of the client device 106 or may be external; in other embodiments, the recording device may be omitted. Although fig. 1A shows one client device 106, the system 100 may be implemented with any number of single type or multiple types of client devices 106.

The network server 102 may include one or more physical computing devices and/or software that may receive requests from client devices 106 and respond to those requests with data, as well as send out unsolicited alerts and other messages. The web server 102 may employ various policies for fault tolerance and scalability, such as load balancing, caching, and clustering. In at least one embodiment, the web server 102 may include caching techniques as known in the art for storing client requests and information related to events.

The web server 102 may maintain or otherwise designate one or more application servers 114 in response to requests received from the client devices 106. In at least one embodiment, the application server 114 provides access to business logic for use by client applications in the client device 106. Application server 114 may be co-located, co-owned, or co-managed with web server 102. Application server 114 may also be remote from web server 102. In at least one embodiment, the application server 114 interacts with one or more analysis servers 116 and one or more data servers 118 to perform one or more operations of the disclosed technology.

The one or more storage devices 153 may act as a "data store" by storing data related to the operation of the system 100. This data may include, for example, but is not limited to, audio data 154 representing one or more audio signals. The audio data 154 may be extracted, for example, from an audiovisual stream or stored audiovisual content representing a sporting event and/or other event.

The audio data 154 may include any information related to audio embedded in the audiovisual stream, such as an audio stream accompanying a video image, a processed version of the audiovisual stream, and metrics and/or vectors related to the audio data 154, such as a time index, duration, amplitude, and/or other parameters of the event. The user data 155 may contain any information describing one or more users 150 including, for example, demographics, purchasing behavior, audiovisual stream viewing behavior, interests, preferences, and the like. The highlight data 164 may include highlight segments, highlight segment identifiers, time indicators, categories, excitement levels, and other data related to highlight segments. The audio data 154, the user data 155, and the highlight data 164 will be described in detail later.

Notably, many of the components of the system 100 may be or may include computing devices. As shown and described above, such computing devices may each have an architecture similar to client device 106. Accordingly, any of the communication network 104, the network server 102, the application server 114, the analytics server 116, the data provider 122, the content provider 124, the data server 118, and the storage 153 may contain one or more computing devices, each of which may optionally have an input device 151, a display screen 152, a memory 156, and/or a processor 157, as described above in connection with the client device 106.

In exemplary operation of the system 100, one or more users 150 of the client devices 106 view content from the content provider 124 in the form of an audiovisual stream. The audiovisual stream may present events such as sporting events. The audiovisual stream may be a digital audiovisual stream that can be readily processed using known computer vision techniques.

As the audiovisual stream is displayed, one or more components of the system 100, such as the client device 106, the web server 102, the application server 114, and/or the analysis server 116, may analyze the audiovisual stream, identify highlight segments within the audiovisual stream, and/or extract metadata from the audiovisual stream, such as from the audio component of the stream. This analysis may be performed in response to receiving a request to identify highlight segments and/or metadata of the audiovisual stream. Alternatively, in another embodiment, highlight segments and/or metadata may be identified without the user 150 making a particular request. In yet another embodiment, the analysis of the audiovisual stream may be performed without displaying the audiovisual stream.

In at least one embodiment, the user 150 may specify certain parameters (e.g., what events/games/teams to include, how much time the user 150 has available to view highlight clips, what metadata is needed, and/or any other parameters) for analyzing the audio data 154 via the input device 151 at the client device 106. User preferences may also be extracted from storage, such as from user data 155 stored in one or more storage devices 153, in order to customize analysis of audio data 154 without necessarily requiring user 150 to specify preferences. In at least one embodiment, user preferences may be determined based on observed behavior and actions of user 150, for example, by observing website access patterns, television viewing patterns, music listening patterns, online purchases, previous highlight identification parameters, highlight and/or metadata actually viewed by user 150, and so forth.

Additionally or alternatively, the user preferences may be retrieved from previously stored preferences explicitly provided by the user 150. Such user preferences may indicate which teams, sports, athletes, and/or event types are of interest to the user 150, and/or they may indicate what types of metadata or other information related to highlight pieces are of interest to the user 150. Thus, such preferences may be used to guide analysis of the audiovisual stream to identify highlight segments and/or to extract metadata for highlight segments.

The analysis server 116, which may include one or more computing devices as described above, may analyze live and/or recorded feeds of sports game onsite commentary statistics related to one or more events from the data provider 122. Examples of data providers 122 may include, but are not limited to, providers of real-time athletic movement information such as STATSTM, perform (Opta Sports available from London, UK) and Sportradar, st. Gallon, switzerland. In at least one embodiment, the analytics server 116 generates different sets of excitement levels for events; such excitement levels may then be stored in connection with highlight clips identified or received by the system 100 in accordance with the techniques described herein.

The application server 114 may analyze the audiovisual stream to identify highlight segments and/or extract metadata. Additionally or alternatively, such analysis may be performed by the client device 106. The identified highlight segments and/or extracted metadata may be specific to the user 150; in this case, it may be advantageous to identify highlight reel(s) in client device 106 that are relevant to a particular user 150. As described above, the client device 106 may receive, retain, and/or retrieve applicable user preferences for highlight identification and/or metadata extraction. Additionally or alternatively, highlight generation and/or metadata extraction may be performed on a global scale (i.e., using objective criteria applicable to the general user population, regardless of the preferences of the particular user 150). In this case, it may be advantageous to identify highlight segments and/or extract metadata in the application server 114.

Content that facilitates highlight identification, audio analysis, and/or metadata extraction may be from any suitable source, including from a content provider 124, which may include websites such as YouTube, MLB.com, and the like; sports data provider; a television station; client or server based DVR, etc. Alternatively, the content may be from a local source such as a DVR or other recording device associated with (or built into) the client device 106. In at least one embodiment, the application server 114 generates a customized highlight show with highlight and metadata that the user 150 may download, stream, or order content or use in some other way.

As mentioned above, it may be advantageous to perform user-specific highlight identification, audio analysis, and/or metadata extraction at a particular client device 106 associated with a particular user 150. This embodiment may avoid the need to unnecessarily transmit video content or other high bandwidth content via the communication network 104, particularly if such content is already available at the client device 106.

For example, referring now to FIG. 1B, an example of a system 160 is illustrated in which at least some of the audio data 154 and highlight data 164 are stored at a client-based storage 158, which may be any form of local storage available to the client device 106. An example may be a DVR that records events, such as video content for a complete sporting event. Alternatively, client-based storage 158 may be any magnetic, optical, or electronic storage for data in digital form; examples include flash memory, magnetic hard drives, CD-ROMs, DVD-ROMs, or other devices integrated with client device 106 or communicatively coupled with client device 106. Based on the information provided by the application server 114, the client device 106 may extract highlight clips and/or metadata from video content (e.g., including audio data 154) stored at the client-based storage 158 and store the highlight clips and/or metadata as highlight clip data 164 without having to retrieve other content from the content provider 124 or other remote source. This arrangement may save bandwidth and may effectively utilize existing hardware that may already be available to the client device 106.

Returning to FIG. 1A, in at least one embodiment, the application server 114 may identify different highlight clips and/or extract different metadata for different users 150 according to individual user preferences and/or other parameters. The identified highlight segments and/or extracted metadata may be presented to the user 150 via any suitable output device, such as a display screen 152 at the client device 106. If desired, multiple highlight segments may be identified and compiled with associated metadata into a highlight segment presentation. Such highlight presentations may be accessed via a menu and/or spliced into a "highlight collection" or set of highlight segments that are played for the user 150 according to a predetermined order. In at least one embodiment, the user 150 can control highlight playback and/or delivery of associated metadata via the input device 151, for example, to:

select a particular highlight and/or metadata to be displayed;

pause, fast reverse, fast forward;

jump forward to the next highlight;

return to the beginning of the previous highlight within the highlight presentation; and/or

Perform other actions.

Additional details regarding such functionality are provided in the related U.S. patent applications referenced above.

In at least one embodiment, one or more data servers 118 are provided. The data server 118 may respond to requests for data from any of the

servers

102, 114, 116, for example, to obtain or provide audio data 154, user data 155, and/or highlight data 164. In at least one embodiment, such information may be stored at any suitable storage 153 that may be accessed by the data server 118, and may be from any suitable source, such as from the client device 106 itself, the content provider 124, the data provider 122, and so forth.

Referring now to FIG. 1C, a system 180 is shown in accordance with an alternative embodiment, wherein the system 180 is implemented in a stand-alone environment. As with the embodiment shown in fig. 1B, at least some of the audio data 154, user data 155, and highlight data 164 may be stored at a client-based storage 158, such as a DVR or the like. Alternatively, the client-based storage 158 may be a flash memory or hard drive, or other device integrated with the client device 106 or other device communicatively coupled with the client device 106.

The user data 155 may include preferences and interests of the user 150. Based on such user data 155, the system 180 may extract highlight clips and/or metadata for presentation to the user 150 in the manner described herein. Additionally or alternatively, highlight clips and/or metadata may be extracted according to objective criteria that are not based on information specific to the user 150.

Referring now to FIG. 1D, an overview of a system 190 with an architecture according to an alternative embodiment is shown. In fig. 1D, the system 190 comprises: broadcast services, such as content provider 124; a content receiver in the form of a client device 106 (e.g., a television with STB); a video server, such as analysis server 116, capable of capturing and streaming audiovisual content; and/or other client devices 106, such as mobile devices and handheld computers, capable of receiving and processing audiovisual content, such as television program content, all connected via a network, such as communication network 104. A client-based storage 158, such as a DVR, may be connected to any of the client devices 106 and/or other components, and may store audiovisual streams, highlight segments, highlight segment identifiers, and/or metadata to facilitate the identification and presentation of highlight segments and/or extracted metadata via any of the client devices 106.

The particular hardware architecture depicted in fig. 1A, 1B, 1C, and 1D is merely exemplary. Those skilled in the art will recognize that other architectures may be used to implement the techniques described herein. Many of the components depicted therein are optional and may be omitted, combined with, and/or replaced with other components.

In at least one embodiment, the system may be implemented as software written in any suitable computer programming language, whether in a stand-alone or client/server architecture. Alternatively, it may be implemented in hardware and/or embedded in hardware.

Data structure

FIG. 2 is a schematic block diagram depicting an example of a data structure that may incorporate audio data 154, user data 155, and highlight data 164, according to one embodiment.

As illustrated, the audio data 154 may contain a recording of each of the plurality of audio streams 200. The audio stream 200 is depicted for illustrative purposes, but the techniques described herein may be applied to any type of audio data 154 or content, whether streaming or storage. In addition to the audio stream 200, the recording of the audio data 154 may also contain other data generated from or contributing to the analysis of the audio stream 200. For example, for each audio stream 200, the audio data 154 may include a spectrogram 202, one or more analysis windows 204, a vector 206, and a time index 208.

Each audio stream 200 may reside in the time domain. Each spectrogram 202 may be calculated in the time-frequency domain for a corresponding audio stream 200. The spectrogram 202 may be analyzed to more easily locate soft entry points.

Analysis window 204 may be a designation of a predetermined time and/or frequency interval of spectrogram 202. Computationally, a single moving (i.e. "sliding") analysis window 204 may be used to analyze the spectrogram 202, or a series of displacement (optionally overlapping) analysis windows 204 may be used.

Vector 206 may be a data set containing temporal results and/or final results from analysis of audio stream 200 and/or corresponding spectrograms 202.

The time index 208 may indicate when a critical event occurred within the audio stream 200 (and/or an audiovisual stream from which the audio stream 200 was extracted). For example, time index 208 may be the time at which a soft entry point within the audiovisual content begins, centers, or ends. Thus, the time index 208 may indicate the beginning or end of a portion of particular interest of the audiovisual stream, such as important or impressive game performance in the context of a sporting event.

As further shown, the user data 155 may include records related to the user 150, each of which may include demographics 212, preferences 214, viewing history 216, and purchase history 218 for a particular user 150.

Demographic data 212 may include any type of demographic data including, but not limited to, age, gender, location, nationality, religious beliefs, education levels, and the like.

Preferences 214 may include selections made by user 150 in view of his or her preferences. The preferences 214 may relate directly to the collection and/or viewing of highlight and metadata or may be more general in nature. In either case, the preferences 214 may be used to facilitate the identification and/or presentation of highlight and metadata to the user 150.

Viewing history 216 may list television programs, audiovisual streams, highlights, web pages, search queries, sporting events, and/or other content retrieved and/or viewed by user 150.

Purchase history 218 may list products or services purchased or requested by user 150.

As further shown, the highlight segment data 164 may include a record of j highlight segments 220, each of which may include an audiovisual stream 222 and/or metadata 224 of a particular highlight segment 220.

The audiovisual stream 222 may include audio and/or video depicting the highlight 220, which may be obtained from one or more audiovisual streams of one or more events (e.g., by cutting the audiovisual stream to include only the audiovisual stream 222 related to the highlight 220). Within metadata 224, identifier 223 may include a time index (e.g., time index 208 of audio data 154) and/or other indicia within the audiovisual stream of the event from which highlight 220 was obtained that indicates where the highlight resides.

In some embodiments, the recording of each of the highlight clips 220 may contain only one of the audiovisual stream 222 and the identifier 223. Highlight playback may be performed by playing the audiovisual stream 222 for the user 150 or by playing only the portion of the audiovisual stream having the highlight for the event from which the highlight 220 was obtained using the identifier 223. The storage of the identifier 223 is optional; in some embodiments, the identifier 223 may be used only to extract the audiovisual stream 222 of the highlight reel 220, which may then be stored in place of the identifier 223. In either case, the time index 208 of the highlight reel 220 may be extracted from the audio data 154 and stored at least temporarily as metadata 224 that is appended to the highlight reel 220 or to the audiovisual stream from which the audio data 154 and highlight reel 220 were obtained. In some embodiments, time index 208 may be stored as boundary 232 of identifier 223.

In addition to or in lieu of the identifier 223, the metadata 224 may contain information about the highlight reel 220, such as the date of the event, the season, and the group or individual involved in the event or the audiovisual stream from which the highlight reel 220 was obtained, such as team, athlete, coach, anchor, broadcaster, fan, and so forth. Among other information, the metadata 224 for each highlight segment 220 may include a stage 226, a clock 227, a score 228, a frame number 229, and/or an excitement level 230.

Stage 226 may be a stage of an event related to highlight reel 220. More specifically, stage 226 may be a stage of a sporting event in which there is a beginning, middle, and/or ending of highlight reel 220. For example, stage 226 may be a "third office," "second office," "bottom half field," or the like.

The clock 227 may be a game clock associated with the highlight reel 220. More specifically, the clock 227 may be the state of the game clock at the beginning, middle, and/or end of the highlight reel 220. For example, the clock 227 of highlight reel 220 may be "15:47" that begins, ends, or spans a period of a sporting event during which fifteen minutes, forty-seven seconds, are displayed on the game clock.

Score 228 may be a game score associated with highlight reel 220. More specifically, the score 228 may be a score at the beginning, end, and/or middle of the highlight reel 220. For example, the score 228 may be "45-38", "7-0", "30-love", and the like.

The frame number 229 may be the number of video frames in the audiovisual stream from which the highlight 220 was obtained or may be the audiovisual stream 222 associated with the highlight 220, which is associated with the beginning, middle and/or end of the highlight 220.

The excitement level 230 may be a measure of the level of excitement or interest of a predicted event or highlight in a particular user 150 or general user. In at least one embodiment, the excitement level 230 may be calculated as indicated in the related applications referenced above. Additionally or alternatively, the excitement level 230 may be determined at least in part by analysis of the audio data 154, which may be components extracted from the audiovisual stream 222 and/or the audio stream 200. For example, audio data 154 containing higher levels of crowd noise, announcements, and/or fast-paced music may indicate a high excitement level 230 for the associated highlight reel 220. The excitement level 230 of the highlight reel 220 is not necessarily static, but instead varies during the highlight reel 220. Thus, the system 100 may be able to further refine the highlight reel 220 to show the user only the portion above the threshold excitement level 230.

The data structure set forth in fig. 2 is merely exemplary. Those skilled in the art will recognize that some of the data of fig. 2 may be omitted or replaced with other data in the execution of highlight identification and/or metadata extraction. Additionally or alternatively, data not specifically shown in fig. 2 or described in the present application may be used in the execution of highlight identification and/or metadata extraction.

Audio data analysis

In at least one embodiment, the system performs several analysis phases of the audio data 154, e.g., an audio stream, in the time-frequency domain in order to detect soft entry points, e.g., pauses or low points in speech, music or other sounds, during a delineated sporting event or another event. The depiction may be a television broadcast, an audiovisual stream, an audio stream, a stored file, and so forth.

First, the compressed audio data 154 is read, decoded, and resampled to a desired sample rate. The resulting PCM stream is then pre-filtered using any of a number of interchangeable digital filter stages to reduce noise, remove rattle, and/or select a desired frequency band. A spectrogram is then constructed for the audio data 154. Spectral amplitude valleys are identified at each location of the sliding two-dimensional time-frequency region window. Further steps may be taken to more fully and/or more reliably identify soft entry points 320.

The time index 208 corresponding to the soft entry point may be used as a boundary 232 (e.g., start or end) of the highlight reel 220. In some embodiments, these time indices 208 may be used to identify the actual starting and/or ending points of the highlight reel 220 that have been identified (e.g., with trial boundaries 232 that may be trial starting and ending points that are subsequently adjusted based on the identification of the soft entry point). The highlight reel 220 may be extracted and/or identified within the video stream for subsequent viewing by the user.

Fig. 3A depicts an example of an audio waveform diagram 300 in an audio stream 310 extracted from sports event television program content in the time domain, according to one embodiment. The highlight region shows an exemplary soft entry point 320, such as a less intense portion of the audio stream 310. The amplitude of the captured audio may be relatively low in the soft entry point 320, representing a relatively quiet portion of the audio stream 310.

Fig. 3B depicts an example of a spectrogram 350 corresponding to the audio waveform diagram 300 of fig. 3A in the time-frequency domain, according to one embodiment. In at least one embodiment, detection and tagging of the occurrence of an event of interest is performed in the time-frequency domain and the boundaries 232 of the event (not shown in fig. 3A and 3B) are presented to the video highlight and metadata generation application in real-time. These boundaries 232 may be used to extract one or more highlight segments 220 from the video stream or to determine the beginning and/or end of each highlight segment 220 in the video stream with greater accuracy so that the highlight segments 220 may be played without playing other content representing portions of the video stream that do not belong to the highlight segments. The boundary 232 may be used to reduce abrupt changes in the transition from one highlight segment 220 to another by helping to determine the appropriate transition point in the content (e.g., at the end of a sentence or during an audio pause). In some embodiments, the boundary 232 may be incorporated into the metadata 224, such as in an identifier 223 that identifies the beginning and/or end of the highlight reel 220, as set forth in the description of fig. 2.

Audio data analysis and metadata extraction

Fig. 4 is a flow chart depicting a method 400 for audio preprocessing by resampling, filtering, and spectrogram construction, in accordance with one embodiment. In at least one embodiment, the method 400 may be performed according to one embodiment by an application (e.g., running on one of the client device 106 and/or the analysis server 116) that receives the audiovisual stream and performs immediate processing of the audio data 154 for identifying the soft entry point 320, e.g., corresponding to the boundary 232 of the highlight reel 220. According to the method 400, audio data 154, such as the audio stream 310, may be processed to detect soft entry points 320 in the audio data 154 by detecting pauses, breaks, or other natural breaks between segments of audio, video, and/or audiovisual program content.

In at least one embodiment, the method 400 (and/or other methods described herein) is performed on audio data 154 that has been extracted from an audiovisual stream or other audiovisual content. Alternatively, the techniques described herein may be applied to other types of source content. For example, audio data 154 need not be extracted from the audiovisual stream; rather, it may be a radio broadcast or other audio depiction of a sporting event or other event.

In at least one embodiment, the method 400 (and/or other methods described herein) may be performed by a system, such as the system 100 of fig. 1A; however, alternative systems (including but not limited to system 160 of FIG. 1B, system 180 of FIG. 1C, and system 190 of FIG. 1D) may be used in place of system 100 of FIG. 1A. Furthermore, the following description assumes that less intense audio events will be identified; however, it should be understood that different types of audible events may be identified and used to extract metadata and/or identify boundaries 232 of highlight segments 220 according to methods similar to those explicitly described herein. In some embodiments, specific audio and/or visual cues (such as tones, black screens, etc.) may be identified as soft entry points.

The method 400 of fig. 4 may begin at step 410 where audio data 154, such as the audio stream 200, is read; the audio data 154 may optionally be decoded if it is in a compressed format. In step 420, the audio data 154 may be resampled to the desired sample rate. In step 430, the audio data 154 may be filtered using any of a plurality of interchangeable digital filter stages. Next, in step 440, a spectrogram 202 may optionally be generated for the filtered audio data 154, for example by computing a Short Time Fourier Transform (STFT) over a one second block of the filtered audio data 154. The time-frequency coefficients of the spectrogram 202 may be saved in a two-dimensional array for further processing.

In some embodiments, when only relatively quiet areas in the audio stream are of interest, step 440 may be omitted, regardless of spectral content, and further analysis may be simplified via the performance of only the time domain audio data 154. However, in this case, undesired soft entry detection may occur due to an inherently unreliable indicator based only on the audio volume threshold, regardless of spectral content related to a particular sound of interest (e.g., the commentator's voice and/or background audience noise), which may be less loud in the time domain, but may have rich spectral content in the time-frequency domain. In some embodiments, analysis of the audio stream may also be performed in both the time and time-frequency domains, with the detected soft entry points then being incorporated into the final result. In the description of fig. 5-10 below, it is assumed that step 440 has been performed and that the audio analysis step is performed on the spectrogram 202 corresponding to the audio data 154 (e.g., after decoding, resampling and/or filtering the audio data 154, as described above). Final vectors of soft entry points in the audio stream may be formed with emphasis on, but not limited to, detecting low volume and low spectral content of audio stream fragments related to reviewer speech gaps (e.g., words, phrases, and end of sentences).

Fig. 5 is a flow chart depicting a method 500 for analyzing audio data 154, such as audio stream 200, in the time-frequency domain, such as by analyzing spectrogram 202 to generate an initial vector of selected qualifier/position pairs, in accordance with one embodiment. First, in step 510, a two-dimensional rectangular time-frequency analysis window 204 of size (F T) is selected, where T is a multi-second value (typically 100 milliseconds) and F is the frequency range to be considered (typically 100Hz to 3 kHz). Next, in step 520, a window overlap region N is selected between adjacent analysis windows 204, and a window sliding step s= (T-N) (typically-20 ms) is calculated.

The method proceeds to step 530 in which the analysis window 204 is slid along the spectral time axis on the time axis of the spectrogram 202 in successive steps S. In step 540, at each position of the analysis window 204, an average spectral amplitude indicator (spectral magnitude indicator; SMI) is calculated. In step 550, a maximum SMI value M for the spectrogram may also be determined. In step 560, an SMI/location pair vector may be generated with an SMI below a threshold.

In step 570, each SMI may be converted to an integer qualifier Q according to the equation q= (M-SMI) MFACT, where MFACT is a factor used to extract a desired number of integers. In step 580, an initial vector of Q/position pairs may be generated as a superset of potential soft entry points 320. The method 500 may then be performed at each one second interval to maximize the qualifier.

Fig. 6 is a flow chart depicting a method 600 for generating a vector with maximized qualifiers at each one-second interval, in accordance with one embodiment. This may include step 610 in which the initial vector with Q/position elements is partitioned on one second boundaries. Within each one second consecutive interval, classification by qualifier Q may be performed. In step 620, only the Q/position pair with the largest Q per one second interval may be reserved. In step 630, a first soft entry vector may be formed having a maximized Q value.

Fig. 7 is a flow chart depicting a method 700 for further selecting soft entry points 320 having variable spacing and maximizing local neighborhood qualifiers, in accordance with one embodiment. Such processing may result in generating extended variable size intervals (mutual distances) for soft entry (typically 2 to 3 seconds on average) and maximizing at the same time the local neighborhood per interval versus spectrum qualifier.

As shown, method 700 may begin 710 with the anchor point (previous location) set to zero. Then, in step 720, method 700 may step through the time components of the first soft entry vector to detect the next time position and load it to the current position. At each step, a query 730 may be performed in which the temporal distance from the current location to the previous location is checked against a threshold. If this distance is greater than the threshold (e.g., two seconds), then the current position may be employed for further processing in step 740. If this distance is not greater than the threshold, step 720 may be repeated for the new time position.

Step 740 may include testing the immediate neighborhood of the reserved current location and identifying the element with the greatest Q. In step 750, this element may be loaded to a previous location, which now becomes a new anchor point for further testing. This locally maximized Q/position pair may also be used to populate the final soft entry vector in step 760.

Method 700 may be performed in successive steps until all elements of the first soft entry vector are exhausted. Specifically, query 770 may determine whether the end of the soft entry vector has been reached. If so, a final soft entry vector may be provided in step 780. If not, method 700 may return to step 720 for further iterations.

Fig. 8 is a flowchart depicting a method 800 for optionally modifying detected entry points of sparse segments, in accordance with one embodiment. For cases where there is less stringent requirements on the boundaries of the event with highlight segments, the method 800 may be used to further maximize the Q qualifier. Method 800 may begin at step 810, where method 800 steps through soft entry vector elements one at a time. In query 820, method 800 may test the Q value for a threshold. In step 840, Q/position pairs below the threshold may be removed. In step 830, Q/position pairs above the threshold may be retained.

The method 800 may be performed in successive steps until all elements of the final soft entry vector are exhausted. Specifically, query 850 may determine whether the end of the final soft entry vector has been reached. If so, the method 800 may proceed to the formation of a list of soft entry points 320 and highlight processing. If not, the method 800 may return to step 810 for further iterations.

Fig. 9 is a flowchart describing a method 900 for assembling an adjusted highlight clip 220 based on a list of available soft entry points according to one embodiment. First, in step 910, a tentative boundary 232 of the highlight reel 220 may be identified. Next, in step 920, a search of the list of soft entry points 320 may be performed and a best approximation to the tentative boundary 232 or boundaries of the highlight reel 220 may be generated. Next, in step 930, the tentative boundary 232 may be adjusted according to the best approximation obtained from the list. In step 940, the highlight reel 220 with modified boundaries may be extracted and optionally processed by deploying an decay function with mirrored front and tail curvatures, allowing further smoothing of the audio transitions between disjoint reels (e.g., multiple highlight reels 220 played in succession).

The present systems and methods have been described in particular detail with respect to possible embodiments. Those skilled in the art will appreciate that the systems and methods may be practiced in other embodiments. First, the particular naming of the components, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms and/or features may have different names, formats, or protocols. Furthermore, the system may be implemented via a combination of hardware and software, or entirely in hardware elements, or entirely in software elements. Moreover, the particular division of functionality between the various system components described herein is merely exemplary and not mandatory; functions performed by a single system component may actually be performed by multiple components, and functions performed by multiple components may actually be performed by a single component.

Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" or "in at least one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

Various embodiments may include any number of systems and/or methods for performing the above-described techniques, alone or in any combination. Another embodiment includes a computer program product comprising a non-transitory computer-readable storage medium and computer program code encoded on the medium for causing a processor in a computing device or other electronic device to perform the techniques described above.

Some portions of the above are presented in terms of algorithms and symbolic representations of operations on data bits within a memory of a computing device. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is sometimes convenient to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices without loss of generality.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the description, discussions utilizing, for example, "processing" or "computing" or "calculating" or "displaying" or "determining" or the like, refer to the action and processes of a computer system, or similar electronic computing module and/or device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.

Certain aspects include the processing steps and instructions described herein in the form of algorithms. It should be noted that the process steps and instructions may be implemented in software, firmware, and/or hardware, and when implemented in software, may be downloaded to reside on and be operated from different platforms for use by a variety of operating systems.

The present document also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computing device selectively activated or reconfigured by a computer program stored in the computing device. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, DVD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (random access memories; RAMs), EPROMs, EEPROMs, flash memory, solid state drives, magnetic or optical cards, application specific integrated circuits (application specific integrated circuit; ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. The program and its associated data may also be hosted and run remotely, for example, on a server. Furthermore, the computing devices referred to herein may comprise a single processor or may be architectures employing multi-processor designs for increased computing capability.

The algorithms and displays presented herein are not inherently related to any particular computing device, virtualization system, or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may be a special purpose apparatus that is more convenient to construct a necessary method steps to perform the required method steps. The desired structure for a variety of these systems will become apparent from the description provided herein. In addition, the systems and methods are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings described herein, and any references above to specific languages are provided for disclosure of enablement and best mode.

Accordingly, various embodiments include software, hardware, and/or other elements for controlling a computer system, computing device, or other electronic device, or any combination or combinations thereof. Such electronic devices may include, for example, processors, input devices (e.g., keyboard, mouse, touch pad, trackpad, joystick, trackball, microphone, and/or any combination thereof), output devices (e.g., screen, speaker, etc.), memory, long-term storage devices (e.g., magnetic storage devices, optical storage devices, etc.), and/or network connectivity, in accordance with techniques well known in the art. Such electronic devices may be portable or non-portable. Examples of electronic devices that may be used to implement the described systems and methods include: desktop computers, laptop computers, televisions, smart phones, tablet computers, music players, audio devices, all-in-one machines, set-top boxes, gaming systems, wearable devices, consumer electronic devices, server computers, and the like. The electronic device may use any operating system, such as, and not limited to: linux; microsoft Windows from Microsoft corporation of Redmond, washington; mac OS X from apple Inc. of Coptis, california; iOS from apple inc of kubi, california; android purchased from google corporation of mountain view, california; and/or any other operating system suitable for use on the device.

While a limited number of embodiments have been described herein, those skilled in the art, having benefit of the above description, will appreciate that other embodiments may be devised. It should also be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the subject matter. Accordingly, the present disclosure is intended to be illustrative of the scope, but not limiting.

Claims

1. A method for identifying boundaries of highlight segments of audiovisual content depicting an event, the method comprising:

storing, at a data store, audio data depicting at least a portion of the event;

processing, at a processor, the audio data to generate a spectrogram for at least a portion of the audio data;

at a processor, the audio data is automatically analyzed to detect soft entry points of the audio data by:

applying a sliding two-dimensional time-frequency analysis window of a sub-second time range to the spectrogram;

calculating an average spectral amplitude indicator for each position of the sliding two-dimensional time-frequency analysis window; and

using the average spectral amplitude indicator to form a vector of spectral amplitude indicator/position pairs for the spectrogram;

For each element of the vector having a spectral amplitude indicator/position pair, converting the spectral amplitude indicator into an integer qualifier according to the spectral amplitude indicator, a maximum value of the spectral amplitude indicator, and a factor based on a predetermined number of target integer qualifiers;

generating an initial vector for the spectrogram having integer qualifier/position pairs; and

detecting a soft entry point using the initial vector; and

at the processor, a time index corresponding to the soft entry point is specified within the audiovisual content as the boundary, the boundary including one of a beginning of the highlight and an end of the highlight.

2. The method of claim 1, wherein the audiovisual content comprises a television broadcast.

3. The method of claim 1, wherein the audiovisual content comprises an audiovisual stream, and wherein the method further comprises: the audio data is extracted from the audiovisual stream before storing the audio data depicting at least part of the event.

4. The method of claim 1, wherein the audiovisual content comprises stored audiovisual content, and wherein the method further comprises: before audio data describing at least part of the event is stored, the audio data is extracted from the stored audiovisual content.

5. The method according to claim 1, wherein:

the event includes a sporting event; and is also provided with

The highlight reel depicts what is believed to be of particular interest to at least one user in the sporting event.

6. The method of claim 5, further comprising: at an output device, at least one of the audiovisual content and the highlight is played during detection of the soft entry point.

7. The method as recited in claim 1, further comprising: the audio data is pre-processed by resampling the audio data to a desired sample rate prior to detecting the soft entry point.

8. The method as recited in claim 1, further comprising: prior to detecting the soft entry point, pre-processing the audio data by filtering the audio data to perform at least one of:

noise is reduced; and

the spectral band of interest is selected.

9. The method of claim 1, wherein detecting the soft entry point further comprises:

dividing the initial vector with integer qualifier/position pairs into successive one second intervals; and

the integer qualifier per second interval is maximized.

10. The method of claim 9, wherein maximizing integer qualifiers for each second interval comprises:

classifying integer qualifiers for each one-second interval; and

non-maximum suppression is performed in each one second interval to form a first vector of integer qualifier/position pairs of the spectrogram.

11. The method of claim 10, wherein detecting the soft entry point further comprises:

traversing progressively time positions of elements of the first vector of integer qualifier/position pairs;

for each temporal location, comparing the time of the current location with the time of the previous location to obtain a temporal distance; and

for each element of the first vector of integer qualifier/position pairs for which the temporal distance is greater than a threshold, finding the largest integer qualifier in the immediate neighborhood of the current position; and populating a new soft entry vector with the integer qualifier/position pair having the largest integer qualifier.

12. The method of claim 11, wherein finding the largest integer qualifier in the immediate neighborhood further comprises:

designating a first element of the first vector as an anchor element; and

A next element is selected that is about two seconds from the first element.

13. The method of claim 12, wherein finding the largest integer qualifier in the immediate neighborhood further comprises:

checking for elements near either side of the next element; and

the next element and the element with the maximized integer qualifier among the elements near either side of the next element are designated as new anchor elements.

14. The method of claim 13, wherein finding the largest integer qualifier in the immediate neighborhood further comprises: all elements of the first vector of integer qualifier/position pairs are processed in successive steps to produce a set of soft entry points having variable mutual distances and having maximized integer qualifiers.

15. The method as recited in claim 14, further comprising:

converting the set of soft entry points into a list of optimal entry points; and

the time index is selected from the list of best entry points.

16. The method as recited in claim 1, further comprising: identifying the highlight segments with tentative boundaries before designating the time index as the boundary;

Wherein:

the soft entry point of a plurality of soft entry points in the audio data is closest in time to the tentative boundary; and is also provided with

Designating the time index as the boundary includes replacing the tentative boundary with the boundary.

17. A non-transitory computer-readable medium for identifying boundaries of highlight segments of audiovisual content depicting an event, comprising instructions stored thereon, which when executed by a processor perform operations comprising:

causing a data store to store audio data depicting at least part of the event;

processing the audio data to generate a spectrogram for at least a portion of the audio data;

automatically analyzing the audio data to detect soft entry points of the audio data by:

detecting a soft entry point using the initial vector; and

a time index corresponding to the soft entry point is designated as the boundary within the audiovisual content, the boundary including one of a start of the highlight and an end of the highlight.

18. The non-transitory computer-readable medium of claim 17, wherein the audiovisual content comprises a television broadcast.

19. The non-transitory computer-readable medium of claim 17, wherein:

the event includes a sporting event; and is also provided with

20. The non-transitory computer-readable medium of claim 19, further comprising instructions stored thereon that, when executed by a processor, cause an output device to play at least one of the audiovisual content and the highlight during detection of the soft entry point.

21. The non-transitory computer-readable medium of claim 17, further comprising instructions stored thereon that, when executed by a processor, pre-process the audio data prior to detecting the soft entry point by performing at least one of:

Resampling the audio data to a desired sampling rate;

filtering the audio data to reduce noise; and

the audio data is filtered to select a spectral band of interest.

22. The non-transitory computer-readable medium of claim 17, wherein detecting the soft entry point comprises:

maximizing the integer qualifier for each second interval;

wherein maximizing the integer qualifier for each second interval comprises:

classifying the qualifier Fu Zhengshu qualifiers for each one-second interval; and

23. The non-transitory computer-readable medium of claim 22, wherein detecting the soft entry point further comprises:

for each temporal location, comparing the time of the current location with the time of the previous location to obtain a temporal distance;

for each element of the first vector of integer qualifier/position pairs for which the temporal distance is greater than a threshold, finding the largest integer qualifier in the immediate neighborhood of the current position; and

Populating a new soft entry vector with the integer qualifier/position pair having the largest integer qualifier;

wherein finding the largest integer qualifier in the immediate neighborhood further comprises:

designating a first element of the first vector as an anchor element;

selecting a next element about two seconds from the first element;

checking for elements near either side of the next element;

designating the next element and the element having the maximized integer qualifier among the elements near either side of the next element as new anchor elements; and

processing all elements of said first vector of integer qualifier/position pairs in successive steps to produce a set of soft entry points having variable mutual distances and having maximized integer qualifiers;

wherein the non-transitory computer-readable medium further comprises instructions stored thereon that, when executed by a processor:

the time index is selected from the list of best entry points.

24. The non-transitory computer-readable medium of claim 17, further comprising instructions stored thereon that, when executed by a processor, identify the highlight segments with tentative boundaries prior to designating the time index as the boundary;

Wherein:

25. A system for identifying boundaries of highlight segments of audiovisual content depicting an event, the system comprising:

a data store configured to store audio data depicting at least a portion of the event; and

a processor configured to:

detecting a soft entry point using the initial vector; and is also provided with

26. The system of claim 25, wherein the audiovisual content comprises a television broadcast.

27. The system according to claim 25, wherein:

the event includes a sporting event; and is also provided with

28. The system of claim 27, further comprising an output device configured to play at least one of the audiovisual content and the highlight during detection of the soft entry point.

29. The system of claim 25, wherein the processor is further configured to pre-process the audio data to perform at least one of the following prior to detecting the soft entry point:

resampling the audio data to a desired sampling rate;

Filtering the audio data to reduce noise; and

the audio data is filtered to select a spectral band of interest.

30. The system of claim 29, wherein the processor is further configured to detect the soft entry point by:

maximizing the integer qualifier for each second interval;

wherein the processor is further configured to maximize the integer qualifier for each second interval by:

classifying integer qualifiers for each one-second interval; and

31. The system of claim 30, wherein the processor is further configured to detect the soft entry point further by:

wherein the processor is further configured to find the maximum Q in the immediate neighborhood by:

designating a first element of the first vector as an anchor element;

selecting a next element about two seconds from the first element;

checking for elements near either side of the next element;

designating the next element and the element having the maximized integer qualifier among the elements near either side of the next element as new anchor elements;

wherein the processor is further configured to:

The time index is selected from the list of best entry points.

32. The system of claim 25, wherein the processor is further configured to identify the highlight segments with tentative boundaries prior to designating the time index as the boundary;

wherein: