KR100870870B1 - High quality time-scaling and pitch-scaling of audio signals - Google Patents

Abstract

In one alternative, the audio signal is analyzed using multiple psychoacoustic criteria to identify areas of the signal where time scaling and / or pitch shifting processing is audible or minimally audible, and the signal is time skewed within that area. Ring and / or pitch shifted. In another alternative, the signal is divided into auditory events, and the signal is time scaled and / or pitch shifted within the auditory event. In another alternative, the signal is divided into divisions into auditory events, wherein the auditory events are audible or minimally audible with time scaling and / or pitch shifting processing of the signal and psychoacoustic to identify the auditory events. Analyze using criteria. Other alternatives consider multiple channels of audio.

Description

HIGH QUALITY TIME-SCALING AND PITCH-SCALING OF AUDIO SIGNALS}

The present invention relates to the field of psychoacoustic processing of audio signals. In particular, the present invention relates to aspects of where and / or how to perform time scaling and / or pitch scaling (pitch shifting) of an audio signal. The processing is particularly applicable to an audio signal represented by samples, such as digital audio signals. The invention also relates to an aspect of splitting audio into "auditory events", each of which tends to be perceived separately.

Time scaling relates to changing the time evolution or duration of an audio signal without changing the spectral content (perceived timbre) or perceived pitch of the signal (pitch is a characteristic associated with a periodic audio signal). Pitch scaling relates to modifying the spectral content or perceived pitch of an audio signal without affecting its time evolution or duration. Time scaling and pitch scaling are overlapping methods. For example, the pitch of a digitized audio signal can be increased by increasing the time period of the signal by 5% and then reading the samples at 5% higher sample rate (e.g., by resampling), thereby reducing the original time. Maintaining the time period scales up to 5% without affecting the time period. The resulting signal has the same time period as the original signal except that it has modified pitch or spectral characteristics. As discussed further below, resampling may be applied but is not a necessary step unless it is required to maintain a constant output sampling rate or to keep the input and output sampling rates the same.

There are many uses for advanced methods that provide independent control of the time and pitch characteristics of an audio signal. This is especially true for high fidelity, multichannel audio that includes a wide range of content, from simple tonal signals to voice signals and complex musical passages. Usage for time and pitch scaling includes audio / video broadcast, audio / video edit synchronization, and multi-track audio recording and mixing. In audio / video broadcasting and editing environments, it is necessary to play the video at a different rate than the source material, resulting in a pitch-scaled version of the accompanying audio signal. Pitch scaling the audio maintains synchronization between the audio and video and preserves the timbre and pitch of the original source material. In multi-track audio or audio / video editing, new material is required to match the time-suppression period of the audio or video portion. Time-scaling audio can time-suppress new portions of audio without modifying the timbre and pitch of the source audio.

According to an aspect of the present invention, a method of time scaling and / or pitch shifting an audio signal is provided. The signal is analyzed using multiple psychoacoustic criteria to identify an area of the audio signal where time scaling and / or pitch shifting processing of the audio signal is not audible or minimally audible, and the signal is timed within that area. Scaling and / or pitch shifting.

According to another aspect of the present invention, a method of time scaling and / or pitch shifting multiple channels of audio signals is provided. Each channel of the audio signals is analyzed using at least one psychoacoustic criterion to identify an area in the channels of audio signals where time scaling and / or pitch shifting processing of the audio signal is audible or minimally audible, All multiple channels of audio signals are time scaled and / or pitch shifted among time segments that are in the region identified in at least one of the channels of audio signals.

According to another aspect of the present invention, a method of time scaling and / or pitch shifting an audio signal is provided, wherein the audio signal is divided into auditory events and the signal is time scaled and / or pitch shifted within the auditory event. .

According to another aspect of the invention, a method of time scaling and / or pitch shifting a plurality of audio signal channels is provided, wherein the audio signal of each channel is divided into an auditory event. Once the combined auditory events are determined, each has a boundary when the auditory event boundary occurs on any audio signal channel. Since all audio signal channels are time scaled and / or pitch shifted within the combined auditory event, time scaling and / or pitch shifting is within the auditory event of each channel.                 

According to another aspect of the present invention, a method of time scaling and / or pitch shifting an audio signal is provided, wherein the signal is divided into auditory events, wherein the auditory event is time scaling and / or pitch shifting processing of the audio signal. This is analyzed using psychoacoustic criteria to identify auditory events that are not audible or minimally audible. Time scaling and / or pitch shifting processing is performed within an auditory event in which the time scaling and / or pitch shifting processing of the audio signal is identified as not audible or minimally audible.

According to another aspect of the invention, a method of time scaling and / or pitch shifting multiple channels of audio signals is provided, wherein the audio signals of each channel are divided into auditory events. Auditory events are analyzed using at least one psychoacoustic criterion to identify auditory events where time scaling and / or pitch shifting processing of the audio signal is not audible or minimally audible. Once the combined auditory events are determined, each has a boundary where the auditory event boundary occurs in the audio signal of any channel. Time scaling and / or pitch shifting processing is performed in a combined auditory event in which time scaling and / or pitch shifting processing in multiple channels of audio signals is identified as being audible or minimally audible.

According to another aspect of the invention, analyzing an audio signal using multiple psychoacoustic criteria comprises analyzing the audio signal to identify an area of the audio signal in which the audio meets at least one criterion in the group of psychoacoustic moods. It includes.

According to another aspect of the invention, the psychoacoustic criteria comprises one or more of the following: (1) The identified region of the audio signal is premasked or postmasked as a result of transient transients, and (2) the audio signal The identified region is largely inaudible, (3) the identified region of the audio signal is predominantly high frequency, and (4) the identified region of the audio signal is a quieter part of the segment of the audio signal and precedes and / or the region. The portion or portions of the segment that follows are louder. Some basic principles of psychoacoustic masking are described below.

An aspect of the invention is that groups of psychoacoustic criteria are arranged in descending order (ie hierarchical criteria) of increasing audible artifacts resulting from time scaling and / or pitch scaling processing. According to another aspect of the invention, an area is identified when the highest grade psychoacoustic criterion (ie, the criterion leading to the least audible workpiece) is met. In contrast, even if a criterion is met, another criterion is examined to identify one or more other areas of audio that meet the criterion. The latter approach is useful in the case of multichannels to determine the location of all possible regions that meet any criterion, including criteria below the hierarchy, so there are more possible common splice points between multiple channels. .

Although aspects of the invention use other types of time scaling and / or pitch shifting processing (e.g., a process disclosed in US Pat. No. 6,266,003 B1, which is incorporated herein by reference in its entirety), aspects of the invention Hear:                 

The junction is selected in the audio signal area, defining the leading segment of the audio signal that in time leads the junction,

An endpoint spaced apart from the junction is selected to define a trailing segment of the audio signal that tracks the endpoint in time and the target segment of the audio signal between the junction and the endpoint,

When the leading and trailing segments are connected at the junction, so that the end point is later than the junction (with a higher number of samples), the target segment can be omitted to shorten the time period of the audio signal (for a digital audio signal represented by a sample). Reduce the number of audio signal samplings), or extend the time period (increase the number of samples) by repeating the target segment when the endpoint is earlier (having a lower number of samples) than the junction.

One type of time scaling and / or pitch shifting process is preferably used, which reads the combined leading and trailing segments at a rate that yields the desired time scaling and / or pitch shifting.

Connected leading and trailing segments are:

The same time period, like the original time period, causes the audio signal to be pitch shifted,

In the case of omitting the target segment, the reduced time period at the same rate as the relative change in the decrease in the number of samples causes the audio signal to be time compressed,

In the case of repeating the target segment, an increased time period at the same rate as the relative change in the increase in the number of samples causes the audio signal to time expand,

A reduced time period at a rate that is different from the relative change in the decrease in the number of samples causes time compression and pitch shifting of the audio signal, or

The increased time period at a different rate than the relative change in the increase in the number of samples is read at the same rate as time expansion and pitch shifting the audio signal.

Even if the target segment is omitted (data compression) or repeated (data expansion), there is only one junction and one junction. In the case of omitting the target segment, the junction is where the junction point and the endpoint of the omitted target segment are connected or joined together. In the case of repeating the target segment, there is only a single junction-the junction is where the end of the first rendition (junction) of the target segment meets at the beginning of the second rendition (endpoint) of the target segment. In the case of reducing the number of audio samples (data compression), for criteria other than premasking or postmasking, it is desirable that the endpoint is in the identified area (and the junction must always be in the identified area). In the case of compression where the junction is premasked or postmasked by transients, the endpoint need not be in the identified area. For other cases (except where processing occurs within the auditory event, as described below), it is desirable that the endpoint be in the identified region so that nothing will be audible without omission or repetition. In the case of increasing the number of audio samples (data expansion), the endpoint of the original audio is preferably in the identified region of the audio signal. As will be discussed below, the possible junction locations have the earliest and most recent time and the possible endpoint locations have the earliest and most recent time. When audio is represented as samples in the data block of the buffer memory, the possible junction positions have the minimum and maximum positions in the block, which have the earliest and most recent possible junction times, respectively, and the endpoint also has the minimum and maximum positions in the block. Which has the earliest and most recent possible endpoint time.

When processing multichannel audio, it is desirable to maintain the relative amplitude and phase between the channels so as not to disturb the directional cues. Therefore, if the target segment of audio in one channel is to be omitted or repeated, the corresponding segments of another channel (with the same sample indices) must also be omitted or repeated. Thus, it is necessary to find a target segment that is generally common to all channels that allow for inaudible splicing on all channels.

Justice

Throughout this document, the term "data compression" relates to reducing the number of samples by omitting a segment, leading to time compression, and the term "data expansion" to repeating a segment It relates to increasing the number of fields, inducing time expansion. Audio "region", "segment", and "portion" refer to the representation of a finite continuous portion of audio from a single channel that is conceptually between any two moments in time. Such an area, segment, or portion is represented by samples having consecutive sample numbers or exponents. An "identified region" refers to a region, segment or portion of audio identified by psychoacoustic criteria and within which there may be junctions, and generally endpoints. A "correlation processing region" relates to an area, segment or portion of audio in which correlations are performed in the search for endpoints or junctions and endpoints. "Psychoacoustic criteria" include criteria based on time domain masking, frequency domain masking, and / or other psychoacoustic factors. As mentioned above, the "target segment" is the portion of audio that is removed in case of data compression or repeated in case of data expansion.

Masking

Aspects of the present invention make use of human audible and in particular psychoacoustic phenomena known as masking. Some simplified masking concepts will be understood with reference to FIG. 1 and the description below. The solid line 10 in FIG. 1 represents the sound pressure level at which sound, such as a sine wave or narrowband noise, can be heard, that is, an audible threshold. Sound at levels above the curve can be heard; No sound under the curve is heard. This threshold is very frequency dependent. One can hear softer sound at 4 kHz than at 50 Hz or 15 kHz. At 25 kHz, the threshold is off scale: no matter how noisy, humans cannot hear it.

Consider a relatively loud signal at one frequency, i.e. the threshold in the presence of a 500 Hz sine wave at 12. The modified threshold 14 increases dramatically in the immediate vicinity of 500 Hz, moderately somewhat off the frequency, and not at all in the far part of the audible range.

This rise in threshold is called masking. In the presence of a noisy 500 Hz sine wave signal ("masking signal" or "masker"), signals below this threshold, referred to as "masking threshold", are concealed or masked by the loud signal. In addition, other signals may rise somewhat at the level above the non-signal threshold, but are still inaudible as they are below the new masked threshold. However, in the far part of the spectrum where the non-signal threshold does not change, any sound that could be heard without a 500 Hz masker remains as it can be heard with it. Therefore, masking is not dependent on the simple presence of one or more masking signals; It depends where the signals are in the spectrum. For example, some music passages contain numerous spectral components distributed over an audible frequency range, thus conveying a masked threshold curve that rises anywhere in proportion to the non-signal threshold curve. Other musical passages, for example, consist of a relatively loud sound from a single instrument having a spectral component defined by a small portion of the spectrum, thus conveying a masked curve closer to the sine wave masker example of FIG.

Masking also has a transient aspect that depends on the time relationship between the masker (s) and the masked signal (s). Some masking signals essentially provide masking only while the masking signal is present ("simultaneous masking"). Other masking signals provide masking while the masker occurs earlier ("backward masking" or "premasking") as well as later ("forward masking" or "postmasking"). "Transient", ie, sudden, short-term and significant increases in signal levels all represent three "types" of masking: backward masking, simultaneous masking, and forward masking, while steady state and quasi-steady state. The signal represents only simultaneous masking. In the context of the present invention, the advantage should not be influenced by the simultaneous masking resulting from the transient, because it is not desirable to interrupt the transient by placing junctions that coincide with or nearly coincide with simultaneous masking.

Audio transient data has long been known to provide forward and backward transient masking. Transient audio material "masks" the audible material before and after the transient so that the preceding and following audio is not perceptible to the listener (simultaneous masking by transient is used to avoid repeating or confusing the transient). Is not). Premasking was measured, relatively short, lasting only a few milliseconds (msec) and forcing masking lasting longer than 50msec. Although postmasking is generally more useful because of its longer duration, both pre- and post-transient masking will be used in connection with aspects of the present invention.

One aspect of the present invention is transient detection. In the actual implementation described below, subblocks (parts of the block of audio samples) are tested. The dimensions of the magnitudes are compared to a smooth moving average representing the signal magnitude up to that point. Operation is performed for the entire audio spectrum and only for high frequencies, respectively, to ensure that high frequency transients are not diluted and lost by the presence of larger low frequency signals. Alternatively, any suitable known method for detecting transients can be used.

Bonding creates an obstruction that results in a workpiece with spectral components that decay over time. The spectrum (and amplitude) of the junction workpiece is: (1) the spectrum of the signals to be joined (as further described below, it is recognized that the workpiece potentially has a different spectrum than the signals to which it is joined), (2) the joints to be joined together When the waveforms match (avoid discontinuity), and (3) the shape and duration of the crossfade where the waveforms are joined together at the junction. Crossfading according to aspects of the invention is further described below. Correlation techniques that assist in matching waveforms at the point of connection are also described below. According to an aspect of the invention, it is preferred that the joining workpiece is masked or not audible or minimally audible. Psychoacoustic criteria contemplated by aspects of the present invention include criteria that result in masked, inaudible, or minimally audible workpieces. Audibleness or minimal audibility is considered as a type of masking. Masking requires that the workpiece be constrained in time and frequency so that the workpiece is below the masking threshold of the masking signal (s) (or in the absence of the masking signal (s), below the non-signal threshold of audibility, which is one form of masking. Will be considered). The duration of the workpiece is well defined as being the first approximation, essentially the length of the crossfade (time period). The slower the crossfade, the narrower the spectrum of the workpiece but the longer the duration.

Some general principles regarding making the junction inaudible or minimally audible will be understood by considering a continuum of rising signal levels. Consider the case of splicing low-level signals that provide less masking or no masking. Well performed junctions (ie, waveforms that are well matched with minimal discontinuity) introduce a workpiece that is somewhat lower in amplitude, perhaps below the audible threshold, so no masking signal is required. As the levels rise, the signals begin to act as masking signals, raising the audible threshold. The workpieces also rise in amplitude, so they are above the non-signal threshold, except when the audible threshold has also been raised (as described above with respect to FIG. 1).

Ideally, according to aspects of the present invention, for transient masking of the workpieces, the workpieces generate transient masking or forward masking transient regions of the transients and the amplitudes of the spectral components of all the workpieces in time. It is below the masking threshold of the transient. However, in practical implementations, not all spectral components of the workpieces are masked every moment of time.

Ideally, according to aspects of the present invention, for a steady state or quasi-steady state of masking the workpieces, the workpieces occur simultaneously as a masking signal (simultaneous masking), and all spectral components are instantaneously corrected for the steady state signal. It is below the masking threshold.

According to another aspect of the invention there is another possibility, that the amplitude of the spectral components of the workpieces is below the non-signal threshold of human audibility. In this case, such audibility is considered to be masking of the workpiece, but no masking signal is required.                 

In principle, with sufficient processing power and / or processing time, it is possible to predict the time and spectral characteristics of the workpieces based on the signal being joined to determine whether the workpieces are masked or not audible. However, in order to save processing power and time, by considering the magnitude of the signals to be bonded near the junction (especially within the crossfade), or in the case of the overwhelming high frequency identification region of steady state or quasi-steady state in the signal, By considering the frequency content of the signals being spliced regardless, useful results will be obtained.

The magnitudes of the workpieces resulting from the junction are generally less than or similar to the magnitudes of the signals to be bonded. In general, however, it is not practical to predict the spectrum of workpieces. If the junction is in the region of the audio signal below the threshold of human audibility, the resulting artifacts, even if small or similar in magnitude, are above the threshold of human audibility because they contain frequencies that are more sensitive to the ear. (With lower threshold). Therefore, when evaluating audibility, it is desirable to compare the signal amplitude with the audible threshold at a fixed level, ie the most sensitive frequency of the ear (about 4 kHz), rather than the true frequency-dependent threshold of audible. This conservative approach ensures that the workpieces are below the actual audible threshold that appears in the spectrum. In this case, the length of the crossfade should not affect the audibility, but it is preferable to use a relatively short crossfade to allow the best space for data compression or expansion.

The human ear has a lack of susceptibility to discontinuities in the dominant high frequency waveforms (eg, high frequency clicks resulting from high frequency waveform discontinuities are more easily masked or not heard than at low frequency clicks). In the case of high frequency waveforms, the components of the workpieces are also dominant high frequencies and masked regardless of the signal magnitude at the junction (because of the steady state or quasi-steady nature of the identified area, the magnitudes at the junction act as maskers) Similar to the magnitudes of the signal in the identified region). This will be considered as the case of simultaneous masking. In this case, although the length of the crossfade probably does not affect the audibility of the workpiece, it is preferable to use a relatively short crossfade to allow the best space for data compression or expansion processing.

If the junction is in the region of the audio signal identified as being masked by transients (i.e., by either premasking or postmasking), the junction, including the craze fading length, takes into account the crossfading characteristics applied. The magnitude of each signal determines if a particular junction is masked by transients. The amount of masking provided by the transients decays over time. Therefore, in the case of premasking or postmasking due to transients, use a relatively short crossfade, which is a larger obstacle but lasts for a shorter time and causes disturbances that may be within the time period of premasking or postmasking. It is desirable to.

When the junction is in the region of an audio signal that is not premasked or postmasked as a result of the transient, an aspect of the invention is to select the quietest sub-segment of the audio signal in the segment of the audio signal (substantially, segment Is a block of samples in the buffer memory). In this case, the magnitude of each signal to be bonded, including the crag fading length, taking into account the crossfading characteristics applied, determines the limit in which workpieces caused by the bonding blockages can be heard. If the level of the sub-segment is low, the level of the workpiece components is also low. Depending on the level and spectrum of the lower sub-segment, there may be some simultaneous masking. In addition, the higher-level portions of the audio surrounding the lower-level sub-segments also provide some temporary premasking or postmasking, which raises the threshold during the crossfade. The workpieces are not always audible but can be less audible if the bonding is performed in a noisy area. Such audibility is minimized by using longer crossfade lengths and by matching waveforms well at the junction. However, the long crossfade limits the length and position of the target segment (in practical cases, the audio samples may block because the length of the fluctuating audio is considerably longer and the junctions and / or endpoints are further away from the end of the block). Divided into two). Thus, the maximum crossfade length is a compromise.

Auditory Scene Analysis

Using psychoacoustic analysis is useful for reducing undesirable audible artifacts in the process to provide time and / or pitch scaling, but the reduction of undesirable audible artifacts also tends to be perceived individually, respectively. This is accomplished by dividing the audio into time segments, referred to as "events" or "auditory events," and by executing time-scaling and / or pitch-scaling processes within the event. The division of sound into individually perceived units is sometimes referred to as "audit event analysis" or "audit scene scene analysis (ASA)". Although psychoacoustic analysis and auditory scene analysis are used independently as aids in reducing undesirable workpieces in the time and / or pitch scaling process, they are beneficial to each other.

Time and / or pitch scaling related to (1) psychoacoustic analysis, (2) auditory scene analysis, and (3) correlated psychoacoustic and auditory scene analysis are all aspects of the present invention. Another aspect of the invention includes the use of psychoacoustic analysis and / or auditory scene analysis as part of the type of time and / or pitch scaling in addition to the type in which segments of audio are deleted or repeated. For example, processes for time scaling and / or pitch correction of audio signals disclosed in US Pat. No. 6,266,003 may only include audio segments and / or auditory events that meet one or more psychoacoustic criteria described herein. It is improved by using the publication's processing technique only for the respective audio segments that do not exceed.

An extensive discussion of the auditory god analysis is Albert S. It is described by Bregman in his book Audition Scene Analysis-The perceptual Organization of Sound (Massachusetts Institute of Technology, 1991, Fourth printing, 2001, Second MIT Press paperback edition). US Pat. No. 6,002,776 to Bhadkamkar et al. Cites publications in 1976 as “a prior art relating to sound separation by auditory scene analysis.” However, Bad de Kamkar et al. Even if the techniques involved in analysis are of interest to human auditory models from a scientific point of view, they are now too technically demanding and are not professional enough to be considered as practical techniques for sound separation until fundamental progress is made. ” It disappoints the practical use of auditory scene analysis.

In accordance with aspects of the present invention, a computationally effective process is provided for dividing audio into temporal segments or "audit events" that tend to be perceived individually.

Bregman mentions, in some cases, that we hear a separate unit when the sound suddenly changes in tone, pitch, loudness, or (with less limits) the location of the space. (Auditory Scene Analysis-The Perceptual Organization of Sound , upra at page 469). Bregman also discusses their perception, for example, when multiple simultaneous sound streams are separated in frequency.

In order to detect fluctuations in timbre and pitch and constant fluctuations in amplitude, the auditory event detection process according to aspects of the present invention detects fluctuations in spectral composition over time. When applied to a multichannel sound arrangement in which the channels are oriented in space, the process according to aspects of the present invention also detects auditory events resulting from variations in spatial location over time. Optionally, according to another aspect of the present invention, the process also detects a change in amplitude with respect to time, which is not detected by detecting a change in spectral configuration with respect to time. Implementing time-scaling and / or pitch-scaling within an auditory event is likely to lead to less audible artifacts, because the audio in the event is fairly constant, perceived to be fairly constant, or by itself an audio object (eg. For example, a note played by an instrument.

In at least computationally demanding implementations, the process involves analyzing the entire frequency band (full bandwidth audio) or generally the entire frequency band (in practical implementations, band limit filtering is often used at the end of the spectrum) and the loudest audio signal components. Split the audio into time segments by giving the largest weight to. This approach uses psychoacoustic phenomena in which the ear concentrates on a single auditory event at a fixed time on a smaller time scale (less than 20 msec). This means that when multiple events occur at the same time, one component is perceptually most prominent and treated separately as if a single event occurred. Using this effect allows the auditory event detection to be made at a constant rate depending on the complexity of the audio being processed. For example, if the input audio signal being processed is a single instrument, the auditory events identified are the individual notes played. Similar to the input speech signal, the individual components of language, ie vowels and consonants, will be identified, for example, as separate audio elements. As the complexity of audio, such as drums or music with multiple instruments and voices, increases, auditory event detection identifies the most prominent (ie, loudest) audio element at any moment. Alternatively, the "most prominent" audio element can be determined by considering the audible threshold and the frequency response.                 

Optionally, according to another aspect of the present invention, at the expense of large computational complexity, the process is a variation of spectral composition with respect to time in the discrete frequency bands (bands fixed or dynamically determined or fixed and dynamically determined) rather than full bandwidth. Consider. This alternative approach considers one or more audio streams in different frequency bands rather than presuming that only a single stream is perceptible at a particular time.

A simple and computationally effective process according to the invention for segmenting audio has been found to be useful for identifying auditory events and for reducing audible artifacts when used with time and / or pitch deformation techniques.

The auditory event detection process of the present invention is implemented by dividing the time domain audio waveform into time intervals or blocks using a filter bank or time-frequency conversion, such as an FFT, and then converting the data of each block into the frequency domain. . The amplitude of the spectral content of each block is normalized to remove or reduce the effects of amplitude variations. Each resulting frequency domain indication provides an indication (amplitude as a function of frequency) of the spectral content of the audio in a particular block. The spectral content of successive blocks is compared and a variation greater than the threshold is taken to indicate a temporary start or temporary end of the auditory event.

In order to minimize the computational complexity, only a single band of frequencies of the time domain audio waveform is processed, or preferably the entire frequency band of the spectrum (which is 50 Hz to 15 kHz for average quality music systems) or generally the entire frequency band. (E.g., a band defining filter excludes high and low frequency extremes).

The degree required for the frequency domain data to be standardized refers to the frequency of the indication. Therefore, if this variation in class exceeds a certain threshold, it is also accepted as indicating an event boundary. Event start and end points resulting from spectral variation and from amplitude variation are ORed together to identify event boundaries resulting from either type of variation.

In practice, the temporal start and end boundaries of the auditory event necessarily coincide with the boundaries of the blocks where the time domain audio waveform is divided. There is a trade-off between real-time processing requirements (large blocks require less processing overhead) and resolution of event locations (small blocks provide more detailed information about the location of the auditory event).

In the case of multiple audio channels, each indicating a direction in space, each channel is processed independently and the resulting event boundaries for all channels are ORed together. Thus, for example, an auditory event that suddenly switches directions is a "end event" boundary in one channel and a "start event" boundary in another channel. When ORed together, two events will be identified. Therefore, the auditory event detection process of the present invention detects auditory events based on spectral (voice and pitch), amplitude and directional variations.

As an additional choice, but at the expense of greater computational complexity, instead of processing the spectral content of the time domain waveform in a single band of frequencies, the spectrum of the time domain waveform in two or more frequency bands prior to the frequency domain transformation. Divided. Each frequency band is then converted into the frequency domain and treated as if it were an independent channel in the manner described above. The resulting event boundaries are ORed together to define the event boundaries for that channel. Multiple frequency bands are fixed, adaptive, or a combination of fixed and adaptive. For example, tracking filter techniques and other techniques used to reduce audio noise are used to define adaptive frequency bands (e.g. two adaptive dominant sine waves at 800 Hz and 2 kHz around their two frequencies). Bands as determined by

Other techniques for providing auditory scene analysis are used to identify auditory events in various aspects of the present invention.

In the embodiment stated in the text, the audio is divided into fixed length sample blocks. However, the principles of the various aspects of the invention do not require to arrange audio into sample blocks, as well as to provide blocks of constant length (blocks are of variable length, each of which is essentially the length of an auditory event. ). When audio is divided into blocks, another aspect of the invention is not to process specific blocks in both single channel and multichannel environments.

Other aspects of the invention will be appreciated and understood as the description thereof is read and understood.

1 is an ideal diagram of an audible threshold of a person in the absence of sound (solid line) and in the presence of a 500 Hz sine wave (dashed line). The horizontal scale is the frequency in hertz (Hz) and the vertical scale is in decibels (dB) with respect to 20 μPa.

2A and 2B are schematic conceptual diagrams illustrating the concept of data compression by removing target segments. The horizontal axis represents time.

2C and 2D are conceptual conceptual diagrams illustrating the concept of data expansion by repeating target segments. The horizontal axis represents time.

3A is a schematic conceptual diagram of a block of audio data represented by a sample, showing a minimum junction position and a maximum junction position in the case of data compression. The horizontal axis is the sample and represents time. The vertical axis is the normalized amplitude.

3B is a schematic conceptual diagram of a block of audio data represented by a sample, showing a minimum junction position and a maximum junction position in the case of data expansion. The horizontal axis is the sample and represents time. The vertical axis is the normalized amplitude.

4 is a schematic conceptual diagram of a block of audio data represented by a sample, showing a junction point, a minimum end point position, a maximum end point position, a correlation processing region, and a maximum processing point position. The horizontal axis is a sample and represents time. The vertical axis is the normalized amplitude.

5 is a flow chart illustrating a time and pitch-scaling process in accordance with an aspect of the present invention in which psychoacoustic analysis is performed.

6 is a flowchart showing the details of the psychoacoustic analysis step 206 of FIG.

7 is a flowchart showing the details of the transient detection sub-step of the transient analysis step.

8 is a schematic conceptual diagram of a block of data samples in a transient analysis buffer. The horizontal axis is the samples in the block.

FIG. 9 is a schematic conceptual diagram illustrating an example of an audio block analysis in which a 450 Hz sine wave has an intermediate portion 6 dB lower in level than a starting and ending section in a block. The horizontal axis is samples representing time and the vertical axis is normalized amplitude.

FIG. 10 is a schematic conceptual diagram of how crossfading is implemented, illustrating an example of data segment conjugation using nonlinear crossfading formed along a Hanning window. The horizontal axis represents time and the vertical axis represents amplitude.

FIG. 11 is a flowchart illustrating details of the multichannel junction selection step 210 of FIG. 5.

12 is a series of ideal waveforms representing blocks of audio data samples on four audio channels, representing regions identified in each channel, each meeting a different criterion, overlapping the identified regions where a common multichannel junction is located. Indicates. The horizontal axis is samples and represents time. The vertical axis is the normalized amplitude.

13 shows time-domain information of the very periodic portion of an exemplary speech signal. Examples of well-chosen junctions and endpoints are shown to maximize the similarity of data on either side of the discarded data segment. The horizontal axis is samples representing time and the vertical axis is amplitude.

FIG. 14 is an ideal diagram of waveforms, showing the instantaneous phase (in radians) of a language signal superimposed between time domain signals x (n). The horizontal axis is the sample and the vertical axis is the normalized amplitude and phase (in radians).

FIG. 15 is a flowchart showing details of the correlation step 214 of FIG. FIG. 15 includes ideal waveforms showing the result of phase correlation in each of the five audio channels and the result of time-domain correlation in each of the five channels. The waveforms represent blocks of audio data samples. The horizontal axis is samples representing time and the vertical axis is normalized amplitude.

FIG. 16 is a schematic conceptual diagram having aspects of a block diagram and a flowchart and also including an ideal waveform showing an example of additional weighted correlation analysis processing. The horizontal axis of the waveform is samples representing time and the vertical axis is the normalized amplitude.

FIG. 17 is a flowchart stating time and pitch-scaling processes in accordance with aspects of the present invention in which psychoacoustic analysis and auditory scene analysis are performed.

18 is a flowchart showing details of auditory scene analysis step 706 of the process of FIG.

19 is a schematic conceptual diagram of a general method of calculating the spectral profile.

FIG. 20 is a series of ideal waveforms on two audio channels, illustrating the auditory event of each channel and the combined auditory event between the two channels.                 

21 is a flow chart showing details of psychoacoustic analysis step 708 of the process of FIG.

22 is a schematic conceptual diagram of a block of data samples in a transient analysis buffer. The horizontal axis is the samples in the block.

23 is an ideal waveform of a single channel of orchestral music showing auditory events and psychoacoustic criteria.

FIG. 24 is a series of ideal waveforms in four audio channels, illustrating the rating of the auditory event, psychoacoustic criteria, and combined auditory event.

FIG. 25 illustrates in more detail one combined auditory event of FIG. 24.

FIG. 26 illustrates an example of an auditory event of a skipped lower psychoacoustic quality class with an ideal waveform of a single channel.

FIG. 27 is a schematic conceptual diagram including an ideal waveform of a single channel, showing an initial step in selecting audio, junction and endpoint positions of a single channel according to an alternative aspect of the invention.

FIG. 28 is similar to FIG. 27 except that it shows the junction region Tc shifted by N samples.

FIG. 29 is a schematic conceptual diagram illustrating an example of multiple correlation calculations when a junction area is advanced in series by a Tc sample. FIG. Three processing steps are superimposed between the audio data block data diagrams. The processing illustrated in FIG. 29 is three correlation functions each having a maximum value as shown in each of FIGS. 30A-C.                 

FIG. 30A is an ideal correlation function for the case of the first junction region Tc position shown in FIG. 29.

FIG. 30B is an ideal correlation function for the case of the second junction region Tc location shown in FIG. 29.

FIG. 30C is an ideal correlation function for the case of the third junction region Tc location shown in FIG. 29.

FIG. 31 shows an example of an ideal audio waveform having three combined auditory event regions in which a target segment of 363 samples is selected in the first combined event region.

2A and 2B schematically illustrate the concept of data compression by eliminating target segments, while FIGS. 2C and 2D schematically illustrate the concept of data expansion by repeating target segments. In practice, the data compression and data expansion process is applied to data in one or more buffer memories, where the data are samples representing an audio signal.

Although the areas identified in FIGS. 2A-2D meet the criteria of postmasking as a result of signal transients, the principles based on the examples of FIGS. 2A-2D, including the other three mentioned above, It also applies to identified areas that meet psychoacoustic criteria.

Referring to FIG. 2A, as depicting data compression, audio 102 is part of audio 102 into psychoacoustical postmasked region 106 that constitutes an “identified region”. Has a transient 104. Audio is analyzed and junction 108 is selected within the identified area 106. As further described below with respect to Figures 3A and 3B, if audio is represented as a block of data in the buffer, then the lowest or earliest junction position within the block (i.e., if the data is represented as samples), the And the maximum or most recent junction location (ie, if the data is represented in samples, have a high sample number or index). Although in most cases it is desirable to position the junction at or near the lowest or earliest junction location in order to maximize the size of the target segment, the location of the junction is within the range of possible junction locations from the lowest junction location to the maximum junction location. Is chosen from and does not matter. At the default junction location, it takes a short time (such as 5ms) after the start of the identified area. Alternative methods of providing more optimized junction locations are described below.

Analysis of the audio continues and endpoint 110 is selected. In one alternative, the analysis involves autocorrelation of the audio 102 in the region 112 from the junction 108 forward (to a higher sample number or exponent) to the maximum processing point location 115. In practice, the maximum endpoint position is earlier (with a lower sample number or exponent) as the maximum processing point by a time (or time-equivalent number of samples) equal to half of the crossfade time, as described further below. . In addition, as described further below, the autocorrelation process searches for the maximum number of correlations between the lowest endpoint location 116 and the maximum endpoint location 114, and either time-domain correlation or time-domain correlation and phase correlation. Use everything. The manner of determining the maximum and lowest endpoint positions is described below. For temporal compression, the endpoint 110, determined by autocorrelation, simultaneously leads to the junction 108 (ie, if the audio is represented in samples, it has a higher sample number or index). Junction 108 defines a leading segment 118 of audio that leads the junction (ie, if the data is represented in samples, it has a lower sample number or index than the junction). Endpoint 110 defines a trailing segment 120 that turns off the endpoint (if the data is represented in samples, it has a higher sample number or index than the endpoint). The junction 108 and the endpoint 110 define the ends of the segment of audio, i.e. the target segment 122.

For data compression, the target segment is removed and in FIG. 2 the leading segment is joined, collided or joined with the trailing segment, preferably using crossfading at the junction, leaving the junction within the identified region 106. Therefore, the crossfaded junction "point" is characterized as the junction "region." The components of the joining workpieces remain in principle in the crossfade, which in the identified region 106 minimizes the audibility of data compression. In Fig. 2B, the compressed data is identified by reference numeral 102 '.

Throughout the various drawings, the same reference numerals apply to similar elements, and reference numerals with major notations are used to designate relevant but modified elements.                 

Referring to FIG. 2C, illustrating the data expansion, audio 124 is a transient 126 that becomes part of the audio 124 into a psychoacoustical postmasked region 128 that constitutes an “identified region”. Has In the case of data expansion, audio is analyzed and junction point 130 is also selected within the identified area. As described further below, if audio is represented in a buffer as a block of data, there is a minimum junction position and a maximum junction position within the block. Audio is analyzed forwards from the junction (higher sample number or exponent if data is represented in samples) and backwards (lower sample number or exponent if data is represented in samples) from the junction to locate the endpoint. do. This forward and backward search is performed to find the data before the junction that most closely resembles the data and after the junction that is appropriate for copying and repeating. More specifically, forward search is performed from junction 130 to the first maximum processing point position 132 and backward search is performed from the junction 130 to the second maximum processing point position 134. The two maximum processing point positions are separated by the same number of samples from junction 130 but are not necessary. As described further below, the two signal segments from the junction to the maximum search point position and the maximum endpoint position are each cross-correlated to find the correlation maximum. Cross correlation uses either time-domain correlation or time-domain correlation and phase correlation. In practice, the maximum endpoint position 135 is later (higher) by a time (or time-equivalent number of samples) equal to half the crossfade time than the second maximum processing point 134, as described further below. Sample number or exponent).                 

In contrast to the data compression of FIGS. 2A and 2B, endpoint 136 is determined by cross correlation and simultaneously precedes junction 130 (ie, lower sample number if audio is represented in samples). Or has an exponent). The junction 130 defines the leading segment 138 of audio that leads to the junction (ie, if the audio is represented in samples, it has a lower sample number or index than the junction). The endpoint 136 defines a trailing segment 140 that tracks the endpoint (ie, if the audio is represented in samples, it has a higher sample number or index than the endpoint). The junction point 130 and the endpoint 136 define both ends of the audio segment, ie the target segment 142. Therefore, the definitions of the junction point, the endpoint, the leading segment, the trailing segment, and the target segment are the same for data compression and data expansion. However, in the case of data expansion, the target segment is a portion (and thus is repeated) on both the leading and trailing segments, but in the case of data compression, the target segment is not a portion on either side (and therefore is deleted).

In FIG. 2D, the leading segment is combined with the target segment, preferably using crossfading at the junction (not shown in this figure), allowing the target segment to repeat in the resulting audio 124 '. In the case of data expansion, the endpoint 136 must be in the identified region 128 of the original audio (thus placing all target segments in the original audio in the identified region). The target segment (this portion is part of the leading segment) and the first notation 142 ′ of the junction point 130 remain within the masked area 128. The second notation 142 "of the target segment (this portion is part of the trailing segment) is after the junction 130 and extends outside the masked area 128, but need not be. However, outside the masked area This extension does not have any audible effect, since the target segment is associated with the trailing segment both in the original audio and in the time extended version.

Preferably, the target segment should not contain transients in order to avoid omission of transients in case of data compression or to avoid repetition of transients in case of data expansion. Therefore, the junction points and endpoints are either higher than the transient on both sides (i.e., if the audio is expressed in samples, have a lower sample number or exponent), or later (i.e., if the audio is expressed in samples) Sample number or exponent).

Another aspect of the invention is that the audibility of the junction can be further reduced by the selection of the crossfade shape and by varying the shape and duration of the crossfade in response to the audio signal. A more detailed description of crossfading is set forth below in connection with FIG. 10 and its description. Indeed, the crossfade time slightly affects the placement of the extreme positions of the junction and the endpoint, as further described below.

3A and 3B state examples of determining the lowest and maximum junction positions within a block of samples representing input audio for compression 3A and expansion 3B. The lowest (earliest) junction position has a lower sample number or index than the maximum (most recent) junction position. The minimum and maximum positions of the junctions with respect to the end of the block for data compression and data expansion vary in relation to the length of the crossfade used for the junction and the maximum length of the correlated processing region. Determination of the maximum length of the correlation processing region is further described with respect to FIG. 4. For time scale compression, the correlation processing region is the region of audio data after the junction used for autocorrelation processing to identify appropriate endpoints. For time scale expansion, there are two correlation processing regions, which are the same length but need not be, one before the junction and the other after the junction. They define two areas used in cross-correlation processing to determine the appropriate endpoints.

Every block of audio data has a lowest junction position and a maximum junction position. As shown in Fig. 3A, the lowest junction position with respect to the end of the block, which represents the earliest time in the case of compression, is limited to half the length of the crossfade because the audio data around the junction is crossfaded around the endpoint. . Similarly, for time scale compression, the maximum junction position relative to the end of the block representing the most recent time in the case of compression is limited to the maximum correlation processing length (the maximum endpoint position is the crossfade length rather than the end of the maximum processing length). Half as much as "early").

3B outlines the determination of the lowest and maximum junction locations for time axis expansion. The lowest junction position relative to the end of the block, which represents the earliest time for time-base expansion, is related to the maximum length of the correlation processing region in a manner similar to the determination of the maximum junction for time scale compression (the lowest endpoint position is the maximum mutual "Late" by half the crossfade length than the end of the relation process length). The maximum junction position with respect to the end of the block representing the most recent time for time scale expansion is related only to the maximum correlation processing length. This is because for time scale expansion, the data after the junction is used only for correlation processing and the endpoint is not located after the maximum junction position.

Although FIGS. 3A-3B are described with respect to blocks of input data, the same principles, including auditory events, as described further below, each subset of input data being processed (ie, a group of consecutive samples). Applies to setting the maximum and minimum endpoints in relation to

As shown in Figure 4, for the case of time scale compression, the area used for correlation processing is located after the junction. The junction and maximum processing point location define the length of the correlation processing region. The locations of the junction and maximum processing points shown in FIG. 4 are arbitrary examples. The lowest endpoint position refers to the lowest sample number or exponent value after the junction where the endpoint is located. Similarly, the maximum endpoint location refers to the maximum sample number or exponent value after the junction where the endpoint is located. The maximum endpoint position is "earlier" by half the crossfade length than the maximum processing point position. If the junction is selected, the lowest and maximum endpoint positions are used for the target segment and control the amount of data that is the default value assigned (available values are 7.5 and 25 msec respectively). Alternatively, the lowest and maximum endpoint positions are variable to dynamically vary depending on the audio content and / or the desired amount of time scaling (the lowest endpoint varies based on the desired time scale rate). For example, for a signal with a significant frequency component of 50 Hz and sampled at 44.1 kHz, a single period audio waveform has approximately 882 samples in length (or 20 msec). This indicates that the maximum endpoint position must be a target segment of sufficient length to contain at least one cycle of audio data. In any case, the maximum processing point may not be later than the end of the processing block (in this example, 4096 samples, or the end of the auditory event when the auditory event is considered, as described below). Not later than). Similarly, if the lowest endpoint position is selected at 7.5 msec after the junction and the processed audio includes a signal that typically selects an endpoint close to the lowest endpoint position, the maximum percentage of time scaling will depend on the length of each input data block. . For example, if the input data block size is 4096 samples (or approximately 93 msec at 44.1 kHz sample rate), then the lowest target segment length of 7.5 msec is 7.5 / 93 = 8% maximum time scale, even if the lowest endpoint position is selected. Will be rate. The lowest endpoint position for time scale compression is set to 7.5 msec (331 samples at 44.1 kHz) for fluctuation rates below 7% and is set as follows:

Lowest endpoint position = ((time_scale_rate-1.0) * block size);

Here, time_scale_rate is> 1.0 for time scale compression (1.10 = 10% increase in reproduction ratio), and the block size is currently 4096 samples at 44.1 kHz. These examples show the advantage of varying the lowest and maximum endpoint positions depending on the audio content and the desired time scale percentage. In any case, the lowest endpoint should not be too large or near the maximum endpoint to excessively limit the search area.

Another aspect of the invention is to match the signal waveforms at the junction and endpoint to reduce the need to rely on masking or inaudibility using a contrasting technique to further reduce the possibility of an audible junction. Matching techniques that constitute another aspect of the invention are searched to match both amplitude and phase of the waveforms connected at the junction. This in turn entails interrelationships, as described above, which is also an aspect of the invention. Correlation includes compensation for variations in ear sensitivity over frequency.

As described in connection with Figures 2A-2D, the data compression or expansion technique used in aspects of the present invention deletes or repeats the audio section. In the first alternative described above, the junction position is selected using general, predefined system parameters based on the length of the crossfade or a predetermined distance of the junction position from signal components such as transients and / or taking into account other signal conditions. do. More detailed analysis of the audio (e.g., correlation) is performed around some arbitrary junction to determine the endpoint.

According to a second alternative, the junction and endpoint positions are chosen in a more signal dependent manner. The windowed data around the series of trailing junction locations are correlated to the data in the correlation processing region to select the associated trailing endpoint location. The trailing junction location, which has a strong correlation between all trailing junction positions, is selected as the final junction and the trailing endpoint is generally located at the position of the strongest correlation. Although, in principle, the trailing junctions can be spaced more widely to reduce processing complexity, even if the spacing between trailing junctions is only one sample. The width of the crossfade region is an appropriate increase in the sample junctions, as described below. This alternative method of selecting junction and endpoint locations applies to data compression and data expansion processing. Although this alternative of selecting junction and endpoint locations is described in more detail below in connection with aspects of the invention using auditory scene analysis, it may also be used in the first embodiment of the invention, using psychoacoustic analysis. .

Psychoacoustic Analysis Example

A flowchart describing a single channel or multichannel time-scaling and / or pitch-scaling process in accordance with aspects of the invention involving psychoacoustic analysis is illustrated in FIG. 5. A flowchart describing a single channel or multichannel time-scaling and / or pitch-scaling process in accordance with aspects of the invention involving psychoacoustic analysis and auditory event analysis is illustrated in FIG. 17, which is described below. Other aspects of the invention form part or variations of the FIGS. 5 and 17 processes. The processes are used to perform real time pitch scaling and non real time pitch and time scaling. The low latency-scaling process cannot operate effectively in real time, because the input audio signal must be buffered in order to reproduce the input audio signal at different rates, resulting in a buffer underfloor or overfloor-the buffer receives the input data. Unlike that, it is emptied at different rates.

Data Entry 202 (Figure 5)

Referring to Fig. 5, a first step, decision step 202 (" data entry? &Quot;) determines whether digitized input audio data is available for data compression or data expansion processing. The source of data is input data in a computer file or block, which is stored in a real time input buffer. If data is available, one of the data blocks of N time synchronous samples, representing a time-matched segment, the respective input channels are subjected to data compression or data expansion processing by step 204 (get N samples for each channel). Accumulated in blocks (the number of channels is one or more). The number N of input data samples used by the process is fixed at any suitable number of samples, thus dividing the input data into blocks. In principle, the processed audio is digital or analog and does not need to be divided into blocks.

FIG. 5 is discussed in connection with a practical embodiment of the aspect of the invention in which input data for each audio channel is data compressed or data expanded processed into blocks of 4096 samples, which corresponds to about 93 msec input audio at a sampling rate of 44.1 kHz. . It is to be understood that aspects of the invention are not limited to such embodiments. As noted above, the principles of the various aspects of the invention do not require arranging audio into sample blocks, and if they do, do not require providing blocks of constant length. However, to minimize complexity, 4096 samples of fixed block length are useful for three main reasons. First, it provides a low enough delay that is acceptable for real time processing applications. Secondly, 2n (power of two) samples, which is useful for fast Fourier transform (FFT) analysis. Third, it provides an appropriately large window size for performing useful psychoacoustic analysis of the input signal.                 

In the discussion below, the input signal is assumed to be data having an amplitude value in the range [-1, +1].

Psychoacoustic analysis 206 (Fig. 5)

Following input data blocking, psychoacoustic analysis 206 (“execute psychoacoustic analysis for each block of input data”) is performed on the block of input data for each channel. In the case of multiple channels, psychoacoustic analysis 206 and subsequent steps are performed on a channel-by-channel basis in parallel or in succession for all channels (provide appropriate storage of each channel's data and respective analysis). Although parallel processing requires larger processing power, it is desirable for real-time applications. The description of FIG. 5 assumes that the channels are processed in parallel.

A more detailed description of step 206 is illustrated in FIG. 6. The analysis 206 identifies one or more regions in the data block for each channel that meets the psychoacoustic criteria (or, for some signal conditions, does not identify those regions in the block) and potentially within each identified region. Or determine a temporary junction point location. If there is only one channel, subsequent step 210 (“select common junction”) is skipped and the temporary junction location is used in one of the areas identified in step 206 (preferably, the “best” area in the block Selected according to hierarchical criteria). For the multichannel case, step 210 re-evaluates the identified regions, identifies common overlapped regions, and selects the best common junctions in those common overlapped regions, which junctions are identified in the psychoacoustic analysis stage 206. Junction location, but not necessarily.                 

The use of psychoacoustic analysis to minimize audible artifacts in time and / or pitch scaling of audio is one aspect of the present invention. Psychoacoustic analysis may be used to identify segments of audio that suppress or minimize artifacts arising from joining one or more of the four criteria described above, or combining waveforms therein or performing time and / or pitch scaling therein. Includes the application of psychoacoustic criteria.

In the FIG. 5 process described herein, there are a number of psychoacoustically identified areas in the block, each with a temporary junction. However, in another embodiment, in the case of a single channel, at most one psychoacoustically identified region is selected for data compression or data expansion processing in each block of input data, and in the case of multiple channels, In each set of time-matched blocks of data (one block for each channel), at most one overlap of psychoacoustically identified regions is selected for data compression or expansion processing. Preferably, the psychoacoustically " best " (e.g., according to a hierarchy such as criteria described in the text) identified region or overlap of the identified regions are each identified in a block or blocks of input data. When there is an overlap of the identified region or identified region, it is selected.

Alternatively, the overlap of one or more identified regions or identified regions is selected for processing in each block or block set of time coincident input data, in which case the selections are preferably psychoacoustically the best (eg For example, according to the hierarchy as described in the text), or alternatively all identified events are selected.                 

Instead of placing a temporary junction in every identified region, in the case of a single channel, the junction (in this case, the junction is not "provisional" and is the actual junction) is identified after the region is selected for processing. Are located in a closed area. In the case of multiple channels, temporary junctions are located in the identified area only after it is determined whether it is overlapping.

In principle, identification of temporary junctions is unnecessary, as long as it is desirable to select a common junction in the overlap region when in multiple channels, which common junction is typically different from each temporary junction of the individual channels. However, as in the implementation description, the identification of temporary junctions is useful because it allows operation with single channels or multiple channels that require a temporary junction (which is the actual junction), in which case the temporary junctions are ignored.

6 is an operational flowchart of the psychoacoustic analysis process 206 of FIG. The psychoacoustic analysis process 206 consists of five general processing substeps. The first four are layered such that an audio region that meets the first substep or the first criterion has a maximum likelihood of splicing (or other time shifting or pitch shifting processing) within the inaudible or least audible region. With arranged psychoacoustic mood analysis substeps, subsequent step criteria have less likelihood of gradual bonding within the inaudible or minimally audible region.

The psychoacoustic reference analysis of each substep uses a psychoacoustic subblock having a size of an input data block of 1.64. In this example, the psychoacoustic subblocks are approximately 1.5 msec (or 74 samples at 44.1. KHz) as shown in FIG. 8. Since the size of psychoacoustic subblocks does not need to be 1.5 msec, this size is chosen according to the actual implementation because it meets real-time processing requirements (large subblock size requires less psychoacoustic processing overhead) and psychoacoustic criteria. This is because they provide a good trade-off between the resolution of the segments being made (small subblocks provide more detailed information about the location of those segments). In principle, the psychoacoustic subblock size need not be the same for each type of psychoacoustic reference analysis, but in practical embodiments, for ease of implementation, this is desirable.

Transient Detection 206-1 (Figure 6)

Process 206-1 analyzes the data block for each channel and determines the location of the audio signal transient, if any. The transient transient information is used in masking analysis and in selecting the location of the temporary junction (substep in the psychoacoustic analysis process of this example). As discussed above, transients are known to introduce instant masking (hiding audio information on both sides before and after the occurrence of transients).

As shown in the flowchart of Fig. 7, the transient detection substep 206-1 to the first sub-substep 206-1a ("high-pass filtering the input full-bandwidth audio") is filtering the input data block (time function Process block content accordingly). The input block data is high pass filtered, for example with a second order IIR high pass filter having a 3 dB cutoff frequency of approximately 8 kHz. The cutoff frequency and filter characteristics are not important. The filtered data according to the original unfiltered data is then used for subsequent transient analysis. The use of bilateral full-range and high-pass filtered data improves the ability to identify transients even in complex materials, such as music. "Full band" data is a band that is restricted by filtering, for example, very high and very low frequencies. The data is also high pass filtered by one or more additional filters having different cutoff frequencies. The high frequency transient components of the signal have an amplitude below the more powerful low frequency components but are still audible to the listener. Filtering the input data isolates high frequency transients and makes them easier to identify.

In the next sub-substep 206-1b (“Place the maximum absolute samples in the full bandwidth and the filtered audio subblock”), the full range and the filtered input blocks place the maximum absolute samples in the full bandwidth and filtered audio subblocks. It is processed into a subblock of approximately 1.5 msec (or 64 samples at 44.1 kHz) as illustrated in FIG. 8 to locate in the blocks.

The third sub-substep 206-1c of the transient detection substep 206-1 ("flatten the full bandwidth and the filtered peak data with the low pass filter") is a low pass of the maximum absolute data values contained in each 64-sample subblock. Perform filtering or leak averaging (processing data values as a function of time). This processing is performed to smooth the maximum absolute data and provide a general indication of the average peak values of the input block to which the actual sub-block maximum absolute data values can be compared.

The fourth sub-substep 206-1d of the transient detection processing 206-1 ("compare the scaled peak absolute value of each full bandwidth and the filtered subblock and the flattened data") is smoothed with the peak of each subblock. The corresponding numbers in the array of moving average peak values are compared to determine if there is a transient. While there are a number of ways to compare these two means, the approach set out below allows tuning of the comparison by the use of a scaling factor that has been set to perform best as determined by analyzing a wide range of audio signals. Allow.

In decision sub-step 206-1e (" scaled data > flattened moving average peak value? &Quot;), the peak value of the kth subblock is increased by the scaling value and calculated to calculate the kth value of the flattened moving average peak value. Is compared with If the scaled peak value of the subblock is greater than the moving average value, it is flagged as having a transient. The presence and location of the transients in the subblocks are stored for continuous processing. This operation is performed on both unfiltered and filtered data. A subblock flagged as a transient or a string of adjacent subblocks flagged as a transient indicates the presence and location of the transient. This information may, for example, be directed to other parts of the process to indicate where premasking and postmasking are provided by the transients and where data compression or expansion should be avoided in order to avoid disturbing the transients. (See, eg, substep 310 of FIG. 6).

Following transient detection, several correction checks are made in sub-substeps 206-1f ("Perform a correction check to eliminate transients") to determine whether the transient flag for the 64-sample subblock should be cleared. (Reset from TRUE to FALSE). This test is performed to reduce false transient detection. First, when the full range or high frequency peak values fall below the minimum peak value, the transient is then canceled (to remove low level transients that provide little or no temporary masking). Second, if the peak of the subblock triggers a transient, but is also not significantly greater than the previous subblock that triggered the transient flag, then the transient of the current subblock is erased. This reduces the smearing of information about the location of the transient. For each audio channel, the number of transients and their locations are stored for later use in the psychoacoustic analysis phase.

The invention is not limited to the specific transient detection disclosed. Other suitable transient detection methods may be used.

Audible Critical Analysis 206-2 (Figure 6)

Referring back to FIG. 6, a second step 206-2, audible threshold analysis of the psychoacoustic analysis process determines the location and duration of an audio segment with a sufficiently low signal strength that can be expected to be at or below an audible threshold. . As discussed above, these audio segments are important because the workpieces introduced by time scaling and pitch shifting are not easy to be audible in such areas.

As discussed above, the audible threshold is a function of frequency (low and high frequencies are less audible than intermediate frequencies). To minimize the processing for real-time processing applications, the audible threshold model for analysis estimates a uniform audible threshold (where the audible threshold of the most sensitive range of frequencies applies to all frequencies). This conservative estimation allows for a higher playback volume than would be expected by an audible response curve and reduces the requirement to run frequency dependent processing on the input data before low energy processing.

The audible threshold analysis step processes the unfiltered audio, also processes the input of approximately 1.5 msec subblock (64 samples for 44.1 kHz input data) and uses the same flattened moving average algorithm described above. Following this algorithm, the flattened moving average value for each subblock is compared with a threshold to determine if the subblock is flagged as an inaudible subblock. The location and duration of each sub-audible critical segment in the input block is stored for later use in this analysis step. The string of adjacent flagged subblocks of sufficient length constitutes an identified region that meets the lower audible threshold psychoacoustic criteria. The minimum length (time period) is long enough that the identified region is a useful location for the junction or both the junction and the endpoint. If only one region is to be identified in the input block, it is useful to identify only the longest contiguous string of flagged subblocks.

High Frequency Analysis 206-3 (Figure 6)

The third substep 206-3, the high frequency analysis step, determines the position and length of the audio segments that predominantly comprise the high frequency audio content. High frequency segments of approximately 10-12 kHz and above are important in psychoacoustic analysis, because in a quiet place the audible threshold rapidly increases to approximately 10-12 kHz and above, and ears are less sensitive to significant high frequency waveform discontinuities than significant low frequency waveform discontinuities. There are a number of possible methods for determining whether an audio signal consists primarily of high frequency energy, but the method disclosed herein provides good detection results and minimizes computational requirements. Nevertheless, other methods may be used. The disclosed method does not classify an area as being high frequency if it contains both strong low frequency content and high frequency content. This is because low frequency content is easier to generate audible artifacts when data compression or data expansion is processed.

The high frequency analysis step also processes the input block of 64 sample subblocks and uses the zero-crossing information of each subblock to determine whether it contains significant high frequency data. The zero crossing threshold (ie, how many zero crossings exist in the block before it is labeled with a high frequency audio block) is set to correspond to a frequency in the range of approximately 10-12 kHz. That is, if the subblock contains the minimum number of zero crossings corresponding to a signal in the range of about 10 to 12 kHz (the 10 kHz signal has 29 zero crossings in a 64-sample subblock with a 44.1 kHz sampling frequency), high frequency audio content Is flagged as including. As in the case of an audible threshold analysis, a string of adjacent flagged subblocks of sufficient length constitutes an identified region that meets high frequency content psychoacoustic criteria. The minimum length (time period) is set to ensure that the identified area is long enough as to be a useful point for the junction or both the junction and the endpoint. If only one region is to be identified in the input block, it is useful to identify only the longest contiguous string of flagged subblocks.

Audio Level Analysis 206-4 (Figure 6)

A fourth substep 206-4 of the psychoacoustic analysis process, audio data block level analysis analyzes the input data block and determines the position on the audio segments of the lowest signal strength (amplitude) in the input data block. If the current input block does not contain any psychoacoustic masking events that may be used during processing (e.g., if the input is a ready signal that does not contain any transients or audio segments below the audible threshold), the audio level analysis Information is used. In this case, the time-scaling processing is based on the theoretical interpretation of lowering the level segment of the audio result in lower level or inaudible joint artifacts, if any, such as the lowest level or quietest segments of the audio of the input block. Is preferred. A simple example using 450 Hz tone (sine wave) is illustrated in FIG. 9. The tone signal illustrated in FIG. 9 does not include any transient below the audible threshold or high frequency content. However, the middle portion of the signal is 6 dB lower in level than the early and late sections of the signal in the block. Focusing attention on the quieter middle section than the noisy termination section is considered to minimize audible data compression or data expansion processing artifacts.

When the input audio block splits into any number of variable-length audio level segments, the audio data block level analysis determines the first part of the signal in each block to find one part or two adjacent parts that are quieter than the remaining part (s). It has been found suitable to divide the block into three equal parts in order to be executed between the first, second and final third parts. In contrast, in a similar manner to the subblock analysis of blocks against the lower audible threshold and high frequency criteria, the subblocks are ranked according to their peak level with the longest adjacent stream of the quietest subblocks that make up the quietest portion of the block. Is numbered. In either case, this substep provides as the output the identified region that meets the quietest region psychoacoustic criteria. Except for non-ideal signal conditions, such as constant amplitude signals in a block during analysis, this last psychoacoustic analysis, generally the audio level, always provides an "last resort" identified area. As in the case of the substeps just disclosed, the minimum length (time period) is set to ensure that the identified area is long enough as to be a useful location for the junction or for both the junction and the endpoint.

Temporary junction and crossfade parameter settings 206-5 (Figure 6)

In the psychoacoustic analysis process of FIG. 6, the final substeps 206-5 (“Temporary Junction and Crossfade Parameter Setting”) use the information collected from the previous steps to select the psychoacoustically best identified region in the input block. Set the junction and crossfade length in the identified area.

Crossfade Parameter Settings

As mentioned above, crossfading is used to minimize audible workpieces. 10 conceptually illustrates a method of applying crossfading. The resulting crossfade straddles the junction where the waveforms are associated together. In FIG. 10, the dashed line starting before the junction shows a non-linear downward fade from the maximum amplitude to the minimum amplitude applied to the signal waveform, half down at the junction. The fade across the junction is times t1 to t2. The dashed line starting before the endpoint represents the complementary nonlinear upward fade from the minimum amplitude to the maximum amplitude applied to the signal waveform, which is half as high at the endpoint. The fade across the endpoints is time t3 to t4. The fade up and fade down are symmetric and add up to unity (Hanning and Kaiser-Bessel windows have their properties; therefore, if crossfades are formed in the manner of such windows, this requirement will be met). The time period of t1 to t2 is the same as the time period of t3 to t4. In this time compression example, it is desirable to discard the data between the junction and the endpoint (illustrated by horizontal lines). This is accomplished by discarding data between the sample representing t2 and the sample representing t3. The junction and endpoints are then placed (conceptually) in addition to each other such that the data of t1 to t2 and t3 to t4 are summed together to form a crossfade of complementary fade up and fade down characteristics.

In general, longer crossfades mask audible workpieces of a better bond than shorter crossfades. However, the length of the crossfade is limited to the fixed size of the input data block. Longer crossfades can also be used for time scaling processing. This means that the crossfade is limited by the block boundary (and / or by the auditory event boundary, when the auditory event is considered) and the data before and after the current data block (and / or the current auditor event, the auditor event) is considered. Is not available for use in data compression or data expansion processing and crossfading. However, the transient masking properties can be used to shorten the length of the crossfade, because some or all of the audible workpiece resulting from the shorter crossfade is masked by the transient.                 

Although the crossfade length varies in response to audio content, the appropriate default crossfade length is 10 msec, because it introduces the minimum audible splicing artifact for a wide range of materials. Transient postmasking and premasking allow the crossfade length to be set somewhat shorter, eg, 5 msec. However, when auditory events are considered, crossfades longer than 10 msec may be used under certain conditions.

Temporary Junction Settings

If the transient exists as determined by substep 206-1 of Figure 6, the temporary junction is located in the block in the temporary masking area before and after the transient, depending on the transient location of the block and whether time expansion or compression processing is performed. To avoid repeating or smearing the transients (ie, preferably, no portion of the transients should be in the crossfade). Transient information is also used to determine the crossfade length. If there is more than one transient masking area available because there is more than one transient, the best masking area (in consideration of, for example, its location in the block, its length and its strength) is identified by which the temporary junction is located. Selected area.

If no signal transients exist, substep 206-5 of setting temporary junction and crossfade parameters analyzes audible threshold segments, high frequencies, and audio levels, and substeps in searching for psychoacoustically identified regions. The results of 206-2, 206-3 and 206-4 are analyzed to locate the temporary junction. If more than one lower level is below or where audible critical segments are present, the temporary junction is set within one such segment or the best such segment (consider, for example, its location and length in the block). If there are no lower audible critical segments, the step retrieves the high frequency segments of the data block and establishes one such segment or the best such segment, taking into account, for example, a temporary junction within the block and its length. If no high frequency segments are found, then the step then searches for any lower level audio segments and establishes a temporary junction in one or the best such segment (consider, for example, its position and length in the block). As a result, there is only one identified region where the temporary junction is located in each input block. As mentioned above, in rare cases, there are no segments in the block that meet the psychoacoustic criteria, in which case there are no temporary junctions in the block.

Alternatively, as mentioned above prior to the discussion of the psychoacoustic analysis description, instead of selecting only one region in each input block that meets the psychoacoustic criteria and (optionally) placing a temporary junction in the identified region, In this case, one or more regions that meet the psychoacoustic criteria are selected and (optionally) temporary junctions are located in each of them. There are several ways to accomplish this. For example, one or more additionally identified regions in a particular input block that have a lower grade in the psychoacoustic layer, even though the region is identified to meet one of the higher grade psychoacoustic criteria and the temporary junction is (optionally) located therein. Are selected and temporary junctions are located on each of them. Another way is that even if multiple regions that meet the same psychoacoustic criteria are found in a particular block, if each such additional identified region is available (eg, its length and position in the block), those regions One or more of them are selected (and temporary junctions are located in each). Another way is to select every identified area and optionally place temporary junctions in each, whether or not there are other identified areas in the subblock and whether the psychoacoustic criteria are met by the identified area. will be. Multiple identified regions in each block are useful for finding common junctions between multiple channels, as described further below.

Therefore, the psychoacoustic analysis process of FIG. 6 (step 206 of FIG. 5) identifies the regions within the input block according to the psychoacoustic criteria and within each of those regions, and (optionally) locates a temporary junction. The process provides for identification of temporary junctions (eg, masking as a result of transients, audible thresholds, high frequencies, lowest audio levels) and the criteria used to identify the number and location of transients in each input block. All of which are useful for determining common junctions when there are multiple channels and for other purposes, as further described below.

Common multichannel junction selection 210 (FIG. 5)

As mentioned above, the psychoacoustic analysis process of FIG. 6 is applied to the input block of every channel. Referring back to FIG. 5, if more than one audio channel is to be processed, temporary junctions, as determined by decision step 208 (“channel number> 1?”), May be placed on multiple channels, even if optionally located in step 206. Not easy to match (eg some or all channels contain audio content not related to other channels). The next step 210 (“common junction selection”) uses the information provided by psychoacoustic analysis step 206 to identify overlapping identified regions in the multiple channels so that a common junction is selected in each of the time-matched blocks of the multiple channels.

Although, alternatively, a common junction, such as the best overall junction, is selected between one or more temporary junctions in each channel selectively determined by step 206 of FIG. 5, potentially in identified regions that overlap across channels. It is desirable to select a more optimal common junction, which is different from all temporary junctions determined by step 206 of FIG.

As a result, the identified regions of each channel are ANDed together to yield a common overlapping segment. Note that in some cases there are no common overlapping segments, and in other cases there is one or more common overlapping segments when an alternative is used to identify one or more psychoacoustic regions in the block. The identified regions of the different channels do not exactly match, but it is sufficient that they overlap so that a common junction location between the channels is chosen and within the region identified in each channel. The multichannel junction processing selection step selects only one common junction for each channel and does not modify or change the location or content of the data itself.

For example, the rank of the overlapped regions according to the hierarchy of psychoacoustic criteria is used to select one or more best overlapped regions for processing in the case of multiple overlapped regions. Although the identified regions of different channels do not have to result from the same psychoacoustic criteria, the distribution of the reference type between the channels affects the quality of the overlapping regions (the highest quality is least when processing is performed in the overlapping regions). Audible). The quality of the overlapped regions is graded taking into account the psychoacoustic criteria that are met in each channel. For example, overlapping regions where the identified regions of each channel meet the "postmasking as a result of transient" criteria are ranked highest. Every channel except one meets the "mask masking as a result of transients" criteria, and the overlapping areas where the other channels meet the "low audible threshold" criteria become the next class. The details of the rating scheme are not important.

In contrast, a common region across multiple channels is selected for processing even if the psychoacoustically identified regions overlap only for some but not all of the channels. In that case, failure to meet psychoacoustic criteria in one or more channels is likely to result in minimal objectionable audible artifacts. For example, cross-channel masking does not need some channels to have a common overlapping identified area; For example, masking signals of different channels can be accepted to perform the junction in areas where the junction is unacceptable if the channel is separately audible.

Another variation in selecting a common junction is to select a temporary junction of one of the channels as a common junction based on the determination that one of the individual temporary junctions causes a minimal objection workpiece.

Skipping

As part of step 210 (FIG. 5), the rank of the overlapped region is also used to determine if processing in the particular overlapped region has been skipped. For example, overlapping areas where all identified areas meet only the lowest rank criterion, the "quietest part" criterion, are skipped. In certain cases, it is not possible to identify a common overlap of the identified regions between channels for a particular set of time-matched input blocks, in which case a skip flag is set for the set of blocks as part of step 210. There are also other factors for setting the skip flag. For example, if there is a large number of transients in one or more channels and there is insufficient space for data compression or data expansion processing without deleting or repeating the transients or otherwise insufficient space for processing, the skip flag is set. do.

The common junction (and common endpoint) between time-matched blocks is preferably selected when deleting or repeating audio segments to maintain phase alignment between multiple channels. This is especially important for the two-channel processing suggested by psychoacoustic studies, that shifts in stereo images can be perceived as little as 10 microseconds difference between the two channels, which is less than one sample at a sample rating of 44.1 kHz. Corresponds to Phase alignment is also important in the case of surround-encoded material. The phase relationship of the surround-encoded stereo channel will be maintained or the decoded signal will be degraded.

Nevertheless, in some cases, processing multichannel data such that not all channels are perfectly sample aligned (ie, processing channels with independent junction and endpoint locations unaligned for at least some of the channels). It is easy to do). For example, align junctions and endpoints of L, C, and R (left, center, and right) channels (for movie or DVD signals) and process individually aligned LS and RS (left surround and right surround) channels. Useful for Information can be shared between the processing steps of the process of FIG. 5 so that some phase difference in processing can be adjusted block-to-block to minimize the difference.

Example of Multichannel Junction Selection

FIG. 11 shows a detailed description of the multi-channel junction selection analysis step 210 of FIG. 5. The first processing step 210-1 ("analyze the block for each channel to locate the psychoacoustically identified regions") uses each channel to locate the regions that were identified using psychoacoustic analysis as described above. Analyze the input block for. Processing step 210-2 (“group the overlapping identified regions”) groups overlapping portions of the identified regions (AND ANDs the identified regions across the channels together). Processing step 210-3 (" select a common junction based on the overlapping identified regions as a priority ... ") selects a common junction between the channels. In the case of multiple overlapping identified regions, the hierarchy of criteria associated with each overlapping identified regions is preferably used to rank the overlap of the identified regions as mentioned above according to the psychoacoustic layer. Cross-channel masking effects can also be considered in ranking multiple overlaps of identified regions. Step 210-3 also takes into account whether there are multiple transients in each channel, proximity to each other and whether time compression or expansion is performed. The type of processing (compression or expansion) is also important in that the endpoint is positioned before or after the junction (described in connection with FIGS. 2A-D).

12 shows an example of selecting a common multichannel junction for the case of time scale compression using the areas identified in psychoacoustic processing of an individual channel as suitable for performing data compression or data expansion processing. Channels 1 and 3 of FIG. 12 include transients that provide a significant amount of temporary post masking as shown in the diagram. The audio of channel 2 in FIG. 12 includes audio with quieter portions used for data compression or data expansion processing and is included approximately at the end of the audio block for channel 2. The audio on channel 4 includes the portion below the audible threshold and is approximately located in the first 3300 samples of the data block. The lower legend of FIG. 12 shows overlapping identified areas that provide minimal audibility to the overall good area where data compression or data expansion processing can be performed in each channel. Preferably, the common junction is positioned after the beginning of the common overlapping portion, as shown in FIG. 12, to prevent the crossfade from transitioning between the identified regions and to maximize the size of the potential target segment.

Set endpoint location

Referring back to FIG. 11, once the common junction has been identified in step 210-3, processing step 210-4 (“minimum and maximum endpoint positioning ..”) is the time scaling rate (ie, the desired rate of data compression or expansion). Set minimum and maximum endpoint positions and maintain the correlated processing region within the overlapping portion of the identified region. Alternatively, instead of considering the time scaling rate and the identified area size before correlation, before the target segment length is known, the minimum and maximum endpoint positions are default values, such as 7.5 and 25 msec, respectively, as mentioned above. Is determined. Step 210-4 outputs a common multichannel junction for all channels along the minimum and maximum endpoint locations (illustrated in FIG. 12). Step 210-4 also outputs the crossfade parameter information provided by substep 206-5 (Fig. 6) of step 206 (Fig. 5). The maximum endpoint position is important when there are multiple cross-channel or cross-channel transients. The junction is set such that data compression or data expansion processing occurs between transients. When setting the endpoint location (and ultimately, the target segment length, which is determined by the junction location, endpoint location, and crossfade length), other transients may be associated with data compression or data expansion processing on the same or different channels. It is necessary to consider.

Block Processing Decision 212 (FIG. 5)

Referring back to FIG. 5, the next step in processing is input block processing decision step 212 (" skip based on complexity? &Quot;). This step checks to determine if the processing skip flag is set by step 210. If so, the current block of data is not processed.

Correlation Processing 214 (Figure 5)

If it is determined that the current input data block has been processed, then two types of correlation processing are provided for each such data block, as shown in correlation step 214 of FIG. Correlation processing of the time domain information of the data blocks is provided by substeps 214-1 ("weighting") and 214-2 ("correlation processing of time block data of each block"). Correlation processing of the phase information of the input signal is provided by substeps 214-3 ("phase computation of each block") and 214-4 (correlation processing of phase data of each block "). Using phased and time-domain information that is accurate provides higher quality time scaling results for voice to complex musical signal ranges than using time-domain information alone, whereas, if reduced performance is deemed to be acceptable, time-domain Only information will be processed and used The details of correlation processing are set out below after explanation of some key principles.

As discussed and illustrated in Figures 2A-D, time scaling in accordance with aspects of the present invention works while ignoring or repeating segments of input blocks. If, according to the first alternative embodiment, the junction and endpoint locations are selected, for a given junction, the endpoint will maintain maximum signal periodicity, so that the audible workpieces will be reduced. An example of well chosen junction and endpoint processing positioning to maximize periodicity is shown in FIG. 13. The signal shown in FIG. 13 is time-domain information of a very intermittent part of the speech signal.

Once the junction is determined, a method is needed to determine the appropriate endpoint location. In so doing, it is desirable to weight the audio in such a way that it has some relationship to the human audible and then carry out the interrelationship. The correlation of the time-domain amplitude data of the signal provides an easy-to-use assessment of signal periodicity, which is useful when choosing endpoint locations. Although weighting and correlation can be achieved in the time domain, doing so in the frequency domain is computationally effective. Fast Fourier Changes (FFT) can be used to effectively compute an estimate of the power spectrum of a signal related to the Fourier transform of the signal's interrelationship. For example, New York, Cambridge University Press, William H. See "Correlation and Autocorrelation Using the FFT" in Numerical Recipes in C, The Art of Scientific Computing, et al.

The appropriate endpoint location is determined using the correlation data of the phase and time-domain information of the input data block. For temporal compression, the autocorrelation of the audio between the junction location and the maximum processing point is used (see Figures 2A, 3A, 4). The autocorrelation is used because it provides a measure of the periodicity of the data and helps determine how to shift the integer number of cycles of the main frequency component of the audio. For time expansion, the cross correlation of the data before and after the junction location is computed to evaluate the periodicity of the data and repeated to increase the duration of the audio (see FIGS. 2C, 3B, 4).

The correlation (auto-correlation for time compression or cross-correlation for time expansion) starts at the junction and, as returned by the previous process, the maximum processing length (the maximum processing length is the maximum endpoint position and crossfade after the endpoint). If so, then it is calculated by terminating at either half of the crossfade length) or the global maximum processing length (default maximum processing length).

The frequency weighted correlation of the time-domain data is computed in substep 214-1 for each input channel data block. The frequency weighting is performed to focus the correlation processing on the most sensitive frequency range of the human audible and instead of filtering the time-domain data prior to the correlation processing. If a number of different weighted acoustic curves are available, one suitable curve is a modified B-weighted acoustic curve. The modified curve is a standard B-weighted curve calculated using the following formula:

Lower frequency components (approximately 97 Hz or less) are set to 0.5.

Sub-frequency signal components, although inaudible, produce an audible high frequency workpiece when bonded. Therefore, it is desirable to weight more low frequency components rather than weighting the standard undeformed B-weighting curve.

Following the weighting step, in process 214-2, the time-domain correlation is calculated as follows:

1) form an L-point sequence (squared of 2) by increasing x (n) to zero,

2) compute the L-point FFT of x (n),

3) multiply the complex FFT result by its conjugate, and

4) Calculate L-point inverse FFT.

Where x (n) is digitized time-domain data contained in an input data block representing an audio sample in the correlation processing region, where n indicates a sample number or exponent and length L is greater than the number of samples in the processing. Is the power of two.                 

As mentioned above, weighting and correlation are effectively achieved by multiplying the signals in the frequency domain by a weighted acoustic curve to correlate them. In that case, the FFT is applied before weighting and correlation, and the weighting is applied during correlation and then the inverse FFT is applied. Even if performed in the time domain or frequency domain, the correlation is then stored for processing by the next step.

As shown in Fig. 5, the instantaneous phase of the data block of each input channel is calculated in substep 214-3, and the instantaneous phase is defined as follows.

phase (n) = arctan (imag (analytic (x (n)) / real (analytic (x (n)))

Where x (n) is digitized time-domain data contained in an input data block representing audio samples in the correlation processing region, where n indicates a sample number or exponent.

The function analytic () contains a complex analytic version of x (n). The analytical signal can be generated by taking a Hilbert transform of x (n) and generating a composite signal, where the real part of the signal is x (n) and the imaginary part of the signal is a Hilbert transform of x (n). In this implementation, the analysis signal takes the FFT of the input signal x (n), zeros the negative frequency components of the frequency domain signal and then performs an inverse FFT. The result is a composite analysis signal. The phase of x (n) is computed by taking the arc tangent of the imaginary part of the analysis signal by dividing it by the real part of the analysis signal. The instantaneous phase of the analysis signal of x (n) is used because it contains key information relating to the local response of the signal, which aids in the analysis of the periodicity of x (n).                 

14 shows the time-domain signal, the speech signal superimposed on x (n), and the instantaneous phase of the radian. A description of the "momentary phase" is given in Sam Shanmugam, John and Wiley & Sons' New York (1979), pages 278-280, Section 6.4.1 ("Analog Modulated Signals") of the Digital and Analog Communication System. Considering both phase and time domain characteristics, additional information is obtained to improve the ability to match waveforms at the junction. Minimizing phase distortion at the junction tends to reduce undesirable workpieces.

The time-domain signal x (n) is related to the instantaneous phase of the analysis signal of x (n) as follows:

Negative running zero crossing of x (n) = + π / 2 phase

Positive running zero crossing of x (n) = -π / 2 phase

Interval maximum of x (n) = 0 phase

Interval minimum of x (n) = ± π phase

In addition to this mapping, the midpoints provide information independent of the x (n) amplitude. Subsequent to the calculation of the phase for the data of each channel, the correlation for the phase information of each channel is computed in step 214-4 and stored for later processing.

Multiple Correlation Processing 216 (FIGS. 5, 15, 16)

If the phase and time-domain correlations have been computed for the data blocks of each input channel, then the correlation-processing step (216 of FIG. 5) ("process multiple correlations to determine the crossfade position"), in FIG. As shown in more detail, the relationships are processed. FIG. 15 shows the phase and time-domain correlations for five (left, center, right, left surround and right surround) input channels containing music. Conceptually represented in FIG. 16, the correlation processing step accepts the phase and time-domain correlations for each channel as inputs, multiplies each by a weighted value, and then inputs all of the time-domain and phase correlation information of all input channels. Sum them to form a single correlation function that represents That is, the FIG. 16 arrangement considers a super-correlation function that sums together 10 different correlations to yield a single correlation. The waveform of FIG. 16 shows the maximum correlation value, which constitutes the desired common endpoint at about sample 500, which is between the minimum and maximum endpoint positions. The junction is at sample 0 in this example. Weights are chosen such that certain channels or correlation types (eg time-domain versus phase) play a dominant role in the overall multichannel analysis. Although very simple, the weighting function available is a unit of measure of sound relative between channels. Such weighting minimizes the contribution of signals that are so low that they can be ignored. Other weighting functions are also possible. For example, larger weighting functions can be given to transients. The purpose of the "super correlation" combined with the weighting function of the individual correlations is to find a common endpoint that is as good as possible. Since multiple channels are different waveforms, there is not only one ideal solution but also one ideal technique for searching for common endpoints. An alternative process for searching for an optimized pair of junction and endpoint locations is described below.                 

The weighted sum of each correlation provides useful insights into the overall periodic nature of the input blocks for all channels. The resulting overall correlation is retrieved in the correlation processing area between the junction and the maximum correlation processing location to determine the maximum value of the correlation.

Block Process Decision Step 218 (FIG. 5)

Returning to the description of FIG. 5, block processing decision step 218 (" blocks process? &Quot;) compares how much data has been time scaled compared to the required amount of time scaling. For example, in the case of compression, the determining step maintains a cumulative tracking of how much compression has been performed relative to the desired compression ratio. The output time scaling factor varies from block to block and slightly fluctuates around the required time scaling factor (can be more or less than the desired amount at any given time). If only one common overlap area is allowed in each time-matched ("current") block (a set of input data blocks representing time-matched audio segments, block for each channel), the block processing decision step is a time required. The scaling factor is compared to the output time scaling factor and a decision is made based on whether the current input data block has been processed. The determination is based on the length of the target segment in the current block, if any, in the common overlap area. For example, if 110% of the time scaling factor is required and the output scaling factor is below the required scaling factor, then the current input blocks are processed. Otherwise the current blocks are skipped. If more than one common overlapping region is allowed in the time-matched set of input data blocks, the block processing decision step determines to process one overlapping region, and if more than one overlapping region, skips the current blocks. Alternatively, other criteria for processing or skipping may be used. For example, instead of based on the determination of whether the current block should be skipped depending on whether the current cumulative expansion or compression is above the desired degree, the determination may be based on the current block, even if the result after processing is still an error in the opposite direction. Is based on whether processing the cumulative cumulative expansion or compression has changed to a desired degree.

Crossfade Processing 220 (FIG. 5)

Subsequent to the determination of the junction and endpoint locations and the determination of whether the block should be processed, the data block of each channel is processed by crossfade block step 220 ("crossfade blocks for each channel") of FIG. 5. This step accepts data blocks, common junctions, common endpoints and crossfade information for each channel.

Referring again to FIG. 10, a crossfade of appropriate shape is applied to the input data and the two segments are joined together to omit (see FIG. 10) or repeat the target segment. The length of the crossfade is preferably at most 10 msec, but may be shorter depending on the crossfade parameters determined in the previous analysis step. However, when auditory events are considered, longer crossfades may be used under certain conditions as discussed below. Nonlinear crossfades are artifacts that sound less than linear (straight) crossfades, especially for simple single frequency signals, such as tonal and tonal sweeps, according to the shape of the half-hanning window, for example This is because the window does not have the discontinuity of the slope of the straight crossfade. Other shapes, such as the shape of the Kaiser-Bessel window, also provide satisfactory results if the rise and envelope crossfades intersect at 50% and add up as a single piece throughout the crossfade period.

Pitch Scaling Processing 222 (FIG. 5)

Following crossfade processing, decision step 222 (“Pitch scale?”) Of FIG. 5 is checked to determine if pitch shifting (scaling) has been performed. As discussed above, time scaling cannot be performed in real time due to buffer underfloor or overfloor. However, pitch scaling may be performed in real time due to the operation of “resampling” step 224 (“Resample all data blocks”). The resampling step reads samples at different rates. In digital implementations with a fixed output clock, this is achieved by resampling. Thus, resampling step 224 resamples the time-scaled input signal with a pitch-scaled signal having the same time evolution or duration as the input signal but with changed spectral information. For real time implementations, resampling is performed with a dedicated hardware sample rate converter to reduce the computation in the DSP implementation. Note that resampling is only required when it is desirable to maintain a constant output sampling rate or to maintain the input sampling rate and the same output sampling rate. In digital systems, a constant output sampling rate or equivalent input / output sampling rate is typically required. However, if the primary output is converted to the analog domain, the variable output sampling rate will be of interest. Therefore, resampling is not a necessary part of any aspects of the present invention.

Following the pitch scaling decision and possible resampling, all processed input data blocks are output in step 226 (“output processed data blocks”) as files for non-real time operation or as output data blocks for real time operation. The process then checks for additional input data and continues processing.

Psychoacoustic analysis and auditory scene analysis example

An embodiment of a multichannel time and / or pitch scaling process using psychoacoustic analysis and auditory scene analysis in accordance with aspects of the present invention is shown in FIG. 17. Although the process is disclosed in the embodiment where the input signals are digital audio of one or more channels in which the samples are represented and the consecutive samples of each channel are divided into blocks of 4096 samples, the processed audio is digital or analog and the block Need not be divided into

Referring to Fig. 17, a first step, decision step 702 (" data input? &Quot;) determines whether digitized input audio data is available for data compression or data expansion processing. The source of data is a computer file or a block of input data, which is stored, for example, in a real time input buffer. If data is available, the data blocks of N time-synchronized samples representing the time-matched segment are taken to step 704 ("N samples for each channel" in one block for each input channel such that data compression or data expansion is processed). Acquisition ") (the number of channels is one or more). The number of input data samples used by the process, N, is fixed to a significant number of samples, thus splitting the input data into blocks. In principle, the processed audio is digital or analog and does not need to be divided into blocks.

FIG. 17 is discussed in connection with a practical embodiment of aspects of the invention in which the input data of each audio channel is data compressed or data expanded processed into blocks of 4096 samples, which translates to about 93 msec input data at a sampling rate of 44.1 kHz. Corresponding. It is to be understood that aspects of the invention are not limited to such practical embodiments. As mentioned above, the principles of the various aspects of the invention do not require arranging audio into samples, or, if so, providing blocks of constant length. However, to minimize complexity, a fixed block length of 4096 samples (or 2 n (power of two) samples) is useful for three main reasons. First, it provides a low enough delay to be acceptable for real time processing applications. Second, it is 2n samples, which is useful for fast Fourier transform (FFT) analysis. Third, it provides an appropriately large window size to perform useful auditory scene and psychoacoustic analysis of the input signal.

In the discussion below, it is assumed that the input signals are data with amplitude values in the range [-1, +1].

Auditory Scene Analysis 706 (FIG. 17)

Following audio input data blocking, the contents of the data blocks of each channel are divided into auditory events, each of which tends to be perceived individually ("perform an auditory scene analysis for each block of channels") (step 706). In the case of multiple channels, auditory scene analysis 706 and subsequent steps are performed in parallel (or sequentially) for all channels, on a channel-by-channel basis (while providing the appropriate storage of each channel's data and each analysis). Although parallel processing requires greater processing power, real time applications are desirable. The description of FIG. 17 assumes that the channels are processed in parallel.

Auditory scene analysis is achieved by the auditory scene analysis (ASA) process discussed above. Although one suitable process for performing auditory scene analysis is described herein, the invention contemplates that other useful techniques for performing ASA may be used. Because auditory events tend to be perceived as fairly constant, auditory scene analysis results provide valuable information useful in performing high quality time and pitch scaling and reducing the introduction of audible processing artifacts. By individually identifying and subsequently processing the auditory event, the audible artifacts introduced by the time and pitch scaling processing will be significantly reduced.

18 outlines a process in accordance with the techniques of the present invention used in the auditory scene analysis step of FIG. 17. The ASA phase consists of three general process substeps. The first substep 706-1 (“calculate the spectral profile of the input audio block”) has N sample input blocks, divides it into subblocks and calculates the spectral profile or spectral content for each subblock. . Therefore, the first substep calculates the spectral content of the continuous time segment of the audio signal. In a practical embodiment, the ASA subblock size, described below, is one-eighth the size (eg, 512 samples) of the input data block size (eg, 4096 samples). In a second substep 706-2, the difference in the spectral content between the subblocks is determined ("perform spectral profile difference measurement"). Therefore, the second substep calculates the difference in the spectral content between successive time segments of the audio signal. In the third substep 706-3 (“identifying the auditory event boundary position”), when the spectral difference between one spectrum-profile subblock and the next block is larger than the threshold, the subblock boundary is taken as the auditory event boundary. Lose. Therefore, the third substep sets the auditory event boundary between successive time segments when the difference in spectral profile content between such successive time segments exceeds a threshold. As discussed above, the effective indicator of the start or end of a recognized auditory event is believed to vary in spectral content.

In this embodiment, the auditory event boundaries define an auditory event with a length that is an integer number of spectral profile subblocks with one spectral profile subblock of minimum length (in this example, 512 samples). In principle, the event boundaries need not be so limited. Also note that the input block size limits the maximum length of the auditory event if the input block size is not variable (as an alternative to the actual embodiment discussed in the text, the input block size is essentially variable to the size of the auditory event). do).

19 outlines a method for calculating a time varying spectral profile. In Fig. 19, overlapping segments of audio are windowed and used to calculate the spectral profile of the input audio. Overlap results in finer resolution with respect to the location of auditory events, and it is hard to miss events such as transients. However, as time resolution increases, frequency resolution decreases. Nesting also increases computational complexity. Therefore, the overlap is omitted in the practical example stated below.

The following variables are used to calculate the spectral profile of the input block:

N = number of samples in the input audio block

M = number of windowed samples used to compute the spectral profile

P = number of samples of spectral arithmetic overlap

Q = number of computed spectral windows / regions

In general, arbitrary integers are used for variables. However, implementations will be more efficient if M is set to 2n (power of 2) so that a standard FFT is used for the spectral profile calculation. In addition, if N, M and P are chosen such that Q is an integer, this will avoid underruning or overrunning the audio at the end of the N sample blocks. In a practical embodiment of the auditory scene analysis process, the listed parameters are set as follows:

N = 4096 samples (or 93 msec at 44.1 kHz)

M = 512 samples (or 12 msec at 44.1 kHz)

P = 0 samples (or nested)

Q = 8 blocks

The above listed values were determined empirically and were found to accurately identify the location and duration of the auditory event for time scaling and pitch shifting. However, setting the value of P to 256 samples (50% overlap) has been found to be useful for identifying some elusive events. Although many different types of windows are used to minimize spectral artifacts due to windowing, the window used for spectral profile calculation is M-point hanning, Kaiser-Bessel or other suitable, preferably non-rectangular window. to be. The indicated values and the hanning window type were chosen after extensive experimental analysis and have been shown to provide good results over a wide range of audio data. Non-rectangular windowing is desirable for the processing of audio signals with significantly lower frequency content. Rectangular windowing creates spectral artifacts that cause inaccurate detection of events.

In substep 706-1, the spectrum of each M-sample subblock is windowed to M-point hanning, Khyber-Bessel or other appropriate window, and converted into the frequency domain using M-point fast Fourier transform and Is computed by calculating the magnitude of the FFT coefficient. The resulting data is normalized so that the largest size is set in units, and M normalized arrays are converted to a log domain. The array need not be converted to the log domain, but conversion simplifies the calculation of the difference measure at substep 706-2. In addition, the log domain more closely matches the log domain amplitude characteristics of the human auditory system. The resulting log domain values range from negative infinity to zero. In a practical embodiment, the lower limit is imposed on the range of values; The lower limit is, for example, fixed at -60 dB or frequency dependent to reflect quiet low audibility at low or high frequencies. (Note that it is possible to reduce the size of the array to M / 2 in that the FFT represents positive frequencies other than negative).

Substep 706-2 calculates the value of the difference between the spectra of adjacent subblocks. For each subblock, each M (log) spectral coefficients from substep 706-1 are subtracted from that coefficient for the preceding subblock, and the magnitude of the difference is calculated. This M difference is then summed to one number. Therefore, for the whole audio signal, the result is an array of Q positive numbers; The larger the number, the greater the subblock differs from the preceding subblock in the spectrum. This difference value is also expressed as the average difference per spectral coefficient by dividing the difference value by the number of spectral coefficients used in the summation, in which case M is the coefficient.

Substep 706-3 identifies the locations of auditory event boundaries by applying a threshold to the array of difference measurements from substep 706-2 having a threshold. When the degree of difference exceeds the threshold, the variation in the spectrum is considered sufficient to signal a new event and the number of subblocks of the variation is recorded as the event boundary. For the given values of M, N, P and Q and for log domain values expressed in dB (substep 706-2), the threshold is set to 2500 if the full size FFT (including the reflected part) is compared or , Or half of the FFT is set to 1250 (as mentioned above, the FFT represents a positive frequency in addition to the negative-for the magnitude of the FFT, one is the reflection of the other). This value was chosen experimentally and provides good auditory event boundary detection. This parameter value is varied to reduce (increase the threshold) or increase (decrease the threshold) the detection of an event.

The details of this practical embodiment are not important. Compute spectral content of consecutive time segments of an audio signal, calculate the difference between successive time segments, and audition at each boundary between successive time segments when the difference in the spectral content profile between such successive time segments exceeds a threshold. Other ways of setting event boundaries can be used.

The outputs of the auditory scene analysis process step 706 of FIG. 17 are the location of the auditory event boundary, the number of auditory events detected in the input block, and the last or L-th spectral profile block computed for the N point input block. As mentioned earlier, the auditory analysis process is performed once for each channel's input data block. As described in more detail below with respect to step 710, if one or more audio channels are to be processed, the auditory event information is combined to generate a complete auditory event overview for all channels ("Combined Auditory Events"). Create segments). This utilizes phase locked multichannel processing. In this way, multiple audio channels are considered multiple individual audio "tracks" that are mixed together to create a single composite audio scene. An example of event detection processing for two channels is shown in FIG. 20 and described below.

Psychoacoustic analysis of auditory events 708 (FIG. 17)

Referring back to FIG. 17, following input data blocking and auditory scene analysis, psychoacoustic analysis is performed on each input data block for each auditory event ("perform psychoacoustic analysis for each event in each block"). ) (Step 708). In general, the psychoacoustic characteristic remains largely uniform in the audio channel over the length or time period of the auditory event because the audio in the auditory event is perceived to be fairly constant. Therefore, the audio information is more detailed in the psychoacoustic analysis process, which considers 64 sample subblocks in the actual example disclosed in the text and 512 sample subblocks in the actual example disclosed in the text than in the auditory event detection process. Although evaluated, the psychoacoustic analysis process generally finds only one salient psychoacoustic condition throughout the auditory event and tags the event. The psychoacoustic analysis executed as part of the process of FIG. 17 differs from the psychoacoustic analysis performed as part of the process of FIG. 5 in that it is applied to each auditory event in the input block rather than to the entire input block.

In general, psychoacoustic analysis of auditory events provides two important pieces of information: first, the information identifies the portion of the event of the input signal that is most likely to produce an audible artifact if processed. It identifies that a portion of the input signal can be beneficially used to mask the processing performed. FIG. 21 states a process similar to the process of FIG. 6 described above, used in the psychoacoustic analysis process. The psychoacoustic analysis process consists of four general processing substeps. As mentioned above, each psychoacoustic processing substep uses a psychoacoustic subblock whose size is one eighth of the spectral profile subblock (or one sixth of an input block). Therefore, in this example, the psychoacoustic subblocks are approximately 1.5 msec (or 64 samples at 44.1 kHz) as shown in FIG. The actual size of the psychoacoustic subblocks is not limited to 1.5 msec and even if they have different values, this size was chosen for the actual implementation because of the real-time processing requirements (requires psychoacoustic processing overhead with a smaller subblock size). ) And the resolution of the transient location (smaller subblocks provide more detailed information about the location of the transient). In principle, the psychoacoustic subblock size does not need to be the same for every type of psychoacoustic analysis, but for practical implementation, this is desirable.

Transient Detection 708-1 (FIG. 21)

Referring to Figure 21, a first substep 708-1 ("Perform transient detection / masking analysis") analyzes each auditory event segment in the input block of each audio channel so that each such segment contains a transient. Decide if This is necessary even though the spectral variation aspect of the ASA process essentially considers transients and identifies audio segments containing transients as auditory events, depending on whether the spectrum-based ASA process described in the text contains transients. This is because they do not identify tory events (since transients cause spectral fluctuations). The resulting transient transient information is used for masking analysis and helps to replace temporary or common junction locations. As discussed above, it is known that transients introduce temporary masking (hiding audio information before and after the occurrence of the transient). An auditory event segment in a particular block is tagged as a transient, whether or not the transient occupies the entire length or time period of the event. The transient detection process in the psychoacoustic analysis step is essentially the same as the transient detection process described above, except that only the segments of the input block that make up the auditory event are analyzed. Therefore, for details of the transient detection process, refer to the process flowchart of FIG. 8 described above.

 Audible Critical Analysis 708-3 (FIG. 21)

Referring to FIG. 21, in the psychoacoustic analysis process, a second step 708-2, “Performing audible threshold analysis”, analyzes each auditory event of the input block of each audio channel such that each such segment is an audible threshold. Or a signal strength that is significantly low enough to be considered or less. As mentioned above, auditory events tend to be perceived as fairly constant in their length or time period, and of course apply to possible deviations close to their boundaries due to the granularity of the spectral-profile subblock size. (E.g., audio varies its characteristics more accurately at possible event boundaries). In the psychoacoustic analysis step, the audible threshold analysis process, except for analyzing only the segments of the input block constituting the auditory event (see, for example, the description of sub-step 206-2 in FIG. 6). Are essentially the same, as in the previous description. Auditory events are important because the workpieces introduced by the time scaling and pitch shifting of such auditory events are not easy to hear in such areas.

High Frequency Analysis 708-3 (Figure 21)

A third substep 708-3 (FIG. 21) (" executing high frequency analysis ") analyzes each auditory event of the input block of each audio channel to determine if each such segment contains high frequency audio content. High-frequency segments are important in psychoacoustic analysis because the audible threshold in quiet places increases rapidly above approximately 10-12 kHz, and the ear is less sensitive to discontinuities in high-frequency waveforms than to discontinuities in low-frequency waveforms. Although there are a number of possible ways of determining whether an audio signal consists primarily of high frequency energy, the method described above with respect to substep 206-3 of FIG. 6 provides good detection results, minimizes computational requirements and organizes auditory events. Applies to the step of analyzing the segment.

Audio Level Analysis 708-4 (FIG. 21)

The fourth substep 708-4 (FIG. 21) of the psychoacoustic analysis process, "Perform a general audio block level analysis" substep analyzes each auditory event segment of the input block of each audio channel to determine the numerical value of the signal's signal strength. Calculate the measure Such information is used if the event does not have the psychoacoustic characteristics that can be used during processing. In this case, data compression or data expansion processing prefers the quietest auditory event or the lowest level in the input data block based on the principle that lower level segments of audio generate lower level processing artifacts that are not audible. A simple example of using a single channel of orchestral music is shown in FIG. Spectral fluctuations that occur as new notes are played cause new events 2 and 3 in samples 2048 and 2590, respectively. The orchestral signal shown in FIG. 23 does not include any transient under the audible threshold or high frequency content. However, the first auditory event of the signal is at a lower level than the second and third events of the block. Audible processing artifacts are minimized by selecting quiet events for data expansion or compression processing rather than louder subsequent events.

To compute the general level of auditory events, substep 708-4 takes data in the event that is divided into 64-sample subblocks, finds the size of the largest sample of each subblock, and finds the 64 in the event. Take the average of the largest magnitudes in terms of the number of sample subblocks. The general audio level of each event is stored for later comparison.

Combined auditory event determination and common junction establishment 710 (Figure 17)

As shown in FIG. 17, following auditory scene analysis and psychoacoustic analysis of each segment constituting the auditory event in each block, the next step 710 ("Combined auditory event determination and common junction establishment") in processing is performed. Determine boundaries of combined auditory events in parallel blocks across the channel (the combined auditory events are described in more detail below in connection with FIG. 20), and one or more combined auditories in a set of each parallel block. It is to determine a common junction in the accompanying block across all channels for the event segments and to grade the psychoacoustic quality of the auditory events in the combined auditory event segments. Such grading is based on the hierarchy of psychoacoustic criteria stated above. When a single channel is processed, the auditory events of that channel are processed in the same way as the combined auditory events of the multiple channels of the present description.

The setting of one or more common junctions is performed in the manner described above in connection with the description of FIG. 5 except that combined auditory events are considered more than common overlap of identified regions. Thus, for example, the common junction is typically set initially in the combined auditory event period for compression and later in the combined auditory event period for expansion. For example, a default time of 5 msec is used after the start of the combined auditory event.

The psychoacoustic quality of the combined auditory event segment in each channel is considered to determine if data compression or expansion processing occurs within a particular combined auditory event. In principle, psychoacoustic quality determination is performed after establishing a common junction for each combined event segment or before establishing a common junction for each combined event segment (in this case any common junction is based on complexity). Need not be set to a combined event with such a negative psychoacoustic quality grade that is skipped).

The psychoacoustic quality ranking of the combined event is based on the psychoacoustic characteristics of the audio in the various channels of the combined event time segment (the combined event where each channel is masked by transients has the highest psychoacoustic quality rating Combined events in which none of the channels meet the psychoacoustic criteria have the lowest psychoacoustic quality rating). For example, the hierarchy of psychoacoustic criteria described above may be used. The relative psychoacoustic quality rating of the combined event is then used in connection with the first decision step (step 712) described below, which takes into account the complexity of the combined event segment in the various channels. Complex segments are susceptible to audible artifacts when performing data compression or expansion. For example, a complex segment may contain at least one of the channels that do not meet any psychoacoustic criteria (described above) or include transients (as mentioned above, it is not desirable to vary the transients). Will. In extreme complexity, for example, every channel fails to meet psychoacoustic criteria or involves transients. The second decision step (step 718) described below takes into account the length of the target segment, which is affected by the length of the combined event segment. In the case of a single channel, an event is ranked according to psychoacoustic criteria to determine whether it should be skipped.

The combined auditory events will be well understood by referring to FIG. 20 which shows the auditory scene analysis results for the two channel audio signal. 20 shows parallel blocks of audio data in two channels. ASA processing of the audio of the first channel, the top waveform of FIG. 20, identifies the auditory event boundaries at samples of multiple spectral profile subblock sizes, in this example 1024 and 1536 samples. The lower waveform of FIG. 20 is the second channel and ASA processing also identifies event boundaries in samples that are multiple spectral profile subblock sizes, in this example 1024, 2048 and 3072 samples. Compound auditory event analysis for both channels results in compound auditory event segments with boundaries in samples 1024, 1536, 2048, and 3072 (the auditory event boundaries of every channel are "OR" together). In practice, the accuracy of the auditory event boundaries depends on the size of the spectral profile subblock size (N is 512 samples in this embodiment), since the event boundaries can only occur at the subblock boundaries. Nevertheless, the subblock size of 512 samples has been found to determine auditory event boundaries with sufficient accuracy in providing satisfactory results.

Referring to FIG. 20, if only a single channel of audio including transient in the top view is processed, then three separate auditory events are available for data compression or expansion processing. These events include (1) the quiet portion of the audio before the transient, (2) the transient event, and (3) the echo / sustain portion of the audio transient. Similarly, if only the voice signal represented in the lower figure is processed, then four separate auditory events are available for data compression or expansion processing. These events include significant high frequency hissing events, events such as hissing or morphing into vowels, the first half of the vowel, and the second half of the vowel.

20 also shows a composite event boundary when auditory event data is shared across two channels of parallel data blocks. Such event segmentation provides five complex event boundaries where event compression or expansion processing can occur (event boundaries are ORed together). Processing within the complex auditory event segment ensures that processing occurs with the auditory events of each channel. Note that depending on the method of data compression or expansion used and the content of the audio data, it is most appropriate to process only data in two channels or only a few composite events (rather than composite events) in one composite event. Compound auditory event boundaries are used to define a segment for data compression or expansion processing that is executed independently of the data of each parallel input channel block, even if they result from ORing the event boundaries of all audio channels. . Therefore, if only a single composite event is selected for processing, the data for each audio channel is processed within the length or time segment of that composite event. For example, in FIG. 20, if the desired overall time scaling amount is 10%, then the minimum amount of audible artifacts is totally varied if only four complex event regions are processed in each channel and the number of samples in four complex event regions varies sufficiently. It will be introduced if the length of N samples varies by 0.10 * N samples. However, it is also possible to distribute the processing and process the respective composite events so that the overall variation in length between all composite events sums up to 0.10 * N samples. It is determined at step 718, described below, that the number of composite events and the numbers are selected for processing.

24 shows an example of four channel input signals. Channels 1 and 4 each contain three auditory events, while channels 2 and 3 each contain two auditory events. Compound auditory event boundaries for parallel data blocks across all four channels are located at sample numbers 512, 1024, 1536 and 3072 as indicated at the bottom of FIG. This means that all six composite auditory events are processed across four channels. However, some composite auditory events have such a relatively low psychoacoustic rating (ie they are too complex) or too short to be desirable to process within them. In the example of FIG. 24, the most preferred compound auditory event for processing is compound event region 4 and the next most preferred is compound event region 6. The other three composite event regions are all of minimum size. In addition, composite event region 2 includes transients in channel one. As mentioned above, it is most desirable to avoid processing during transients. Compound event region 4 is preferred because it is the longest and satisfactory psychoacoustic characteristic of each channel-it has transient postmasking on channel 1, channel 4 is below the audible threshold and channels 2 and 3 are relatively Low level.

The maximum correlation processing length and crossfade length limit the maximum amount of audio that can be removed or repeated within the complex auditory event time segment. The maximum correlation processing length, however few, is limited to the length or predetermined value of the complex auditory event time segment. The maximum correlation processing length requires that data compression or expansion processing be within the start and end boundaries of the event. Failure to do so causes "smearing" or "blurring" of the event boundary.

FIG. 25 illustrates details of the four-channel data compression processing example of FIG. 24 using the fourth composite auditory event time segment of the channels as the segment to be processed. In this example, channel 1 includes a single transient in composite event 2. For this example, the junction location is selected to be sample 1757 located at the largest compound auditory event following the transient at sample 650 of audio channel 1. This junction location was chosen based on positioning the location 5 msec (at 44.1 kHz, 221 samples or half the length of the crossfade) after the initial compound event boundary to avoid damaging the event boundary during crossfading. . Placing junction points in these segments takes advantage of the postmasking provided by Compound Event 2. In the example shown in FIG. 25, the maximum processing length takes into account the location of the composite multichannel auditory event boundary in sample 2560 which should be avoided during processing and crossfading. As part of step 710, the maximum processing length is set to 582 samples. This value is calculated by estimating the following 5msec length of 1/2 crossfade (221 samples at 44.1kHz):

Max Processing Length =

Event boundary-crossfade length-processing junction location

582 = 2560-221-1757

The output of step 710 is the boundary of each compound auditory event, the common junction of parallel data blocks across the channel for each compound auditory event, the compound auditory event, the psychoacoustic quality class of crossfade parameter information, and each compound auditory event. The maximum processing length over channels for.

As described above, a composite auditory event with a low psychoacoustic quality rating indicates that no data compression or expansion occurs in the segment across the audio channels. For example, as shown in FIG. 26, which only considers a single channel, the audio of events 3 and 4, each 512 samples long, contains mainly low frequency content, which is not suitable for data compression or expansion processing (a sufficient amount of useful significant frequency). Not periodicity). Such events are assigned a low psychoacoustic quality class and are skipped.

Skip 712 based on complexity (FIG. 17)

Therefore, step 712 ("Skip based on complexity?") Sets the skip flag when the psychoacoustic quality grade is low (indicating high complexity). By making this complexity determination before rather than after the correlation processing of step 714 described below, it avoids executing unnecessary correlation processing. Step 718, described below, further determines whether audio over various channels should be processed during a particular composite auditory event segment. Step 718 considers the length of the target segment in the composite auditory event with respect to the current processing length requirement. The length of the target segment is unknown until the common endpoint is determined in correlation step 714 described below.

Correlation processing

For each common junction, an appropriate common endpoint is needed to determine the target segment. If it is determined that input data for the current composite auditory event segment should be processed (step 712), then two types of correlation processing (step 714) occur, as shown in FIG. 17, which indicates the correlation of time domain data. Processing (steps 714-1 and 714-2) and correlation processing (steps 714-3 and 714-4) of phase information of the input signal. The use of complex phase and time domain information of input data is believed to provide high quality time scaling results for signals ranging from voice to complex music as compared to using only time-domain information. The details of the processing step 714 and the multi-correlation step 716, including substeps 714-1, 2, 3, and 4, refer to the composite auditory rather than the area where processing is psychoacoustically identified in steps 714 and 716. It is essentially the same as described above in connection with step 214 (and its substeps 214-1, 2, 3 and 4) and 216 except for event segmentation.

Alternate junction and endpoint selection process

As mentioned above, aspects of the invention contemplate an alternative method for selecting a junction point location and an accompanying endpoint location. The processes described above select the junction somewhat randomly and then select the endpoint based on the average period (essentially one degree of freedom). The alternative method described instead selects junction / endpoint pairs (two degrees of freedom) based on the purpose of providing the smallest audible workpiece through the junction to the best possible crossfade.

Figure 27 illustrates a first step of selecting for audio, junction and endpoint locations of a single channel in accordance with an alternative to the invention. In FIG. 27, the signal consists of three auditory events. Psychoacoustic analysis of the events indicates that event 2 includes transients that provide temporary masking, predominantly post-masking, which extends to event 3. Event 3 is also a large event, providing the longest processing area. In order to determine the optimal junction location, the region of data Tc (time of crossfade) sample length is correlated to the data of the processing region. The main junction is located in the middle of the Tc junction region.

The cross-correlation of the junction area and the processing area results in a correlation measure used to determine the best endpoint (in a similar way to the first alternative), where the best endpoint for a particular junction is the maximum correlation in the calculated correlation function. Is determined by finding the value. According to a second alternative method, the optimal junction / endpoint pair is determined by correlating a series of sample junctions in the correlation processing region adjacent to the trial junctions.

As shown in Figures 30A-C, this best endpoint is preferably after the minimum endpoint. The minimum endpoint is set such that the minimum number of samples are always processed (added or removed). The best endpoint is preferably at or before the maximum endpoint. As shown in FIG. 28, the maximum endpoint is no closer than a half crossfade length from the end of the event segment being processed. As mentioned above, in the actual implementation described, no auditory event exceeds the end of the input block. This is the case for event 3 in FIG. 28, which is limited to the end of the 4096 sample input block.

The value of the correlation function at its maximum between the minimum and maximum endpoints determines how similar the junction is to the optimal endpoint for a particular junction. In order to optimize the junction / endpoint pair (rather than simply optimizing the endpoint for a particular junction), a series of correlations can be obtained by selecting a different Tc sample junction region from N samples each located to the right of the previous region and in FIG. 28. Calculated by recalculating the correlation function shown.

The minimum number of N samples is one sample. However, selecting N as one sample significantly increases the number of correlations needed to calculate, which significantly hinders real-time implementation. Simplification can be made such that N is set to the length of the multiple samples, such as Tc samples, crossfade. This gives good results and reduces the processing required. 29 conceptually shows an example of the number of correlation calculations required when the junction area is continuously advanced by the Tc sample. Three processing steps are superimposed on each audio data block data diagram. The processing shown in FIG. 29 results in three correlation functions each having a maximum value as shown in FIGS. 30A-C.

As shown in FIG. 30B, the maximum correlation value is calculated from the second junction repetition. This means that the relevant maximum determined by the correlation with the second junction should be used as the distance from the junction to the termination.

In executing the correlation, conceptually, the Tc samples slide exponentially to the right and the corresponding samples in the Tc and in the processing area are multiplied together. Tc samples are windowed, in this example a rectangular window around the sample junction. Window shapes that place more emphasis on the sample junction and less on the area away from the sample junction provide better results. Initially (no slides, no nesting), the correlation function is zero by definition. The function rises and falls until it finally falls back to zero when the sliding goes away so that it does not overlap again. In a practical implementation, the FFT is used to calculate the correlation. The correlation functions shown in FIGS. 30A-C are limited to ± 1. These values are not arbitrary normalization functions. Standardization of correlations discards the relative weights between the correlations used to select the best junctions and endpoints. When determining the best junction, compare the non-standard maximum correlation values between the minimum and maximum processing point locations. The largest correlation value with the largest value indicates the best junction and endpoint combination.

This alternative junction and endpoint location method has been described for the case of data compression where the endpoint is after the junction. However, it is equally applicable to the case of data expansion. For data expansion, there are two alternatives. According to a first alternative, the optimized junction / endpoint pair is determined as described above. The identification of the junction and the endpoint then changes so that the junction becomes an endpoint and the endpoint becomes a junction. According to a second alternative, the area around the sample junctions is "backward" rather than "forward" to determine the best endpoint / junction phase that the endpoint is "earlier" than the junction.                 

Multichannel processing is performed in a similar manner to the processing described above. After the auditory event regions are combined, the correlations from each channel are combined for each junction evaluation step and the combined correlations are used to determine the maximum value and the best junction and endpoint pair.

Additional reduction in processing is provided by decimating the time domain data into the elements of M. This reduces the computational strength to a factor of 10 but provides only a lower endpoint (in M sample). Fine-tuning is achieved by performing different correlations using all of the undecimated audio to find the best endpoint with the resolution of one sample after some decimated processing.

Another alternative is to correlate the windowed area around the sample junction locations with respect to the windowed area around the sample endpoint locations instead of with respect to the larger un-wided correlation area. Although it is not computationally complex to implement cross-correlation between a windowed sample junction area and an un-windowed correlation area (such correlations are performed in the time domain before conversion to the frequency domain for the remaining correlation calculations). , Cross-correlating two windowed regions in the time domain is computationally required.

Although this alternative junction / endpoint selection process has been described in the context of an embodiment where audio signals are divided into auditory events, the principles of this alternative process are equally applicable to other environments, including the process of FIG. In the FIG. 5 environment, junctions and endpoints are within the overlap of psychoacoustically articulated or identified regions, rather than in auditory events or compound auditory events.

Event processing decision

Returning to the description of FIG. 17, the next step of processing is an event block processing decision step 718 ("process complex event?"). The time scaling process uses the periodicity of the time domain or time domain and phase information and uses this information to process the audio signal data, and the output time scaling factor is not linear over time and varies slightly around the requested input time scaling factor. do. Among other functions, the event processing decision compares how much preceding data has been time scaled with the requested amount of time scaling. If processing up to the time of the composite auditory event segment exceeds the desired amount of time scaling, then this composite auditory event segment is skipped (ie not processed). However, if the amount of time scaling performed up to this time is below the desired amount, then the composite auditory event segment is processed.

For cases where a composite auditory event segment should be processed (according to step 712), the event processing decision step compares the requested time scaling element with the output time scaling element achieved by processing the current composite auditory event segment. The decision step then determines whether the current compound auditory event segment of the input data block should be processed. Note that the actual processing is the target segment included in the composite auditory event segment. An example of how this works for the event level for the input block is shown in FIG. 31.                 

31 shows an example in which the overall input block length is 4096 samples. The audio of this block contains three auditory events (or compound auditory events, in which case the figure represents one of multiple channels), which are 1536, 1024 and 1536 samples in length, respectively. As indicated in FIG. 17, since each auditory event or compound auditory event is processed separately, the 1536 sample auditory event at the beginning of the block is preferentially processed. In the example above, the junction and correlation analysis finds that when starting at junction sample 500, the process can remove or repeat the audio (target segment) of 363 samples with minimal audible artifacts. This means that the following time scaling factor

363 samples / 4096 samples = 8.86%

It currently provides 4096 sample input blocks. If the combination of 363 samples of processing available according to the processing provided from the subsequent auditing event or composite auditory event segment is greater than or equal to the desired amount of time scaling processing, then the first auditory event or complex auditory event segment is deleted. Processing should be sufficient and the remaining auditory event or composite auditory event segments of the block are skipped. However, if the 363 samples processed in the first auditory event are not sufficient to meet the desired amount of time scaling, then the second and third events will also be considered for processing.

Splicing and Crossfade Processing 710 (FIG. 5)

Following determination of the junction and endpoint, each compound auditory event that was not discarded by step 712 or 718 is processed by “junction and crossfade” step 710 (FIG. 17). This step receives each event or composite event data segment, junction location, processing endpoint and process parameters. Step 720 generally operates in the manner of step 218 of the process of FIG. 5 described above, except that it acts on auditory events or compound auditory events and the length of the pro spade is longer.

Crossfade parameter information is affected not only by the presence of transient events that allow shorter crossfades to be used, but also by the overall length of the complex auditory event in which the common junction location is located. In a practical implementation, the crossfade length is scaled in proportion to the size of the auditory event or composite auditory event segment where data compression or expansion processing should occur. As described above, in a practical embodiment, the smallest auditory event allowed is 512 points, and the size of the event is increased to a maximum size of an input block size of 4096 samples by 512 sample increments. The crossfade length is set to 10 msec for the smallest (512 point) auditory event. The crossfade length increases to a maximum or 30-35 msec in proportion to the size of the auditory event. Such scaling is useful because, as already discussed, longer crossfades tend to mask artifacts, but also cause problems when the audio fluctuates rapidly. Since auditory events limit the element that contains the audio, crossfading takes advantage of the fact that the audio is significantly static within the auditory event and longer crossfades cannot be used without introducing an audible artifact. Although the block sizes and crossfade times mentioned above have been found to provide useful results, they are not critical to the invention.

Pitch scaling processing 722 (FIG. 5)

Following the splicing / prospade processing of the complex auditory events, decision step 722 (“Pitch Scale?”) Is checked to determine if pitch shifting should be performed. As already discussed, time scaling cannot be performed in real time due to block underfloor or overfloor. Pitch scaling can be performed in real time because of resampling step 724 (“Resample all data blocks”). The resample step resamples a time scaled input signal that has the same time evolution as the input signal but results in a pitch scaled signal with modified spectral information. For real-time implementation, resampling is performed with a dedicated hardware sample-rate converter to reduce computational requirements.

Following the pitch scaling decision and possible resampling, all processed input data blocks are output to a file for non-real time operation or to an output data buffer for real time operation (“output processed data blocks”) (step 726). . The process flow then checks for additional input data ("data entry?") And continues processing.

Implementations of other variations and modifications of the invention and their various aspects will be apparent to those skilled in the art, and it will be understood that the invention is not limited to the specific embodiments described. Accordingly, it is contemplated to include this invention and any modifications, variations, or equivalents that fall within the spirit and scope of the underlying principal principles disclosed and claimed herein.

The present invention and its various aspects may be implemented as software functions executed on a digital signal processor, a programmed general purpose digital computer, and / or a special purpose digital computer. The interface between the analog and digital signal streams is implemented in the appropriate hardware and the functions are implemented in software and / or firmware.

Claims (84)

In a method of time scaling or pitch shifting an audio signal, Dividing the audio signal into auditory events; And Including time scaling or pitch shifting processing within the auditory event; Dividing the audio signal into auditory events, Calculating spectral content of successive time segments of the audio signal; Calculating a difference in spectral content between successive time segments of the audio signal; And Setting an auditory event boundary at the boundary between successive time segments if the difference in spectral content between successive time segments exceeds a threshold; Time scaling or pitch shifting the audio signal comprising a. The method of claim 1, wherein the time scaling or pitch shifting processing comprises: Selecting a splice point and an end point in the auditory event; Deleting the portion of the audio signal starting at the junction or repeating the portion of the audio signal ending at the junction; And Reading the resulting audio signal at a rate that yields a predetermined time scaling or pitch shifting; Time scaling or pitch shifting the audio signal comprising a. The method of claim 1, wherein the time scaling or pitch shifting processing comprises: Selecting a junction within the auditory event to define a leading segment of the audio signal leading the junction; Selecting an endpoint spaced from the junction within the auditory event to define a trailing segment of the audio signal tracking the endpoint and a target segment of the audio signal between the junction and the endpoint; Connecting the leading and trailing segments at the junction to reduce the number of audio signal samples by omitting a target segment if the endpoint has a sample number higher than the junction, or if the endpoint has a sample number lower than the junction Increasing the number of samples by repeating the target segment; And Reading the connected leading and trailing segments at a rate that yields a predetermined time scaling or pitch shifting; Time scaling or pitch shifting the audio signal comprising a. The method of claim 3, wherein the connected leading and trailing segments, Pitch-shift the audio signal in the same time period as the original time period, When omitting the target segment, time compress the audio signal in a reduced time period at the same rate as the relative change in the decrease in the number of samples, When repeating the target segment, time-expand the audio signal in an increased time period at the same rate as the relative change in the increase in the number of samples, Time compression and pitch shifting the audio signal in a reduced time period at a rate that is different from the relative change in the decrease in the number of samples, or A method of time scaling or pitch shifting an audio signal, characterized in that it is read at a rate of time expansion and pitch shifting of the audio signal in an increased time period at a rate different from the relative change in the number of samples. 4. The method of claim 3, wherein connecting the leading and trailing segments at a junction includes crossfading the leading and trailing segments. 4. The method of claim 3, wherein when reducing the number of audio signal samples by omitting a target segment, the endpoint is selected by autocorrelating a segment of audio that tracks the junction. How to pitch shift. 4. The method of claim 3, wherein when increasing the number of audio signal samples by repeating a target segment, the endpoint is selected by cross correlating the segment of audio leading the junction and the segment of audio tracking the junction. Time scaling or pitch shifting an audio signal. 4. The junction point and endpoint position of claim 3, wherein if the number of audio signal samples is increased by omitting the target segment or increasing the number of audio signal samples by repeating the target segment, Correlate the windows of audio samples around the series of sample junction locations with respect to the area of audio samples adjacent to each of the series of junction points, and Time scaling or pitch shifting an audio signal, characterized in that it is selected by determining a sample junction location that results in the strongest correlation, designating the sample junction location as the junction location, and setting the endpoint location to the location of the strongest correlation. How to. 9. The method of claim 8, wherein the window is a rectangular window.  10. The method of claim 9, wherein the window has a width of crossfade. 9. The method of claim 8, wherein the series of sample junction points are spaced apart by at least one audio sample. 12. The method of claim 11, wherein the series of sample junction locations are spaced apart by the width of the window. 13. The method of claim 12, wherein the window has a width of crossfade. 9. The method of claim 8, wherein when reducing the number of audio samples by omitting a target segment, the region of audio samples adjacent to each series of sample junction locations follows each sample junction location, whereby the junction is less than the endpoint. A method of time scaling or pitch shifting an audio signal, characterized by the preceding. 9. The method of claim 8, wherein when increasing the number of audio samples by repeating a target segment, the region of audio samples adjacent to each series of sample junction locations follows each sample junction location and identifies the identity of the junction and endpoint. ), Whereby the endpoint precedes the junction. 9. The method of claim 8, wherein when increasing the number of audio samples by repeating a target segment, the region of audio samples adjacent to each series of sample junction locations precedes each sample junction location, whereby the endpoint is a junction point. A method of time scaling or pitch shifting an audio signal, characterized by preceding. The junction position and endpoint position of claim 3, wherein when the number of audio signal samples is increased by omitting the target segment or by increasing the number of audio signal samples by repeating the target segment, Correlating a window of audio samples around a series of sample junction locations relative to an area of audio samples adjacent to each series of sample junction locations and decimated by a factor of M, Determine the sample joint location that gives the strongest correlation, specify the sample joint location as the decimated joint, Undecimated audio around a second series of sample junction locations within an M sample of the decimated junction for an area of undecimated audio samples adjacent to each second series of sample junction locations. Correlate the windows of the samples, and Audio, characterized in that it is selected by determining a sample junction location at the second series of sample junction locations that yield the strongest correlation, designating the sample junction location as the junction, and setting the endpoint location at the location of the strongest correlation. A method of time scaling or pitch shifting a signal. In the method of time scaling or pitch shifting a plurality of audio signal channels, Dividing an audio signal of each channel into auditory events; And Determining combined auditory events, each having a boundary when the auditory event boundary occurs in the audio signal channel; And Time scaling or pitch shifting all of the audio signal channels within a combined auditory event, such that the processing is within the auditory event or portion of the auditory event of each channel; Dividing the audio signal of each channel into auditory events, Calculating spectral content of successive time segments of the audio signal of each channel; Calculating a difference in spectral content between successive time segments of the audio signal of each channel; And Setting an auditory event boundary to an audio signal of each channel at the boundary between successive time segments if the difference in spectral content between successive time segments exceeds a threshold; Time scaling or pitch shifting a plurality of audio signal channels comprising a. 19. The method of claim 18, wherein the time scaling or pitch shifting processing is: Selecting a common junction within an auditory event combined between channels of audio signals, such that junctions from one or more common junctions in each of the multiple channels of audio signals are aligned with each other; Deleting a portion of each channel of audio signals starting at the common junction or repeating a portion of each channel of audio signals ending at the common junction; And Reading the resulting channels of audio signals at a rate that yields some time scaling or pitch shifting processing for the multiple channels of audio; Time scaling or pitch shifting a plurality of audio signal channels comprising a. 19. The method of claim 18, wherein the time scaling or pitch shifting processing is: By selecting a common junction within a combined auditory event between channels of audio signals, junctions originating from the common junction of multiple channels of audio signals are aligned with each other, defining a leading segment of the audio signal where each junction leads the junction. Doing; Selecting a common endpoint spaced from the common junction within the auditory event so that the endpoints from the common endpoint in each of the multiple channels of audio signals are aligned with each other and the trailing segment and junction of the audio signal tracking the endpoint Defining a target segment of the audio signal between the endpoints and the endpoints; Connect the leading and trailing segments at the endpoints in each of the channels of audio signals to reduce the number of audio signal samples by omitting a target segment if the endpoint has a sample number higher than the junction, or the endpoint is at the junction Increasing the number of audio signal samples by repeating the target segment if it has a lower sample number; And Reading the leading and trailing segments connected in each of the channels of audio signals at a rate that yields predetermined time scaling or pitch shifting for the multiple channels of audio; Time scaling or pitch shifting a plurality of audio signal channels comprising a. The method of claim 20, wherein the connected leading and trailing segments are: Pitch-shift the audio signal in the same time period as the original time period, When omitting the target segment, time compress the audio signal in a reduced time period at the same rate as the relative change in the decrease in the number of samples, When repeating the target segment, time-expand the audio signal in an increased time period at the same rate as the relative change in the increase in the number of samples, Time compressing and pitch shifting the audio signal in a reduced time period at a rate different from the relative change in the number of samples reduced, or A method of time scaling or pitch shifting a plurality of audio signal channels, characterized in that the audio signal is read at a rate of time expansion and pitch shifting in an increased time period at a rate different from the relative change in the number of samples. . In the method of time scaling or pitch shifting an audio signal, Dividing the audio signal into auditory events; Analyzing the auditory events using psychoacoustic criteria to identify auditory events for which time scaling or pitch shifting processing of an audio signal is not audible or minimally audible; And Time scaling or pitch shifting processing within an auditory event identified as time scaling or pitch shifting processing of the audio signal as not audible or minimally audible; Dividing the audio signal into auditory events, Calculating spectral content of successive time segments of the audio signal; Calculating a difference in spectral content between successive time segments of the audio signal; And Setting an auditory event boundary at the boundary between successive time segments if the difference in spectral content between successive time segments exceeds a threshold; Time scaling or pitch shifting the audio signal comprising a. 23. The method of claim 22, wherein the psychoacoustic criteria is one or more of a group of psychoacoustic criteria. The method of claim 23, wherein the psychoacoustic criteria, The identified region of the audio signal is premasked or postmasked as a result of transient; The identified region of the audio signal is not audible; The identified region of the audio signal has higher frequency content than lower frequency content; And The identified region of the audio signal is a segment portion of the audio signal that is quieter than the portion of the segment that precedes or follows the identified region; A method of time scaling or pitch shifting an audio signal comprising one or more of the following. The method of claim 22, wherein the time scaling or pitch shifting processing is performed by: Selecting a junction in the auditory event; Deleting the portion of the audio signal starting at the junction or repeating the portion of the audio signal ending at the junction; And Reading the resulting audio signal at a rate that yields a predetermined time scaling or pitch shifting; Time scaling or pitch shifting the audio signal comprising a. The method of claim 22, wherein the time scaling or pitch shifting processing is performed by: Selecting a junction in the auditory event to define a leading segment of the audio signal leading the junction; Selecting an endpoint spaced apart from the junction in the auditory event to define a trailing segment of the audio signal tracking the endpoint and a target segment of the audio signal between the junction and the endpoint; Connect the leading and trailing segments at the junction to reduce the number of audio signal samples by omitting a target segment if the endpoint has a higher sample number than the junction, or target if the endpoint has a lower sample number than the junction Increasing the number of audio signal samples by repeating the segment; And Reading the connected leading and trailing segments at a rate that yields a predetermined time scaling or pitch shifting; Time scaling or pitch shifting the audio signal comprising a. 27. The system of claim 26, wherein the connected leading and trailing segments are: Pitch-shift the audio signal in the same time period as the original time period, When omitting the target segment, time compress the audio signal in a reduced time period at the same rate as the relative change in the decrease in the number of samples, When repeating the target segment, time-expand the audio signal in an increased time period at the same rate as the relative change in the increase in the number of samples, Time compressing and pitch shifting the audio signal in a reduced time period at a rate different from the relative change in the number of samples reduced, or A method of time scaling or pitch shifting an audio signal, characterized in that it is read at a rate of time expansion and pitch shifting of the audio signal in an increased time period at a rate different from the relative change in the number of samples. 27. The method of claim 26, wherein joining the leading and trailing segments at a junction comprises crossfacing the leading and trailing segments. 27. The method of claim 26, wherein when reducing the number of audio signal samples by omitting a target segment, the endpoint is selected by autocorrelating a segment of audio that tracks the junction. How to. 27. The audio signal of claim 26, wherein when increasing the number of audio signal samples by repeating a target segment, the endpoint is selected by cross correlating the segment of audio leading the junction and the segment of audio tracking the junction. Time scaling or pitch shifting. 27. The junction point and endpoint positions of claim 26, wherein if the number of audio signal samples is increased by omitting the target segment or increasing the number of audio signal samples by repeating the target segment, Correlate a window of audio samples around the series of sample junction locations for an area of audio samples adjacent to each series of sample junction locations, and Time scaling or pitch shifting an audio signal, characterized in that it is selected by determining a sample junction location that results in the strongest correlation, designating the sample junction location as the junction location, and setting the endpoint location to the location of the strongest correlation. How to. 32. The method of claim 31 wherein the window is a rectangular window. 33. The method of claim 32, wherein the window has a width of crossfade. 32. The method of claim 31, wherein the series of sample junction points are spaced apart by one or more audio samples. 35. The method of claim 34, wherein the series of sample junction points are spaced apart by the width of the window. 36. The method of claim 35, wherein the window has a width of crossfade. 32. The method of claim 31, wherein when reducing the number of audio signal samples by omitting a target segment, the region of audio samples adjacent to each series of sample junction positions follows each sample junction position, whereby the junction point is an endpoint. A method of time scaling or pitch shifting an audio signal, characterized by preceding. 32. The method of claim 31, wherein when increasing the number of audio samples by repeating a target segment, the region of audio samples adjacent to each series of sample junction locations follows each sample junction location and the IDs of the junction and endpoint are changed. And thereby the endpoint precedes the junction. 32. The method of claim 31, wherein when increasing the number of audio samples by repeating a target segment, the region of audio samples adjacent to each series of sample junction locations precedes each sample junction location, whereby the endpoint is greater than the junction. A method of time scaling or pitch shifting an audio signal, characterized by the preceding. 27. The junction point and endpoint positions of claim 26, wherein if the number of audio signal samples is increased by omitting the target segment or increasing the number of audio signal samples by repeating the target segment, Correlating a window of audio samples around the series of sample junction locations with respect to the region of audio samples adjacent to each series of sample junction locations and decimated by the factor of M, Determine the sample joint location that gives the strongest correlation, specify the sample joint location as the decimated joint, A window of undecimated audio samples around a second series of sample junction locations within an M sample of the decimated junction for an area of undecimated audio samples adjacent to each second series of sample junction locations. To correlate, and Audio, characterized in that it is selected by determining a sample junction location at the second series of sample junction locations that yield the strongest correlation, designating the sample junction location as the junction, and setting the endpoint location at the location of the strongest correlation. A method of time scaling or pitch shifting a signal. A method of time scaling or pitch shifting multiple channels of audio signals, the method comprising: Dividing an audio signal of each channel into auditory events; Analyzing the auditory events using psychoacoustic criteria to identify auditory events for which time scaling or pitch shifting processing of an audio signal is not audible or minimally audible; Determining combined auditory events, each having a boundary when an auditory event boundary occurs in the audio signal of the channel; And Time scaling or pitch shifting processing within an auditory event identified as time audible or minimally audible in multiple channels of audio signals. Dividing the audio signal of each channel into auditory events, Calculating spectral content of successive time segments of the audio signal of each channel; Calculating a difference in spectral content between successive time segments of the audio signal of each channel; And Setting an auditory event boundary to an audio signal of each channel at the boundary between successive time segments if the difference in spectral content between successive time segments exceeds a threshold; Time scaling or pitch shifting multiple channels of audio signals comprising a. 42. The method of claim 41, wherein the combined auditory event is not audible or based on the psychoacoustic characteristics of the audio in each of the multiple channels during the auditory event time segment in which time scaling or pitch shifting processing of the multiple channels of audio is combined. A method of time scaling or pitch shifting multiple channels of audio signals, characterized as being identified as least audible. 43. The method of claim 42, wherein the psychoacoustic quality grade of the combined auditory event is determined by applying a layer of psychoacoustic criteria to the audio in each of the channels during the combined auditory event. How to time scale or pitch shift. 42. The method of claim 41 wherein the time scaling or pitch shifting processing is: Selecting a common junction within an identified and combined auditory event between channels of audio signals, such that the junctions from the common junction in each of the multiple channels of audio signals are aligned with each other; Deleting a portion of each channel of audio signals starting at the common junction or repeating a portion of each channel of audio signals ending at the common junction; And Reading the resulting channels of audio signals at a rate that yields a predetermined time scaling or pitch shift for the multiple channels of audio; Time scaling or pitch shifting multiple channels of audio signals comprising a. 42. The method of claim 41 wherein the time scaling or pitch shifting processing is: By selecting a common junction within an identified and combined auditory event between channels of audio signals, the junctions from each of the common junctions in each of the multiple channels of audio signals are aligned with each other, with each junction leading an audio signal. Defining a leading segment of; A trailing segment of the audio signal that tracks endpoints such that endpoints derived from the common endpoint in each of the multiple channels of audio signals are aligned with each other by selecting a common endpoint spaced from the common junction within the combined auditory event Defining a target segment of the audio signal between the junction and the endpoint; Connect the leading and trailing segments at each junction in each of the channels of audio signals to reduce the number of audio signal samples by omitting a target segment if the endpoint has a sample number higher than the junction, or the endpoint points to the junction Increasing the number of samples by repeating the target segment if it has a lower sample number; And Reading the connected leading and trailing segments in each of the channels of audio signals at a rate that yields predetermined time scaling or pitch shifting for the multiple channels of audio; Time scaling or pitch shifting multiple channels of audio signals comprising a. The method of claim 23, wherein the psychoacoustic criteria, The identified region of the audio signal is premasked or postmasked as a result of transient; The identified region of the audio signal is not audible; The identified region of the audio signal has higher frequency content than lower frequency content; And The identified region of the audio signal is a segment portion of the audio signal that is quieter than the portion of the segment preceding the identified region and the portion of the segment following the identified region; A method of time scaling or pitch shifting an audio signal comprising one or more of the following. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images