CN118645110A - Audio processing method and medium - Google Patents

Abstract

The embodiment of the application discloses an audio processing method and a medium, wherein the method comprises the following steps: acquiring a first audio signal of a listener; performing noise estimation on the first audio signal, determining a noise estimation result of the first audio signal, and transmitting the noise estimation result of the first audio signal to a sounding party; the noise estimation result is used for indicating the intensity of noise in the first audio signal; receiving an encoded signal sent by a sounding party, performing audio decoding on the encoded signal to obtain a decoded signal, and performing audio playing on the decoded signal; the coding signal is obtained by audio coding the second audio signal by using the coding parameter determined by the noise estimation result after the second audio signal of the sounding party is obtained by the sounding party, the coding parameter can be dynamically adjusted based on the background environment noise of the listener, the setting flexibility of the coding parameter is improved, the change of the background environment noise can be adapted, and the balance of the playing tone quality and the listening environment effect is realized.

Description

Audio processing method and medium

Technical Field

The present application relates to the field of computer technology, and in particular to an audio processing method and medium.

Background Art

In audio coding technology, the coding parameters required for audio coding of audio signals usually affect the audio quality. For example, different coding parameters will produce different sound quality effects. At present, in actual services (such as call services), the coding parameters in the entire process of audio coding are usually set at the call startup stage and will not change. The selected coding parameters will be fixed and used in actual services. It can be seen that the setting flexibility of such coding parameters is poor and cannot be well adapted to the real-time call environment.

Summary of the invention

The embodiments of the present application provide an audio processing method and medium, which can dynamically adjust encoding parameters based on the background environmental noise of the listener, improve the flexibility of encoding parameter setting, and adapt to changes in the background environmental noise of the listener, so as to better adapt to the real-time call (listening) environment and achieve a balance between playback sound quality and listening environment effect.

In a first aspect, an embodiment of the present application provides an audio processing method, comprising:

Acquiring a first audio signal from a listening party; the listening party has established a communication connection with a sounding party;

Performing noise estimation on the first audio signal, determining a noise estimation result of the first audio signal, and sending the noise estimation result of the first audio signal to the speaker; the noise estimation result is used to indicate the intensity of noise in the first audio signal;

Receive the encoded signal sent by the speaker, perform audio decoding on the encoded signal to obtain a decoded signal, and perform audio playback on the decoded signal; the encoded signal is obtained by audio encoding the second audio signal after the speaker obtains the second audio signal of the speaker using the encoding parameters determined by the noise estimation result.

In a second aspect, an embodiment of the present application provides another audio processing method, including:

receiving a noise estimation result sent by a listener; the noise estimation result is obtained by estimating the noise of the first audio signal after the listener acquires the first audio signal of the listener, and the listener has established a communication connection with the sounding party;

Determining encoding parameters for audio encoding based on the noise estimation result;

If the second audio signal of the speaker is obtained, the second audio signal is audio-encoded using the encoding parameters to obtain an encoded signal corresponding to the second audio signal, and the encoded signal is sent to the listener.

In a third aspect, an embodiment of the present application provides an audio processing device, including:

An acquisition unit, configured to acquire a first audio signal from a listener; the listener has established a communication connection with a speaker;

an estimating unit, configured to perform noise estimation on the first audio signal, determine a noise estimation result of the first audio signal, and send the noise estimation result of the first audio signal to the sounding party; the noise estimation result is used to indicate the intensity of noise in the first audio signal;

A decoding unit is used to receive the encoded signal sent by the speaker, perform audio decoding on the encoded signal to obtain a decoded signal, and perform audio playback on the decoded signal; the encoded signal is obtained by the speaker obtaining the second audio signal of the speaker and then performing audio encoding on the second audio signal using the encoding parameters determined by the noise estimation result.

In a fourth aspect, an embodiment of the present application provides another audio processing device, including:

a receiving unit, configured to receive a noise estimation result sent by a listening party; the noise estimation result is obtained by estimating the noise of the first audio signal after the listening party acquires the first audio signal of the listening party, and the listening party has established a communication connection with the sounding party;

A determination unit, configured to determine encoding parameters for audio encoding based on the noise estimation result;

The encoding unit is used to, if a second audio signal of the speaker is acquired, perform audio encoding on the second audio signal using the encoding parameters to obtain an encoded signal corresponding to the second audio signal, and send the encoded signal to the listener.

In a fifth aspect, an embodiment of the present application provides a computer device, comprising: a processor and a memory, wherein the processor is used to execute the method described in the first aspect and/or the second aspect above.

In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which program instructions are stored, and when the program instructions are executed, the method described in the first aspect and/or the second aspect above is implemented.

In the seventh aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect and/or the second aspect is implemented.

The embodiment of the present application can obtain the first audio signal of the listener; and the listener has established a communication connection with the speaker; then, the first audio signal can be noise estimated to determine the noise estimation result of the first audio signal, and the noise estimation result of the first audio signal can be sent to the speaker; the noise estimation result is used to indicate the intensity of the noise in the first audio signal. Further, the listener can receive the encoded signal sent by the speaker, perform audio decoding on the encoded signal, obtain the decoded signal, and perform audio playback on the decoded signal; the encoded signal is obtained by the speaker obtaining the second audio signal of the speaker and performing audio encoding on the second audio signal using the encoding parameters determined by the noise estimation result. Based on the above method, the background environmental noise of the listener can be estimated, and the noise estimation result can be fed back to the speaker, so that the speaker can adjust the encoding parameters according to the feedback noise estimation result to adapt to the changes in the background environmental noise of the listener, so as to achieve the best match between the playback sound quality and the listening environment effect.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of the architecture of an audio processing system provided in an embodiment of the present application;

FIG2 is a flow chart of an audio processing method provided in an embodiment of the present application;

FIG3 is a schematic diagram of an interface of a business scenario provided in an embodiment of the present application;

FIG4 is a flow chart of another audio processing method provided in an embodiment of the present application;

FIG5 is a flow chart of another audio processing method provided in an embodiment of the present application;

FIG6 is a curve diagram representing a mapping relationship provided in an embodiment of the present application;

FIG7 is a schematic diagram of the structure of an audio processing device provided in an embodiment of the present application;

FIG8 is a schematic diagram of the structure of an audio processing device provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

Cloud Technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, allowing various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". From the user's perspective, the resources in the "cloud" are infinitely scalable and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

The present application can store the data required for audio processing in the "cloud", and the data in the cloud can be obtained and expanded at any time according to needs. For example, the noise estimation result can be associated with the encoding parameters and stored in the "cloud". If the noise estimation result is obtained later, the encoding parameters corresponding to the noise estimation result can be obtained from the "cloud", and then audio encoding can be performed based on the encoding parameters.

Please refer to Figure 1, which is a system architecture diagram of an audio processing provided by an embodiment of the present application. The present application involves at least one speaker and at least one listener. Among them, any speaker can establish a communication connection with any listener to achieve mutual communication between the speaker and the listener. Among them, the communication connection can be a connection for audio and video, that is, the speaker and the listener can have an audio and video call.

The listening party and the speaking party may be terminals, and the terminals may be devices such as smart phones, tablet computers, laptop computers, and desktop computers. For example, the listening party and the speaking party may use smart phones to make a phone call. Alternatively, the listening party and the speaking party may also be clients configured on the terminal, such as clients with audio and video call functions, such as game clients, instant messaging clients, etc. For example, the listening party and the speaking party may make an audio call through an instant messaging client.

As shown in FIG1 , after the speaker and the listener establish a communication connection, the listener can obtain the background environment audio of the communication environment in which the listener is located. Usually, the background environment audio may contain background noise (environmental noise), such as white noise, babble noise (multi-path overlap noise), car horn, gray noise, etc. After the listener obtains the background environment audio, it can estimate the noise level based on the background environment audio, and feed back the estimated noise estimation result (or noise level result) to the speaker.

After obtaining the noise estimation result, the speaker can adjust (or modify, adjust, update) the coding parameters of the audio coding (such as one or more of the coding bit rate, sampling rate, number of channels, etc.). It should be understood that the default coding parameters required for audio coding can be configured at the beginning of the communication. In one embodiment, the speaker can determine the coding parameters based on the obtained noise estimation result, and use the coding parameters to adjust the default coding parameters, such as adjusting the default coding parameters to the coding parameters obtained by noise estimation. That is, the coding parameters of the speaker can be dynamically adjusted according to the noise estimation result to adapt to the changes in the background noise environment of the listener, so as to achieve the best match between the playback sound quality and the listening environment effect.

When the speaker obtains the input audio signal, the audio signal can be encoded based on the encoding parameters to obtain an encoded signal; then, the speaker can send the encoded signal to the listener so that the listener can perform audio decoding on the encoded signal to obtain a decoded signal, which is the original input audio signal. In one embodiment, the listener can also output the audio signal, such as playing the audio signal through a speaker on the listener so that a user using the listener can hear the audio signal.

In one embodiment, for example, taking a Voice over Internet Protocol (VoIP) scenario based on IP as an example, at the beginning of a call, the default encoding parameters required for audio encoding can be configured first, and then, in the subsequent call process, the listener can estimate the background environment noise of the listener and feed back the noise estimation result to the speaker. The speaker can adjust the encoding parameters required for audio encoding according to the feedback noise estimation result, such as adjusting the default encoding parameters to the encoding parameters corresponding to the noise estimation result, so as to adapt to the changes in the background noise environment of the listener, so as to achieve the best match between the playback sound quality and the listening environment effect.

In specific application scenarios, the audio processing system provided in the embodiments of the present application can be used in scenarios where audio encoding of audio signals is required, such as audio and video calls, multi-person audio and video, conferences, karaoke, live broadcasts, games, and the like.

The implementation details of the technical solution of the embodiment of the present application are described in detail below:

Please refer to FIG. 2, which is a flow chart of an audio processing method provided in an embodiment of the present application. The audio processing method involves a listening party and a sounding party. This embodiment mainly describes the interaction process between the listening party and the sounding party. The audio processing method includes the following steps:

S201: A listener obtains a first audio signal of the listener.

The listening party has established a communication connection with the speaking party. The first audio signal may include: background environment audio of the communication environment where the listening party is located, or may also include: voice audio emitted by the user through the listening party at the current time.

In one implementation, the listener may obtain the first audio signal corresponding to the listener through a first audio acquisition device. For example, the first audio acquisition device may be a device with an audio acquisition function, such as a microphone on the listener.

Optionally, the number of audio acquisition devices may be one or more. If the number of audio acquisition devices is multiple, the first audio signal may be an audio signal acquired by any one or more of the multiple audio acquisition devices. In one embodiment, when there are multiple audio acquisition devices (such as microphones) on the listener, the audio signal acquired by the audio acquisition device may be selected as the first audio signal based on the call scenario applicable to each audio acquisition device. The call scenario applicable to an audio acquisition device here may be understood as: in this call scenario, the audio acquisition device acquires the audio signal best. For example, among the multiple audio acquisition devices, there are audio acquisition devices suitable for audio acquisition in hands-free and video call scenarios, and there are also audio acquisition devices suitable for audio acquisition in non-hands-free and non-video call scenarios.

Based on this, it can be known that the call scene at the current time can be first obtained, and the audio acquisition device matching the call scene can be determined, and then the audio signal acquired by the audio acquisition device matching the call scene can be used as the first audio signal. Among them, the specific implementation method of determining the audio acquisition device matching the call scene can be: first obtain the mapping relationship between the reference call scene and the reference audio acquisition device, and the mapping relationship can be pre-set; then, the audio acquisition device matching the call scene can be determined from the mapping relationship.

It should be noted that the specific implementation of subsequently obtaining the second audio signal of the speaker at the speaker can be understood in the same way.

In one implementation, after the listener establishes a communication connection with the speaker, the listener and the speaker can communicate with each other. For example, in scenarios such as audio and video calls, multi-person audio and video, conferences, karaoke, live broadcasts, games, etc., after the listener establishes a communication connection with the speaker, the listener and the speaker can communicate with each other. As shown in FIG3 , the interface 31 in the figure shows the scene of audio and video calls, the interface 32 shows the scene of multi-person audio and video, the interface 33 shows the live broadcast scene, and the interface 34 shows the game scene. Exemplarily, in a live broadcast scene, the live broadcast user can broadcast live through the speaker, and other viewing users can watch the live broadcast through the listener. The speaker and the listener here can specifically be a terminal or client with a live broadcast function. In the process of watching the live broadcast, these viewing users can initiate a microphone connection request through the listener. After the sound-emitting device responds to the microphone connection request, a communication connection between the speaker and the listener can be established, thereby realizing a voice call between the live broadcast user and the viewing user.

In one implementation, the listener may detect in real time whether the current time is the preset noise estimation time. If it is detected that the current time is the preset noise estimation time, the listener's first audio signal may be acquired to perform noise estimation using the acquired first audio signal.

The preset noise estimation time may be a preset setting, and the number of the preset noise estimation time may be one or more. For example, when the number of the preset noise estimation time is one, the preset noise estimation time may be set at any time point after the communication connection is established, and may usually be set at a time point as close to a reference time as possible, the reference time referring to the time when the communication connection is established, so that the encoding parameters can be adjusted in a timely manner according to the environment in which the listener is located, thereby improving the listening experience.

In the case where there are multiple preset noise estimation times, these multiple preset noise estimation times can be periodic time points, that is, the listener can perform periodic noise estimation, so that the encoding parameters required for the sound emitter to perform audio encoding can be dynamically adjusted according to the periodic real-time noise estimation results to adapt to the changes in the background noise environment of the listener, so as to achieve the best match between the playback sound quality and the listening environment effect. Based on this, it can be known that the audio signal of the listener can be obtained with a reference period, wherein the reference period can be used to indicate the time interval for obtaining the audio signal. For example, the reference period can be 20ms, that is, the audio signal of the listener can be obtained every 20ms.

In one implementation, when a communication scene switch is detected, the first audio signal of the listener can be obtained to reduce the computational overhead. For example, the communication scene switch may refer to the switch between a voice call and a video call; exemplarily, when the current time switches from a voice call to a video call, or from a video call to a voice call, the operation of obtaining the first audio signal of the listener can be triggered. For another example, the communication scene switch may refer to the switch between a two-person call and a multi-person (more than two people) call; exemplarily, when the current time switches from a two-person voice call (such as the interface 31 in FIG. 3 showing two audio and video call scenes) to a four-person voice call (the interface 32 in FIG. 3 showing four audio and video call scenes), or from a four-person voice call to a two-person voice call, the operation of obtaining the first audio signal of the listener can be triggered. For another example, the communication scene switch may refer to the switch of the audio playback type; exemplarily, when the current time switches from the voice audio type to the music audio type, or from the music audio type to the voice audio type, the operation of obtaining the first audio signal of the listener can be triggered.

S202: The listener performs noise estimation on the first audio signal to determine a noise estimation result of the first audio signal.

The noise estimation result may be used to indicate the intensity of noise in the first audio signal.

Considering that the ultimate beneficiary of the audio call itself is the listener, and the listener's audio experience is strongly related to the acoustic environment of the listener in addition to the sound quality of the playback signal itself. For example, if the listener is in a relatively quiet environment (such as a conference room), when there is a slight sound quality problem with the playback signal, the listener may easily distinguish it, thus affecting the listening effect; on the contrary, if the listener is in a relatively noisy environment (such as a shopping mall or a bar), the overlapped part of the audio signal played and the ambient noise in the frequency domain spectrum at this time, due to the acoustic masking effect, the ambient noise in the listener's environment will mask the flaws of the played audio signal, and the subtle differences in the sound quality of the played audio signal will be difficult for the listener to distinguish. In this case, the difference in the encoding parameters of the audio encoding (such as the encoding bit rate) has little effect on the difference in listening sound quality. From the above, it can be seen that we can consider analyzing the ambient noise of the listener, so as to use the noise level of the ambient noise of the listener to adjust the encoding parameters required for audio encoding, so that the encoding parameters can adapt to the changes in the ambient noise of the listener and achieve the best match between the playback sound quality and the listening environment effect.

In the embodiment of the present application, the encoding parameter adjusted by the noise level of the ambient noise of the listener may be one or more of the encoding bit rate, sampling rate, number of channels, and the like.

For example, in terms of encoding bit rate, it should be understood that the encoding bit rate in audio processing is an important encoding parameter of the audio encoder, and the encoding bit rate is strongly related to the compression ratio of the audio input signal. If the input audio signal is configured with a lower encoding bit rate, it can bring greater data compression, but the playback sound quality (sound quality) will usually decrease; and the higher the encoding bit rate, the better the encoding quality of the audio signal, then the quality of the decoded output audio signal is higher, that is, the playback sound quality is better, but the compression ratio of the audio signal is not high, the larger the storage occupied by the encoding file, and the higher the encoding bit rate, the more bandwidth the corresponding encoding stream occupies. Therefore, in order to improve the quality of communication, a suitable encoding bit rate can usually be configured to achieve audio encoding of the audio signal. Based on this, it can be seen that the encoding bit rate required for audio encoding can be adjusted in real time according to the feedback of the noise level of the listener, so as to achieve the purpose of effectively utilizing the encoding capacity and transmission bandwidth, and achieve the best balance between the playback sound quality experience effect of the audio signal and the operating cost (such as the cost of audio transmission bandwidth and storage).

For another example, in terms of sampling rate, the sampling rate and encoding quality can be positively correlated, that is, the higher the sampling rate, the better the encoding quality of the audio signal, that is, the better the playback quality of the audio signal; correspondingly, the lower the sampling rate, the worse the encoding quality of the audio signal, that is, the worse the playback quality of the audio signal. It should be understood that a lower sampling rate means fewer samples per second, which means less audio data is obtained; it is also understandable that the higher the sampling rate, the more storage space and stronger processing power are required. Based on this, it can be seen that the sampling rate required for audio encoding can be adjusted in real time according to the feedback of the noise level of the listener to achieve the best balance between the playback sound quality experience effect of the audio signal and the amount of signal sampling data. The amount of sampling data can directly affect the storage space and processing power of the data, so the best balance between the playback sound quality experience effect of the audio signal and the storage space and processing power can also be achieved.

For another example, in terms of the number of channels (number of channels), the number of channels and the playback sound quality of the audio signal (or the playback sound quality experience effect) can be positively correlated. Generally, the more channels there are, the stronger the stereoscopic sense and presence of the audio signal. That is, the larger the number of channels, the better the playback sound quality of the audio signal; correspondingly, the smaller the number of channels, the worse the playback sound quality of the audio signal. The number of channels here can include the first number of channels and the second number of channels. The first number of channels can be understood as a single channel (mono), and the second number of channels can be understood as a dual channel (dual channel). It should be understood that channels usually refer to independent audio signals collected at different spatial positions when the sound is recorded, so the number of channels can also be understood as the number of sound sources when the sound is recorded; the larger the number of channels, the higher the complexity of obtaining the audio signal. Based on this, it can be seen that the number of channels required for audio encoding can be adjusted in real time according to the feedback of the noise level of the listener to achieve the best balance between the playback sound quality experience effect of the audio signal and the complexity of signal acquisition.

S203: The listener sends the noise estimation result of the first audio signal to the speaker.

In one implementation, the listener may send the noise estimation result of the first audio signal to the speaker, so that the speaker may adjust the encoding parameters required for audio encoding in real time based on the noise estimation result.

S204: The speaker receives the noise estimation result sent by the listener.

As mentioned above, the noise estimation result is obtained by performing noise estimation on the first audio signal after the listener obtains the first audio signal of the listener.

S205: The speaker determines encoding parameters for audio encoding based on the noise estimation result.

Among them, audio coding is to perform redundant analysis and compression of the original lossless audio signal collected in the time domain and frequency domain, and the compression process is to preserve the main or all information in the audio. Through compression processing, the amount of audio data can be reduced, thereby reducing the audio transmission bandwidth and storage space, and thereby reducing the cost of audio transmission and storage; at the same time, good audio quality can also be maintained. Among them, the coding parameters involved in audio coding may include: sampling rate, number of channels, coding bit rate, etc. The embodiment of the present application considers using the noise level of the listener to adjust the coding parameters required for audio coding, so as to adapt the appropriate coding parameters to achieve audio coding of the audio signal and improve the listening effect of the listener.

In one implementation, a mapping relationship between a reference noise estimation result and a reference coding parameter may be preset, and the mapping relationship may be a better match between the noise estimation result and the coding parameter obtained through a large number of tests. After receiving the noise estimation result sent by the listener, the speaker may obtain the mapping relationship to determine the coding parameter corresponding to the noise estimation result using the mapping relationship. For example, a reference noise estimation result consistent with the noise estimation result may be searched from the mapping relationship, and the reference coding parameter corresponding to the found reference noise estimation result may be determined as the coding parameter.

It should be understood that the default encoding parameters are usually configured at the beginning of the communication. In the embodiment of the present application, if the speaker does not adjust the default encoding parameters, the default encoding parameters can be used for audio encoding. In one embodiment, after the speaker obtains the encoding parameters, the default encoding parameters can be directly replaced with the encoding parameters. In another embodiment, the default encoding parameters and the encoding parameters can also be retained, and a corresponding estimated time can be set for the encoding parameters. The estimated time can refer to the time when the encoding parameters are obtained. In the subsequent audio encoding, the current latest encoding parameters can be determined based on the estimated time of the encoding parameters, so that the latest encoding parameters can be used for audio encoding.

S206: If the speaker obtains the second audio signal of the speaker, audio encoding is performed on the second audio signal using the encoding parameter to obtain an encoded signal corresponding to the second audio signal.

It is understandable that a user using the speaking party can initiate voice through the speaking party to achieve mutual communication with a user using the listening party. If the speaking party obtains an audio signal corresponding to the speaking party, the audio encoding can be performed using the encoding parameters. For the convenience of description, the audio signal here can be referred to as a second audio signal, and the second audio signal is the voice emitted by the user through the speaking party. In one implementation, the speaking party can obtain the second audio signal of the speaking party through a second audio acquisition device. For example, the second audio acquisition device can be a device with an audio acquisition function, such as the second audio acquisition device can be a microphone on the speaking party.

It can be understood that the encoding parameter is obtained based on the communication environment in which the listener is located. Compared with using a default (fixed) encoding parameter to perform audio encoding on all audio signals in the entire communication connection, the embodiment of the present application can adjust the default encoding parameter based on the noise level of the listener to achieve audio encoding of the audio signal using the encoding parameter adapted to the noise level of the listener.

In one implementation, after acquiring the second audio signal, the second audio signal may be audio-encoded using the encoding parameters to compress the second audio signal and obtain a compressed second audio signal. For example, the compressed second audio signal may be referred to as an encoded signal.

S207, the speaker sends the encoded signal to the listener.

S208, the listening party receives the coded signal sent by the speaking party, performs audio decoding on the coded signal to obtain a decoded signal, and performs audio playback on the decoded signal.

Among them, audio decoding is to decode the compressed audio signal (i.e., the encoded signal) to restore the original audio signal, and the original audio signal can be called a decoded signal. As mentioned above, the encoded signal is obtained by the speaker obtaining the second audio signal of the speaker and then performing audio encoding on the second audio signal using the encoding parameters determined by the noise estimation result.

In one implementation, after the listener decodes and obtains the decoded signal, the decoded signal can be played as an audio so that the user can hear the voice of the speaker. For example, the decoded signal can be played by a device with a play function on the listener. For example, if the listener is a mobile phone, the device with a play function can be a speaker on the mobile phone.

In one implementation, when there are multiple listeners, each listener can estimate their own background environmental noise and feed back their own noise estimation results to the speaker, and the speaker can adjust the default encoding parameters based on their own noise estimation results to obtain encoding parameters suitable for different listeners, so that the audio signal output by the codec can be adapted to different listeners, thereby improving the multi-person voice experience.

In an embodiment of the present application, the listener can estimate the background environmental noise of the listener and feed back the noise estimation result to the speaker, so that the speaker can adjust the encoding parameters according to the feedback noise estimation result to adapt to the changes in the background noise environment of the listener, so as to achieve the best match between the playback sound quality and the listening environment effect.

Please refer to FIG. 4, which is a flowchart of another audio processing method provided in an embodiment of the present application. This embodiment mainly describes a specific implementation process of noise estimation on the first audio signal by the listener side. The audio processing method described in this embodiment includes the following steps:

S401: Acquire a first audio signal of a listener.

The specific implementation of this step can refer to the description in the above step S201, which will not be repeated here.

S402: Perform noise estimation on the first audio signal, determine a noise estimation result of the first audio signal, and send the noise estimation result of the first audio signal to a speaker.

The noise estimation result may be used to indicate the intensity of noise in the first audio signal.

In one implementation, the listener may estimate the background environmental noise of the listener and feed back the noise estimation result to the speaker, so that the speaker may adjust the encoding parameters required for audio encoding according to the fed-back noise estimation result.

In one implementation, the first audio signal may be firstly processed by frame division to obtain M audio frames; then, noise estimation may be performed on each audio frame to obtain a noise estimation result for each audio frame. It should be noted that after obtaining the noise estimation result of each audio frame, the noise estimation result may be fed back to the speaker so that the speaker can adjust the encoding parameters in a timely manner. Based on this, it can be known that each audio frame in the M audio frames can be traversed. If the mth (m∈[1,M]) audio frame in the M audio frames is traversed, the mth audio frame may be noise estimated to obtain the noise estimation result of the mth audio frame, and the noise estimation result of the mth audio frame may be used as the noise estimation result of the first audio signal, so as to feed back the noise estimation result of the first audio signal to the speaker, that is, it can be understood that the noise estimation result of the first audio signal is also updated in real time based on the noise estimation result of the obtained audio frame.

The specific value of M is not limited, such as 20; the frame length of an audio frame is not limited, such as 15ms. When dividing the frames, there may be a time overlap between two adjacent audio frames, such as a 30% overlap between frames; or there may be no time overlap, which is not limited.

In one implementation, it should be understood that in noise estimation, the power spectrum of the signal is used to perform relevant calculations to obtain the corresponding noise estimation results, and the mth audio frame can be converted from the time domain to the frequency domain. Based on this, it can be seen that the above-mentioned noise estimation of the mth audio frame to obtain the noise estimation result of the mth audio frame can be specifically implemented by first performing Fourier transform processing on the mth audio frame to obtain the power spectrum of the mth audio frame. In simple terms, the collected first audio signal is Fourier transformed to obtain the complex values of each frequency point in the frequency domain. Then, the noise estimation of the mth audio frame can be performed based on the power spectrum of the mth audio frame to determine the noise estimation result of the mth audio frame.

In one implementation, when performing noise estimation on the mth audio frame, the power spectrum corresponding to the mth audio frame may be divided into sub-bands, so as to determine the noise estimation result of the mth audio frame based on the noise evaluation of each sub-band. Based on this, it can be known that performing noise estimation on the mth audio frame based on the power spectrum of the mth audio frame and determining the noise estimation result of the mth audio frame may be specifically implemented as the following steps S01-S03:

S01 , dividing the power spectrum of the m th audio frame into K sub-bands, and determining a sub-band power spectrum of each sub-band in the K sub-bands.

The value of K is not specifically limited, for example, it can be 10, 15, etc.

In one implementation, when performing sub-band division, it can be equal bandwidth sub-band division or unequal bandwidth sub-band division. For equal bandwidth sub-band division, the bandwidth of each sub-band is the same. For unequal bandwidth sub-band division, the bandwidth of each sub-band may be different; in this case, when performing sub-band division, the width of each sub-band may increase with the increase of K, that is, the bandwidth of the sub-band in the low frequency band is narrow, and the bandwidth of the sub-band in the high frequency band is wide.

In one implementation, considering that the method for determining the subband power spectrum of each subband in the K subbands is the same, the determination of the subband power spectrum is described below using the kth subband as an example: for the kth subband in the K subbands, first, the number of frequency points in the kth subband can be obtained; for example, the starting frequency point index value and the ending frequency point index value of the kth subband can be obtained to determine the number of frequency points in the kth subband based on the starting frequency point index value and the ending frequency point index value. Then, the subband power spectrum of the kth subband can be determined based on the number of frequency points in the kth subband and the frequency domain values corresponding to each frequency point in the kth subband. Under this implementation, the computer device can calculate the subband power spectrum of the kth subband of the mth audio frame by the following formula (1):

Where m corresponds to the serial number of the audio frame, k corresponds to the serial number of the subband, and z corresponds to the frequency index value in the subband. freq ₁ (k) represents the starting frequency index value of the kth subband, freq ₂ (k) represents the ending frequency index value of the kth subband; freq ₂ (k)-freq ₁ (k)+1 represents the number of frequency points in the kth subband. X(m,z) is the frequency domain value (complex value in the frequency domain) of the zth frequency point in the i-th audio frame after Fourier transform.

S02: performing noise estimation on the corresponding subband based on the subband power spectrum of each subband, and determining a noise estimation result of the corresponding subband.

Among them, considering that the noise estimation process for each subband of the mth audio frame is the same, the kth subband among the K subbands can be taken as an example to explain the noise estimation process of the subband. Its specific implementation can refer to the following steps S11-S14. The specific implementation of steps S11-S14 is the specific implementation of subband noise power spectrum estimation. Through this specific implementation, a more accurate noise power spectrum estimation value can be obtained.

S11 , for a k-th subband among the K subbands, performing time-frequency domain smoothing processing on a subband power spectrum of the k-th subband to obtain a smoothed subband power spectrum of the k-th subband.

In one implementation, the subband power spectrum of the kth subband may be first subjected to frequency domain smoothing of adjacent subbands, and then the frequency domain smoothing result may be subjected to time domain smoothing of historical audio frames to obtain a corresponding time domain smoothing result, and the time domain smoothing result may be used as the smoothed subband power spectrum of the kth subband. For specific implementation, see the following description.

In one implementation, the frequency domain smoothing process may be:

First, a subband power spectrum corresponding to each of the r adjacent subbands adjacent to the kth subband may be obtained.

Wherein, adjacent may refer to one or more of forward adjacent and backward adjacent, that is, the r adjacent subbands may be the subbands that are forward adjacent to the kth subband, such as the adjacent subband may be the k-1th subband; or, the r adjacent subbands may be the subbands that are backward adjacent to the kth subband, such as the adjacent subband may be the k+1th subband; or, the r adjacent subbands may be the subbands that are both forward adjacent and backward adjacent to the kth subband, such as the adjacent subband may be the k-1th subband and the k+1th subband. Wherein, the value of r is not limited, such as 1, 2, etc. When adjacent refers to forward adjacent and backward adjacent, usually the number of forward adjacent subbands is the same as the number of backward adjacent subbands. For example, if there are 2 forward adjacent subbands, there are also 2 backward adjacent subbands.

Then, based on the subband power spectra corresponding to each adjacent subband, the subband power spectrum of the kth subband can be smoothed in the frequency domain to obtain the smoothed subband power spectrum of the kth subband in the frequency domain. Optionally, before performing the frequency domain smoothing process, a frequency domain smoothing factor group can also be obtained to perform frequency domain smoothing process on the subband power spectra corresponding to each adjacent subband and the subband power spectrum of the kth subband based on the frequency domain smoothing factor group to obtain the smoothed subband power spectrum of the kth subband in the frequency domain. The frequency domain smoothing factor group can include the frequency domain smoothing factors corresponding to each adjacent subband and the kth subband. The above frequency domain smoothing process can be understood as performing weighted summation process on the subband power spectra of the corresponding subbands using each frequency domain smoothing factor, so as to obtain the smoothed subband power spectrum of the kth subband in the frequency domain.

In this implementation manner, the computer device may calculate the smoothed sub-band power spectrum of the k-th sub-band of the m-th audio frame in the frequency domain by using the following formula (2):

in, represents the smoothed subband power spectrum of the kth subband of the mth audio frame in the frequency domain, w represents the number of unidirectionally adjacent (such as forward adjacent or backward adjacent) subbands, and 2w represents the total number of adjacent subbands (i.e., r). b(j+w) represents a frequency domain smoothing factor group, and each frequency domain smoothing factor in the frequency domain smoothing factor group is the frequency domain smoothing factor of each adjacent subband and the kth subband in turn. For example, if w is 2, then b(j+w)=[b(0), b(1), b(2), b(3), b(4)], and b(0), b(1), b(2), b(3), b(4) are the frequency domain smoothing factors of the k-2th subband, the k-1th subband, the kth subband, the k+1th subband, and the k+2th subband in turn. For example, b(j+w)=[0.1, 0.2, 0.4, 0.2, 0.1]. S(m,k+j) represents the subband power spectrum of the k+jth subband of the mth audio frame.

In one implementation, the process of time domain smoothing may be:

First, the smoothed subband power spectrum of the kth subband of the m-1th audio frame can be obtained. It can be understood that the process of calculating the noise estimation results of M audio frames is an iterative process, that is, starting from the 1st audio frame, the noise estimation results of each audio frame are calculated in sequence. When the smoothed subband power spectrum of the kth subband of the mth audio frame is calculated, the smoothed subband power spectrum of the kth subband of the m-1th audio frame has been calculated, and the smoothed subband power spectrum of the kth subband of the m-1th audio frame can be directly obtained.

Then, time domain smoothing can be performed based on the smoothed subband power spectrum of the kth subband of the m-1th audio frame and the smoothed subband power spectrum of the kth subband in the frequency domain to obtain the smoothed subband power spectrum of the kth subband in the time domain, and the smoothed subband power spectrum of the kth subband in the time domain can be used as the smoothed subband power spectrum of the kth subband.

Optionally, before performing the time domain smoothing process, a time domain smoothing factor may be obtained to perform time domain smoothing process on the smoothed subband power spectrum of the kth subband of the m-1th audio frame and the smoothed subband power spectrum of the kth subband of the mth audio frame in the frequency domain based on the time domain smoothing factor, so as to obtain the smoothed subband power spectrum of the kth subband of the mth audio frame in the time domain. In one embodiment, the time domain smoothing factor may be multiplied with the smoothed subband power spectrum of the kth subband of the m-1th audio frame, and the difference between 1 and the time domain smoothing factor may be multiplied with the smoothed subband power spectrum of the kth subband of the mth audio frame in the frequency domain; the two product operation results may be summed, and the summed result may be used as the smoothed subband power spectrum of the kth subband of the mth audio frame in the time domain, that is, the smoothed subband power spectrum of the kth subband of the mth audio frame. In this implementation manner, the computer device may calculate the smoothed sub-band power spectrum of the k-th sub-band of the m-th audio frame by the following formula (3):

in, represents the smoothed subband power spectrum of the kth subband of the mth audio frame, represents the smoothed subband power spectrum of the kth subband of the m-1th audio frame, c ₀ (c ₀ <1) represents a time domain smoothing factor, for example, c ₀ =0.9.

S12, obtaining a preset period, and determining a minimum subband power spectrum of the kth subband for the mth audio frame based on the smoothed subband power spectrum of the kth subband and the preset period.

Among them, this step is mainly to find the minimum sub-band power spectrum within a time window. For example, the time window is the preset period mentioned here. The preset period can be T. The T value can be used to indicate to find the minimum sub-band power spectrum every T audio frames, that is, to search for the minimum value within T audio frames, and use the search result as a rough estimate of the noise of the kth sub-band. For example, assuming T is 5, the minimum sub-band power spectrum is found every 5 audio frames.

In one implementation, for the kth subband of the mth audio frame, a modulo process may be performed on k and T, and a processing result may be determined. The processing result may include a specified value and a non-specified value, wherein the specified value indicates a value of 0, and the non-specified value indicates a value that is not 0.

If the processing result is a specified value, the minimum subband power spectrum of the previous period can be obtained, and the minimum value of the minimum subband power spectrum of the previous period and the smoothed subband power spectrum of the kth subband of the mth audio frame can be used as the minimum subband power spectrum for the kth subband of the mth audio frame.

If the processing result is not a specified value, the minimum subband power spectrum of the kth subband for the m-1th audio frame can be obtained, and the minimum value of the smoothed subband power spectrum of the kth subband and the minimum subband power spectrum of the kth subband for the m-1th audio frame can be used as the minimum subband power spectrum of the kth subband for the m-1th audio frame.

For example, assuming that T is 5, the minimum subband power spectrum is searched for every 5 audio frames. If k is 7, the result of the modulo processing of k and T is a non-specified value (i.e., not 0), then the minimum subband power spectrum of the 6th subband of the mth audio frame can be obtained, and the minimum value of the minimum subband power spectrum of the 6th subband of the mth audio frame and the smooth subband power spectrum of the 7th subband is used as the minimum subband power spectrum of the 7th subband of the mth audio frame. If k is 10, the result of the modulo processing of k and T is a specified value (i.e., 0), then the minimum subband power spectrum between the 6th subband and the 9th subband of the mth audio frame can be obtained, and the minimum value of the minimum subband power spectrum between the minimum subband power spectrum and the smooth subband power spectrum of the 10th subband is used as the minimum subband power spectrum of the 10th subband of the mth audio frame.

The above-mentioned implementation method of determining the minimum sub-band power spectrum of the k-th sub-band for the m-th audio frame can be called a minimum value search (search) method, and the processing logic of the minimum value search (search) method can be shown in the following code:

Wherein, T represents a preset period, S _min (m, k) represents the minimum subband power spectrum of the kth subband of the mth audio frame, and S _tmp (m, k) represents an intermediate variable. It means: the minimum subband power spectrum of the previous period and the minimum value of the smoothed subband power spectrum of the k-th subband of the m-th audio frame are used as the minimum subband power spectrum of the k-th subband of the m-th audio frame. It means: the minimum subband power spectrum of the k-th subband of the m-1-th audio frame and the minimum value of the smoothed subband power spectrum of the k-th subband of the m-th audio frame are used as the minimum subband power spectrum of the k-th subband of the m-th audio frame.

S13, determining a speech probability that speech exists in the kth subband of the mth audio frame based on the smoothed subband power spectrum and the minimum subband power spectrum.

First, the signal-to-noise ratio can be determined based on the smoothed subband power spectrum and the minimum subband power spectrum. For example, the ratio between the smoothed subband power spectrum and the minimum subband power spectrum can be used as the signal-to-noise ratio. In this implementation, the computer device can calculate the signal-to-noise ratio using the following formula (4):

Wherein, S _r (m,k) represents the signal-to-noise ratio.

Then, the signal-to-noise ratio can be compared with the threshold, and the initial speech probability of speech existing in the kth subband of the mth audio frame can be determined based on the comparison result. Wherein, the threshold is a ratio threshold value, which can be used to determine whether speech exists; the threshold can be preset, and its specific value is not limited; the threshold can be an empirical value, and generally, for a quieter communication environment, the threshold can be set larger, and for a noisier communication environment, the threshold can be set smaller. In a specific implementation, if the comparison result is that the signal-to-noise ratio is greater than the threshold, the initial speech probability of speech existing in the kth subband of the mth audio frame can be determined as a first probability; if the comparison result is that the signal-to-noise ratio is less than or equal to the threshold, the initial speech probability of speech existing in the kth subband of the mth audio frame can be determined as a second probability. Wherein, the specific value of the first probability can be 1, and the specific value of the second probability can be 0. In this embodiment, the computer device can calculate the initial speech probability of speech existing in the kth subband of the mth audio frame by the following formula (5):

Wherein, p(m,k) represents the initial speech probability that speech exists in the k-th subband of the m-th audio frame, and δ represents the threshold.

Finally, the speech probability of speech existing in the kth subband of the m-1th audio frame can be obtained, and based on the speech probability corresponding to the kth subband of the m-1th audio frame and the initial speech probability, the speech probability of speech existing in the kth subband of the m-1th audio frame can be determined. In one embodiment, a smoothing factor for the speech existence probability can also be obtained, and the smoothing factor can be understood as a weight, that is, there is a weight between the probability of speech existing in the current audio frame and the probability of speech existing in the previous audio frame. Based on this, it can be known that the speech probability of speech existing in the kth subband of the m-1th audio frame can be determined based on the smoothing factor, the speech probability corresponding to the kth subband of the m-1th audio frame and the initial speech probability.

In this implementation, the computer device may calculate the speech probability that speech exists in the kth subband of the mth audio frame by the following formula (6):

in, represents the speech probability of speech in the k-th subband of the m-th audio frame, α _p (α _p <1) represents the smoothing factor for the speech probability, represents the speech probability that speech exists in the k-th subband of the m-1-th audio frame.

S14, determining a noise estimation result of the kth subband of the mth audio frame based on the speech probability, the subband power spectrum of the kth subband of the mth audio frame, and the noise estimation result of the kth subband of the (m-1)th audio frame.

The noise estimation result of the k-th subband is also the noise power spectrum estimation value of the k-th subband. In this implementation, the computer device can calculate the noise estimation result of the k-th subband of the m-th audio frame by the following formula (7):

in, represents the noise estimation result of the kth subband of the mth audio frame, represents the noise estimation result of the kth subband of the m-1th audio frame,

S03: Perform noise estimation on the mth audio frame based on the noise estimation results corresponding to the sub-bands included in the mth audio frame to determine the noise estimation result of the mth audio frame.

In one implementation, after obtaining the noise estimation results corresponding to each subband contained in the mth audio frame, the overall noise estimation of the mth audio frame can be achieved based on the noise estimation results corresponding to each subband, that is, the overall noise estimation result of the mth audio frame needs to be determined. Among them, the noise estimation result of the mth audio frame refers to the overall noise estimation result of the mth audio frame, and the noise estimation result can be used to indicate the noise intensity (noise level) of the mth audio frame. The noise estimation result can be a specific numerical value, and the noise estimation result can be called a noise estimation value, a noise intensity value, a noise level value, etc.

In one implementation, noise estimation may be performed on the mth audio frame based on noise estimation results corresponding to all subbands included in the mth audio frame to determine the noise estimation result of the mth audio frame. Noise estimation may also be performed on the mth audio frame based on noise estimation results corresponding to some subbands included in the mth audio frame to determine the noise estimation result of the mth audio frame.

In the embodiment of the present application, the noise estimation of the mth audio frame using some sub-bands is specifically described. The specific implementation of the noise estimation of the mth audio frame using the noise estimation results corresponding to all sub-bands is similar to that using some sub-bands, except that the sub-band screening operation mentioned later is not required. For example, only the sub-band range overlapping with the main spectrum range of human voice can be counted, for example, the sub-band range can be 100hz~6khz. That is, only the noise estimation results of the sub-bands within the sub-band range are used to estimate the noise of the mth audio frame.

In this case, the K subbands of the mth audio frame can be first screened based on the subband range, so as to use the noise estimation results corresponding to the subbands within the subband range to perform noise estimation on the mth audio frame, and then determine the noise estimation result of the mth audio frame.

In a specific implementation, a preset subband range may be first obtained, and a subband whose frequency range is within the preset subband range may be determined from the K subbands contained in the mth audio frame; wherein the preset subband range here is also the subband range mentioned above, and the number of subbands within the preset subband range may be N, and the subbands here may be referred to as reference subbands, that is, N reference subbands may be determined. After determining the N reference subbands, the noise estimation result of the m-1th audio frame may be further obtained to determine the noise estimation result of the current audio frame in combination with the noise estimation result of the historical audio frame; that is, based on the noise estimation result of the m-1th audio frame and the noise estimation results corresponding to each reference subband in the N reference subbands, the noise estimation of the mth audio frame may be performed to determine the noise estimation result of the mth audio frame.

In one implementation, based on the noise estimation result of the m-1th audio frame and the noise estimation results corresponding to each reference subband in the N reference subbands, noise estimation is performed on the mth audio frame, and a specific implementation of determining the noise estimation result of the mth audio frame may include the following steps S21-S23:

S21 , a noise estimation result of each reference subband in the N reference subbands may be determined.

Considering that the method for determining the noise estimation result of each reference subband is consistent, the following takes the nth reference subband among N reference subbands as an example to explain the determination of the noise estimation result of a reference subband. In a specific implementation, for the nth reference subband among the N reference subbands, the number of mid-frequency points of the nth reference subband can be first obtained, and the noise estimation result of the nth reference subband can be determined based on the number of mid-frequency points of the nth reference subband and the noise estimation result corresponding to the nth reference subband. In one embodiment, the number of mid-frequency points of the nth reference subband can be multiplied by the noise estimation result corresponding to the nth reference subband, and the product operation result can be used as the noise estimation result of the nth reference subband.

Based on this, it can be known that, through the above method, the noise estimation result of each reference sub-band in the N reference sub-bands can be determined.

S22: Based on the noise estimation results of each reference subband in the N reference subbands, an initial noise estimation result of the mth audio frame may be determined.

In one embodiment, the noise estimation results of each reference sub-band may be summed up, and the summed up result may be used as the initial noise estimation result of the mth audio frame.

S23 , based on the noise estimation result of the m-1 th audio frame and the initial noise estimation result of the m th audio frame, noise estimation may be performed on the m th audio frame to determine the noise estimation result of the m th audio frame.

In one embodiment, the noise estimation result of the m-1th audio frame and the initial noise estimation result of the mth audio frame may be summed, and the summation result is used as the noise estimation result of the mth audio frame.

In another embodiment, a noise estimation smoothing factor may be obtained to smooth the noise estimation result of the m-1th audio frame and the initial noise estimation result of the m-th audio frame based on the noise estimation smoothing factor to obtain the noise estimation result of the m-1th audio frame. In one embodiment, the noise estimation smoothing factor may be multiplied with the noise estimation result of the m-1th audio frame, and the difference between 1 and the noise estimation smoothing factor may be multiplied with the initial noise estimation result of the m-th audio frame; the two multiplication results may be summed, and the summed result may be used as the noise estimation result of the m-th audio frame. In this embodiment, the computer device may calculate the noise estimation result of the m-th audio frame by the following formula (8):

in, represents the noise estimation result of the mth audio frame, represents the noise estimation result of the m-1th audio frame, and β (β<1) represents the noise estimation smoothing factor. _K0 and _K1 are the subband numbers of the reference subbands within the preset subband range (such as the 100 Hz to 6 kHz spectrum range) mentioned above, _K0 represents the first reference subband among N reference subbands, and _K1 represents the Nth reference subband among N reference subbands.

S403, receiving the coded signal sent by the speaker, performing audio decoding on the coded signal to obtain a decoded signal, and performing audio playback on the decoded signal.

The specific implementation of this step can refer to the description in the above step S203, which will not be repeated here.

In an embodiment of the present application, the listener can estimate (evaluate) the background environmental noise of the listener and feed back the noise estimation result to the speaker, so that the speaker can adjust the default encoding parameters according to the feedback noise estimation result to adapt to the changes in the background noise environment of the listener, so as to achieve the best match between the playback sound quality and the listening environment effect.

Please refer to FIG. 5, which is a flowchart of another audio processing method provided in an embodiment of the present application. This embodiment mainly describes the specific implementation process on the sounding side. The audio processing method described in this embodiment includes the following steps:

S501: Receive a noise estimation result sent by a listener.

As mentioned above, after the noise estimation result obtains the first audio signal of the listener, the noise estimation is performed on the first audio signal, and the listener and the speaker have established a communication connection. Among them, the noise estimation result can be a specific estimation value, and the noise estimation result can also be called a noise estimation value. It can be understood that the noise estimation result can be used to characterize the intensity of the noise in the audio signal or the level of the noise, and the noise estimation result can also be called a noise intensity value or a noise level value.

S502: Determine encoding parameters for audio encoding based on the noise estimation result.

The encoding parameters may include one or more of encoding bit rate, number of channels, sampling rate, etc.

It is understandable that the settings of different encoding parameters in the audio encoding process will affect the audio quality of the codec output, and the audio quality may refer to the playback sound quality. For example, the encoding bit rate in the audio encoding process will affect the audio quality of the codec output, and will also affect the transmission bandwidth and storage consumption of the encoded data, so the encoding parameters can be dynamically adjusted according to the actual business and actual listening needs.

As mentioned above, the ultimate beneficiary of the audio call itself is the listener, and the listener's audio experience is strongly related to the listener's acoustic environment in addition to the sound quality of the playback signal (i.e., the audio signal played) itself. If the listener is in a relatively quiet environment, that is, the noise level of the listener is low, then when there is a slight sound quality problem with the playback signal, the listener may easily distinguish it, thereby affecting the listening effect; in this case, in order to make the listener unable to perceive the sound quality problem as much as possible, the encoding parameters on the encoding side can be set to parameters that make the audio output by the codec have better playback sound quality. For example, one or more of the encoding bit rate, sampling rate, number of channels, etc. can be set to a higher level.

On the contrary, if the listener is in a relatively noisy environment, that is, the noise level of the listener is high, considering that the background noise in the listener's environment can mask the defects of the playback signal, the subtle differences in the sound quality of the playback signal are difficult for the listener to distinguish; in this case, the difference in encoding parameters has little effect on the difference in listening sound quality, and the encoding parameters on the encoding side can be set to lower values, while minimizing the impact on the sound quality while reducing the amount of data, transmission bandwidth, storage space, etc., thereby reducing the cost of audio transmission and storage, and then achieving the best balance between the audio signal playback sound quality experience and operating costs (such as audio transmission bandwidth and storage costs).

Based on this, it can be seen that the basic principle for adjusting the encoding bit rate based on the noise level of the listener can be: the higher the noise level of the listener, the encoding parameters on the encoding side can be adjusted in the direction of worsening the playback sound quality; conversely, if the noise level of the listener is lower, the encoding parameters on the encoding side can be adjusted in the direction of improving the playback sound quality.

In one embodiment, taking the encoding parameter as the encoding bit rate, the basic principle for adjusting the encoding bit rate based on the noise level of the listener may be: the higher the noise level of the listener, the encoding bit rate on the encoding side may be adjusted downward; conversely, if the noise level of the listener is lower, the encoding bit rate on the encoding side may be adjusted upward. The adjustment here may generally refer to an upward adjustment or a downward adjustment relative to the default encoding bit rate. In general, the higher the noise level of the listener, the lower the encoding bit rate on the encoding side may be, and the lower the noise level of the listener, the higher the encoding bit rate on the encoding side may be.

In one embodiment, taking the encoding parameter as the sampling rate, the basic principle for adjusting the sampling rate based on the noise level of the listener may be: the higher the noise level of the listener, the sampling rate of the encoding side may be adjusted downward; conversely, if the noise level of the listener is lower, the sampling rate of the encoding side may be adjusted upward. The adjustment here may generally refer to an upward adjustment or a downward adjustment relative to the default sampling rate. In general, the higher the noise level of the listener, the lower the sampling rate of the encoding side may be, and the lower the noise level of the listener, the higher the sampling rate of the encoding side may be.

In one embodiment, taking the encoding parameter as the number of channels, the basic principle for adjusting the number of channels based on the noise level of the listener may be: the higher the noise level of the listener, the number of channels on the encoding side may be adjusted downward; conversely, if the noise level of the listener is lower, the number of channels on the encoding side may be adjusted upward. The adjustment here may generally refer to an upward adjustment or a downward adjustment relative to the default number of channels. In general, the higher the noise level of the listener, the lower the number of channels on the encoding side may be, and the lower the noise level of the listener, the higher the number of channels on the encoding side may be.

In one implementation, a mapping relationship between a reference noise estimation result and a reference coding parameter may be preset, and the mapping relationship may be a better match between the noise estimation result and the coding parameter obtained through a large number of actual measurement results. After receiving the noise estimation result sent by the listener, the speaker may obtain the mapping relationship to determine the coding parameter corresponding to the noise estimation result by using the mapping relationship.

Optionally, assuming that the encoding parameters include the encoding bit rate, a specific implementation of determining the encoding bit rate for audio encoding based on the noise estimation result may be described as follows.

In one embodiment, a mapping relationship between a reference noise estimation result and a reference encoding bit rate may be obtained; after obtaining the mapping relationship, the encoding bit rate for audio encoding may be determined based on the noise estimation result and the mapping relationship between the reference noise estimation result and the reference encoding bit rate.

For example, the mapping relationship can be represented by a mapping two-dimensional table. The data in the mapping two-dimensional table can be completed according to the actual measurement results, such as the mapping two-dimensional table can be shown in Table 1:

Table 1

Reference encoding bit rate Reference noise estimate A1 B1 A2 B2 A3 B3 … …

It should be noted that the corresponding values of B1 to B3 gradually increase, but A1 to A3 gradually decrease, which means that the greater the noise of the listener, the more likely it is that the audio encoding needs to be performed using encoding parameters that make the playback sound quality worse (such as the encoding bit rate here). As shown in Table 1, after the noise estimation result of the mth audio frame, the encoding bit rate corresponding to the noise estimation result can be determined based on Table 1. For example, assuming that the noise estimation value of the mth audio frame is B2, the corresponding encoding bit rate can be known to be A2 based on Table 1.

For another example, the mapping relationship can also be represented by a mapping formula. In one embodiment, the mapping formula can be obtained by fitting the data in the above-mentioned two-dimensional mapping table (such as Table 1). For example, the mapping formula can be shown as the following formula (9):

Wherein, bitrate(m) indicates the encoding bitrate of the mth audio frame, bitrate0 indicates the default encoding bitrate; a, b1, b2, c are constants; Represents the noise estimate of the mth audio frame.

It can be seen from formula (9) that after the noise estimation value of the mth audio frame is determined, it can be substituted into formula (9), and the calculation result of formula (9) can be used as the encoding bit rate.

In order to more vividly display the mapping relationship between the reference noise estimation result indicated by the mapping formula and the reference coding bit rate, a corresponding curve graph can be displayed based on the mapping formula. For example, the curve graph can be shown in Figure 6, wherein the horizontal axis in the coordinate system shown in Figure 6 is the noise level value (noise estimation value), and the vertical axis is the coding bit rate value.

Optionally, assuming that the encoding parameters include a sampling rate, a specific implementation of determining the sampling rate for audio encoding based on the noise estimation result may be described as follows.

A mapping relationship between a reference noise estimation result and a reference sampling rate may be obtained; after obtaining the mapping relationship, a sampling rate for audio encoding may be determined based on the noise estimation result and the mapping relationship between the reference noise estimation result and the reference sampling rate.

For example, the mapping relationship can be represented by a mapping two-dimensional table. The data in the mapping two-dimensional table can be completed according to the actual measurement results. For example, the mapping two-dimensional table can be shown in Table 2:

Table 2

Reference sampling rate Reference noise estimation results D1 C1～C2 D2 C2～C3 D3 C3～C4 … …

Among them, a reference noise estimation value range can correspond to a sampling rate gear. As shown in Table 2, the reference sampling rate corresponding to the reference noise estimation value between C1 and C2 is D1. It should be noted that the corresponding values of C1 to C4 gradually increase, but D1 to D3 gradually decrease, which means that the greater the noise of the listener, the more necessary it is to use encoding parameters (such as the sampling rate here) that make the playback sound quality worse for audio encoding. As can be seen from Table 2, after the noise estimation result of the mth audio frame, the sampling rate corresponding to the noise estimation result can be determined based on Table 2. For example, assuming that the noise estimation value of the mth audio frame is C0, and C0 is between C2 and C3, based on Table 2, it can be known that the corresponding sampling rate is D2.

Optionally, assuming that the encoding parameters include the number of channels, the specific implementation of determining the number of channels for audio encoding based on the noise estimation result may be described as follows.

A mapping relationship between a reference noise estimation result and a reference number of channels may be obtained; after obtaining the mapping relationship, the number of channels for audio encoding may be determined based on the noise estimation result and the mapping relationship between the reference noise estimation result and the reference number of channels.

For example, the mapping relationship can be represented by a mapping two-dimensional table. The data in the mapping two-dimensional table can be completed according to the actual measurement results. For example, the mapping two-dimensional table can be shown in Table 3:

Table 3

Number of reference channels Reference noise estimation results Second channel number E≤θ Number of first channels E>θ

Wherein, θ represents a preset noise estimation result (preset noise estimation value), which can be preset and the specific value is not limited; E represents a reference noise estimation result, that is, the noise estimation result can be compared with the preset noise estimation result to determine the final number of channels based on the comparison result. If the comparison result is that the noise estimation result is less than or equal to the preset noise estimation result, the encoding parameter can be determined as the second number of channels; if the comparison result is that the noise estimation result is greater than the preset noise estimation result, the encoding parameter can be determined as the first number of channels.

Based on this, the specific implementation of determining the number of channels can also be explained as:

First, the initial number of channels corresponding to the sounding party can be obtained, and the noise estimation result can be compared with the preset noise estimation result. If the comparison result is that the noise estimation result is less than or equal to the preset noise estimation result, it indicates that the noise of the listening party is not large, that is, it is necessary to use the encoding parameters that make the playback sound quality better for audio encoding; in this case, if the initial number of channels is the first number of channels (i.e., mono), the first number of channels can be adjusted to the second number of channels (i.e., dual channels), and if the initial number of channels is the second number of channels, there is no need to adjust the number of channels. If the comparison result is that the noise estimation result is greater than the preset noise estimation result, it indicates that the noise of the listening party is large, that is, it is not necessary to use the encoding parameters that make the playback sound quality better for audio encoding; in this case, if the initial number of channels is the second number of channels, the second number of channels can be adjusted to the first number of channels, and if the initial number of channels is the first number of channels, there is no need to adjust the number of channels.

Optionally, in the case where the coding parameters include multiple types of coding bit rate, sampling rate, and number of channels, in one embodiment, after obtaining the noise estimation result, multiple mapping relationships can also be obtained to determine the corresponding coding parameters using the noise estimation result and the multiple mapping relationships. For example, if the coding parameters include coding bit rate, sampling rate, and number of channels, the mapping relationship between the reference noise estimation result and the reference coding bit rate, the mapping relationship between the reference noise estimation result and the reference sampling rate, and the mapping relationship between the reference noise estimation result and the reference number of channels can be obtained. After obtaining these mapping relationships, the corresponding coding parameters can be determined based on these mapping relationships and the noise estimation result. For example, the coding bit rate can be determined based on the noise estimation result and the mapping relationship between the reference noise estimation result and the reference coding bit rate, the sampling rate can be determined based on the noise estimation result and the mapping relationship between the reference noise estimation result and the reference sampling rate, and the number of channels can be determined based on the noise estimation result and the mapping relationship between the reference noise estimation result and the reference number of channels.

In another embodiment, after obtaining the noise estimation result, a reference mapping relationship can also be obtained to determine the corresponding coding parameters using the noise estimation result and the reference mapping relationship. The reference mapping relationship includes a mapping relationship between the reference noise estimation result and the reference coding parameter. For example, if the coding parameters include coding bit rate, sampling rate, and number of channels, the reference coding parameters in the reference mapping relationship may include reference coding bit rate, reference sampling rate, and reference number of channels. In this case, only one mapping relationship is required to determine the corresponding coding parameters.

In one implementation, for different business scenarios, different mapping relationships between reference noise estimation results and reference coding parameters can be configured, that is, one business scenario corresponds to a mapping relationship between a reference noise estimation result and a reference coding parameter. The division of business scenarios is not specifically limited, such as business scenarios can be divided based on different communication methods, such as business scenarios can be divided into telephone call scenarios, audio and video call scenarios, audio and video conference scenarios, karaoke scenarios, live broadcast scenarios, game scenarios, etc.

Based on this, it can be known that in a specific implementation, the business scenario to which the established communication connection belongs can be first obtained. After determining the business scenario, a mapping relationship between a reference noise estimation result and a reference coding parameter adapted to the business scenario can be obtained; and based on the noise estimation result and the mapping relationship between the reference noise estimation result and the reference coding parameter, the coding parameters for audio coding can be determined.

The encoding parameters here may include one or more of encoding bit rate, sampling rate, and number of channels. For different encoding parameters, it is only necessary to obtain the corresponding mapping relationship.

In one implementation, for different audio playback types, different mapping relationships between reference noise estimation results and reference coding parameters can be configured, that is, one audio playback type corresponds to a mapping relationship between a reference noise estimation result and a reference coding parameter. Among them, the audio playback type can be divided into an audio playback type in which the audio signal corresponding to the sounding party is a voice (which can be simply referred to as a voice playback type), and an audio playback type in which the audio signal corresponding to the sounding party is music (which can be simply referred to as a music playback type). It can be understood that for different audio playback types, there are differences in the default encoding parameters set for audio encoding. Usually, the encoding effect corresponding to the encoding parameters required for the music playback type is better than the encoding effect corresponding to the encoding parameters required for the voice playback type. For different audio playback types, when setting the mapping relationship, there may be differences between the reference noise estimation result and the reference coding parameter in the mapping relationship. In order to achieve the best match between the playback sound quality and the listening environment effect, different mapping relationships can be set for different audio playback types.

Based on this, it can be seen that in a specific implementation, after obtaining the second audio signal of the speaker, the audio playback type of the second audio signal can also be determined to obtain a mapping relationship between a reference noise estimation result and a reference coding parameter suitable for the audio playback type; and based on the noise estimation result and the mapping relationship between the reference noise estimation result and the reference coding parameters, the coding parameters for audio encoding are determined.

In one implementation, for different audio timbre types, different mapping relationships between reference noise estimation results and reference coding parameters can be configured, that is, one audio timbre type corresponds to a mapping relationship between a reference noise estimation result and a reference coding parameter. Among them, the audio timbre type can be divided into an audio timbre type (which can be referred to as a male timbre type) in which the audio signal corresponding to the speaker is a male, and an audio timbre type (which can be referred to as a female timbre type) in which the audio signal corresponding to the speaker is a female. Based on this, it can be seen that in a specific implementation, after obtaining the second audio signal of the speaker, the audio timbre type of the second audio signal can also be determined to obtain the mapping relationship between the reference noise estimation result and the reference coding parameter suitable for the audio timbre type; and based on the noise estimation result and the mapping relationship between the reference noise estimation result and the reference coding parameter, the coding parameters used for audio coding are determined.

In one implementation, as described above, the coding parameters adjusted using the noise estimation results may include one or more of the coding bit rate, sampling rate, and number of channels. In actual scenarios, the coding parameters that ultimately need to be adjusted may also be determined based on actual business needs, such as adjusting only the coding bit rate, or only the sampling rate, and so on.

Optionally, the business demand can be obtained, and the coding parameters to be adjusted can be determined based on the business demand. For example, if the business demand is a first business demand, and the first business demand is to ensure the balance between the transmission bandwidth and the playback sound quality, then the coding parameter can be determined to be a coding parameter under the coding bit rate dimension, that is, the coding parameter is the coding bit rate. For another example, if the business demand is a second business demand, and the second business demand is to ensure the balance between the signal sampling data volume and the playback sound quality, then the coding parameter can be determined to be a coding parameter under the sampling rate dimension, that is, the coding parameter is the sampling rate. For another example, if the business demand is a third business demand, and the third business demand is to ensure the balance between the signal acquisition complexity and the playback sound quality, then the coding parameter can be determined to be a coding parameter under the channel number dimension, that is, the coding parameter is the number of channels. Among them, the business demand can be specified by the relevant R&D personnel when the default coding parameter setting is set, or it can be set in other ways, without specific limitation.

It should be noted that the mapping relationship between the various reference noise estimation results and the reference coding parameters mentioned above can be obtained based on the pre-measured results. For example, the following specifically describes the determination of the mapping relationship between a reference noise estimation result and a reference coding rate. First, for a reference noise estimation result (i.e., a reference noise estimation value), L candidate coding rates can be configured for it. In one embodiment, when configuring the L candidate coding rates, they can be set in combination with the size of the reference noise estimation value. For example, the larger the reference noise estimation value, the smaller the set candidate coding rate can be, and the smaller the reference noise estimation value, the larger the set candidate coding rate can be.

Then, each of the L candidate encoding rates can be used to perform audio encoding and audio decoding on the same audio signal to obtain a decoded signal corresponding to the audio signal. Based on this, it can be known that L decoded signals can be obtained, and one candidate encoding rate corresponds to one decoded signal. Finally, the L decoded signals can be played audio, and the listening effect can be subjectively evaluated, and the reference encoding rate corresponding to the reference noise estimation result can be determined from the L candidate encoding rates based on the listening effect. For example, the candidate encoding rate used by the decoded signal with the best listening effect can be determined as the reference encoding rate; or, if there are multiple decoded signals with good listening effects and the effect difference is small, the minimum candidate encoding rate used by the multiple decoded signals can be determined as the reference encoding rate, so as to ensure the listening effect while using a smaller encoding rate as much as possible for subsequent audio encoding, thereby reducing bandwidth costs and storage costs.

S503: If the second audio signal of the speaker is obtained, the second audio signal is audio-encoded using the encoding parameter to obtain an encoded signal corresponding to the second audio signal, and the encoded signal is sent to the listener.

The specific implementation of this step may refer to the description in the above step S206, which will not be repeated here.

As mentioned above, in the process of establishing a communication connection between the listener and the speaker, the listener can periodically perform noise estimation and send the corresponding noise estimation results to the speaker. The speaker can then dynamically adjust the encoding parameters required for audio encoding based on the periodic noise estimation results to adapt to changes in the background noise environment of the listener, thereby achieving the best match between the playback sound quality and the listening environment effect.

It should be noted that the default audio encoding parameters (such as encoding bit rate) are configured at the beginning of the communication, that is, when the user speaks through the speaker, the acquired audio signal will be encoded based on the default encoding parameters. When the audio signal is acquired again, it can be determined whether there is an updated value for the current encoding parameter. If there is an updated value, the updated value can be used to encode the currently received audio signal. Similarly, for the subsequently acquired audio signals, the latest encoding parameters can be continuously used for audio encoding.

In an embodiment of the present application, the speaker can adjust the default encoding parameters according to the noise estimation result fed back by the listener to adapt to the changes in the background noise environment of the listener, so as to achieve the best match between the playback sound quality and the listening environment effect.

Please refer to FIG. 7 , which is a schematic diagram of the structure of an audio processing device provided in an embodiment of the present application. The audio processing device described in this embodiment includes:

The acquisition unit 701 is used to acquire a first audio signal of a listener; the listener has established a communication connection with a speaker;

an estimating unit 702, configured to perform noise estimation on the first audio signal, determine a noise estimation result of the first audio signal, and send the noise estimation result of the first audio signal to the sounding party; the noise estimation result is used to indicate the intensity of noise in the first audio signal;

The decoding unit 703 is used to receive the encoded signal sent by the speaker, perform audio decoding on the encoded signal to obtain a decoded signal, and perform audio playback on the decoded signal; the encoded signal is obtained by the speaker obtaining the second audio signal of the speaker and then performing audio encoding on the second audio signal using the encoding parameters determined by the noise estimation result.

In one implementation, the estimating unit 702 is specifically configured to:

Performing frame processing on the first audio signal to obtain M audio frames;

Traversing each audio frame in the M audio frames, if traversing to the m-th audio frame in the M audio frames, performing Fourier transform processing on the m-th audio frame to obtain a power spectrum of the m-th audio frame;

Noise estimation is performed on the mth audio frame based on the power spectrum of the mth audio frame, a noise estimation result of the mth audio frame is determined, and the noise estimation result of the mth audio frame is used as the noise estimation result of the first audio signal.

In one implementation, the estimating unit 702 is specifically configured to:

Dividing the power spectrum of the m-th audio frame into K sub-bands, and determining a sub-band power spectrum of each sub-band in the K sub-bands;

Performing noise estimation on the corresponding subband based on the subband power spectrum of each subband, and determining a noise estimation result of the corresponding subband;

Based on the noise estimation results corresponding to the sub-bands included in the m-th audio frame, noise estimation is performed on the m-th audio frame to determine the noise estimation result of the m-th audio frame.

In one implementation, the estimating unit 702 is specifically configured to:

For a k-th subband among the K subbands, performing time-frequency domain smoothing processing on a subband power spectrum of the k-th subband to obtain a smoothed subband power spectrum of the k-th subband;

Obtaining a preset period, and determining a minimum subband power spectrum of the kth subband for the mth audio frame based on the smoothed subband power spectrum of the kth subband and the preset period;

Determining, based on the smoothed subband power spectrum and the minimum subband power spectrum, a speech probability that speech exists in the kth subband of the mth audio frame;

A noise estimation result of the kth subband of the mth audio frame is determined based on the speech probability, the subband power spectrum of the kth subband of the mth audio frame, and the noise estimation result of the kth subband of the (m-1)th audio frame.

In one implementation, the estimating unit 702 is specifically configured to:

Obtaining a subband power spectrum corresponding to each of r adjacent subbands adjacent to the kth subband;

Based on the subband power spectra corresponding to the adjacent subbands, the subband power spectrum of the k-th subband is smoothed in the frequency domain to obtain a smoothed subband power spectrum of the k-th subband in the frequency domain;

Obtaining a smoothed subband power spectrum of the kth subband of the m-1th audio frame;

Based on the smoothed subband power spectrum of the kth subband of the m-1th audio frame and the smoothed subband power spectrum of the kth subband in the frequency domain, time domain smoothing is performed to obtain the smoothed subband power spectrum of the kth subband in the time domain, and the smoothed subband power spectrum of the kth subband in the time domain is used as the smoothed subband power spectrum of the kth subband.

In one implementation, the estimating unit 702 is specifically configured to:

Performing modulo processing on the k and the T, and determining a processing result;

If the processing result is a specified value, obtaining a minimum subband power spectrum of a previous period, and taking a minimum value of the minimum subband power spectrum of the previous period and a smoothed subband power spectrum of the kth subband of the mth audio frame as the minimum subband power spectrum of the kth subband of the mth audio frame;

If the processing result is not a specified value, then the minimum subband power spectrum of the kth subband for the m-1th audio frame is obtained, and the minimum value of the smoothed subband power spectrum of the kth subband and the minimum subband power spectrum of the kth subband for the m-1th audio frame is used as the minimum subband power spectrum of the kth subband for the m-1th audio frame.

In one implementation, the estimating unit 702 is specifically configured to:

determining a signal-to-noise ratio based on the smoothed sub-band power spectrum and the minimum sub-band power spectrum;

Comparing the signal-to-noise ratio with a threshold, and determining an initial speech probability that speech exists in the kth subband of the mth audio frame based on the comparison result;

Obtain a speech probability that speech exists in the kth subband of the m-1th audio frame, and determine a speech probability that speech exists in the kth subband of the m-1th audio frame based on the speech probability corresponding to the kth subband of the m-1th audio frame and the initial speech probability.

In one implementation, the estimating unit 702 is specifically configured to:

If the comparison result is that the signal-to-noise ratio is greater than the threshold, determining the initial speech probability that speech exists in the kth subband of the mth audio frame as a first probability;

If the comparison result is that the signal-to-noise ratio is less than or equal to the threshold, the initial speech probability that speech exists in the kth subband of the mth audio frame is determined as the second probability.

In one implementation, the estimating unit 702 is specifically configured to:

Obtaining a preset subband range, and determining N reference subbands whose frequency range is within the preset subband range from the K subbands included in the m-th audio frame;

Obtain a noise estimation result of an m-1th audio frame, perform noise estimation on the m-1th audio frame based on the noise estimation result of the m-1th audio frame and the noise estimation results corresponding to each reference subband in the N reference subbands, and determine a noise estimation result of the m-1th audio frame.

In one implementation, the estimating unit 702 is specifically configured to:

For an nth reference subband among the N reference subbands, obtaining the number of frequency points in the nth reference subband, and determining a noise estimation result of the nth reference subband based on the number of frequency points in the nth reference subband and a noise estimation result corresponding to the nth reference subband;

Determining an initial noise estimation result of the m-th audio frame based on the noise estimation result of each reference subband in the N reference subbands;

Based on the noise estimation result of the m-1th audio frame and the initial noise estimation result of the mth audio frame, noise estimation is performed on the mth audio frame to determine the noise estimation result of the mth audio frame.

In one implementation, the estimating unit 702 is specifically configured to:

For a k-th subband among the K subbands, obtaining the number of frequency points in the k-th subband;

Based on the number of frequency points in the kth sub-band and the frequency domain value corresponding to each frequency point in the kth sub-band, a sub-band power spectrum of each sub-band in the K sub-bands is determined.

Please refer to FIG8, which is a schematic diagram of the structure of another audio processing device provided in an embodiment of the present application. The audio processing device described in this embodiment includes:

The receiving unit 801 is configured to receive a noise estimation result sent by a listener; the noise estimation result is obtained by estimating the noise of the first audio signal after the listener obtains the first audio signal of the listener, and the listener has established a communication connection with the sounding party;

A determining unit 802, configured to determine encoding parameters for audio encoding based on the noise estimation result;

The encoding unit 803 is configured to, if a second audio signal of the speaker is acquired, perform audio encoding on the second audio signal using the encoding parameters to obtain an encoded signal corresponding to the second audio signal, and send the encoded signal to the listener.

In one implementation, the encoding parameter includes one or more of an encoding bit rate and a sampling rate; the determining unit 802 is specifically configured to:

Obtaining a mapping relationship between a reference noise estimation result and a reference coding parameter;

Based on the noise estimation result and a mapping relationship between the reference noise estimation result and a reference encoding parameter, encoding parameters for audio encoding are determined.

In one implementation, the encoding parameter includes the number of channels; the determining unit 802 is specifically configured to:

Comparing the noise estimation result with a preset noise estimation result;

If the comparison result is that the noise estimation result is greater than the preset noise estimation result, determining the encoding parameter as the first channel number;

If the comparison result is that the noise estimation result is less than or equal to the preset noise estimation result, the encoding parameter is determined as the second channel number.

Please refer to Figure 9, which is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device can be the above-mentioned speaker and/or listener, or can execute some or all of the steps executed by the above-mentioned speaker and/or listener. The computer device described in this embodiment includes: a processor 901, a memory 902, and a network interface 903. The above-mentioned processor 901, memory 902, and network interface 903 can exchange data.

The processor 901 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

The memory 902 may include a read-only memory and a random access memory, and provide program instructions and data to the processor 901. A portion of the memory 902 may also include a non-volatile random access memory.

Optionally, in some embodiments, the computer device may be a listener, or may execute some or all of the steps executed by the listener. For example, when the processor 901 calls the program instruction, it is used to execute:

Acquiring a first audio signal from a listening party; the listening party has established a communication connection with a sounding party;

Performing noise estimation on the first audio signal, determining a noise estimation result of the first audio signal, and sending the noise estimation result of the first audio signal to the speaker; the noise estimation result is used to indicate the intensity of noise in the first audio signal;

Receive the encoded signal sent by the speaker, perform audio decoding on the encoded signal to obtain a decoded signal, and perform audio playback on the decoded signal; the encoded signal is obtained by audio encoding the second audio signal after the speaker obtains the second audio signal of the speaker using the encoding parameters determined by the noise estimation result.

In one implementation, the processor 901 is specifically configured to:

Performing frame processing on the first audio signal to obtain M audio frames;

Traversing each audio frame in the M audio frames, if traversing to the m-th audio frame in the M audio frames, performing Fourier transform processing on the m-th audio frame to obtain a power spectrum of the m-th audio frame;

Noise estimation is performed on the mth audio frame based on the power spectrum of the mth audio frame, a noise estimation result of the mth audio frame is determined, and the noise estimation result of the mth audio frame is used as the noise estimation result of the first audio signal.

In one implementation, the processor 901 is specifically configured to:

Dividing the power spectrum of the m-th audio frame into K sub-bands, and determining a sub-band power spectrum of each sub-band in the K sub-bands;

Performing noise estimation on the corresponding subband based on the subband power spectrum of each subband, and determining a noise estimation result of the corresponding subband;

Based on the noise estimation results corresponding to the sub-bands included in the m-th audio frame, noise estimation is performed on the m-th audio frame to determine the noise estimation result of the m-th audio frame.

In one implementation, the processor 901 is specifically configured to:

For a k-th subband among the K subbands, performing time-frequency domain smoothing processing on a subband power spectrum of the k-th subband to obtain a smoothed subband power spectrum of the k-th subband;

Obtaining a preset period, and determining a minimum subband power spectrum of the kth subband for the mth audio frame based on the smoothed subband power spectrum of the kth subband and the preset period;

Determining, based on the smoothed subband power spectrum and the minimum subband power spectrum, a speech probability that speech exists in the kth subband of the mth audio frame;

A noise estimation result of the kth subband of the mth audio frame is determined based on the speech probability, the subband power spectrum of the kth subband of the mth audio frame, and the noise estimation result of the kth subband of the (m-1)th audio frame.

In one implementation, the processor 901 is specifically configured to:

Obtaining a subband power spectrum corresponding to each of r adjacent subbands adjacent to the kth subband;

Based on the subband power spectra corresponding to the adjacent subbands, the subband power spectrum of the k-th subband is smoothed in the frequency domain to obtain a smoothed subband power spectrum of the k-th subband in the frequency domain;

Obtaining a smoothed subband power spectrum of the kth subband of the m-1th audio frame;

Based on the smoothed subband power spectrum of the kth subband of the m-1th audio frame and the smoothed subband power spectrum of the kth subband in the frequency domain, time domain smoothing is performed to obtain the smoothed subband power spectrum of the kth subband in the time domain, and the smoothed subband power spectrum of the kth subband in the time domain is used as the smoothed subband power spectrum of the kth subband.

In one implementation, the processor 901 is specifically configured to:

Performing modulo processing on the k and the T, and determining a processing result;

If the processing result is a specified value, obtaining a minimum subband power spectrum of a previous period, and taking a minimum value of the minimum subband power spectrum of the previous period and a smoothed subband power spectrum of the kth subband of the mth audio frame as the minimum subband power spectrum of the kth subband of the mth audio frame;

If the processing result is not a specified value, then the minimum subband power spectrum of the kth subband for the m-1th audio frame is obtained, and the minimum value of the smoothed subband power spectrum of the kth subband and the minimum subband power spectrum of the kth subband for the m-1th audio frame is used as the minimum subband power spectrum of the kth subband for the m-1th audio frame.

In one implementation, the processor 901 is specifically configured to:

determining a signal-to-noise ratio based on the smoothed sub-band power spectrum and the minimum sub-band power spectrum;

Comparing the signal-to-noise ratio with a threshold, and determining an initial speech probability that speech exists in the kth subband of the mth audio frame based on the comparison result;

Obtain a speech probability that speech exists in the kth subband of the m-1th audio frame, and determine a speech probability that speech exists in the kth subband of the m-1th audio frame based on the speech probability corresponding to the kth subband of the m-1th audio frame and the initial speech probability.

In one implementation, the processor 901 is specifically configured to:

If the comparison result is that the signal-to-noise ratio is greater than the threshold, determining the initial speech probability that speech exists in the kth subband of the mth audio frame as a first probability;

If the comparison result is that the signal-to-noise ratio is less than or equal to the threshold, the initial speech probability that speech exists in the kth subband of the mth audio frame is determined as the second probability.

In one implementation, the processor 901 is specifically configured to:

Obtaining a preset subband range, and determining N reference subbands whose frequency range is within the preset subband range from the K subbands included in the m-th audio frame;

Obtain a noise estimation result of an m-1th audio frame, perform noise estimation on the m-1th audio frame based on the noise estimation result of the m-1th audio frame and the noise estimation results corresponding to each reference subband in the N reference subbands, and determine a noise estimation result of the m-1th audio frame.

In one implementation, the processor 901 is specifically configured to:

For an nth reference subband among the N reference subbands, obtaining the number of frequency points in the nth reference subband, and determining a noise estimation result of the nth reference subband based on the number of frequency points in the nth reference subband and a noise estimation result corresponding to the nth reference subband;

Determining an initial noise estimation result of the m-th audio frame based on the noise estimation result of each reference subband in the N reference subbands;

Based on the noise estimation result of the m-1th audio frame and the initial noise estimation result of the mth audio frame, noise estimation is performed on the mth audio frame to determine the noise estimation result of the mth audio frame.

In one implementation, the processor 901 is specifically configured to:

For a k-th subband among the K subbands, obtaining the number of frequency points in the k-th subband;

Based on the number of frequency points in the kth sub-band and the frequency domain value corresponding to each frequency point in the kth sub-band, a sub-band power spectrum of each sub-band in the K sub-bands is determined.

Optionally, in some embodiments, the computer device may be the speaker, or may execute some or all of the steps executed by the speaker. For example, when the processor 901 calls the program instruction, it is used to execute:

receiving a noise estimation result sent by a listener; the noise estimation result is obtained by estimating the noise of the first audio signal after the listener acquires the first audio signal of the listener, and the listener has established a communication connection with the sounding party;

Determining encoding parameters for audio encoding based on the noise estimation result;

If the second audio signal of the speaker is obtained, the second audio signal is audio-encoded using the encoding parameters to obtain an encoded signal corresponding to the second audio signal, and the encoded signal is sent to the listener.

In one implementation, the encoding parameter includes one or more of an encoding bit rate and a sampling rate; the processor 901 is specifically configured to:

Obtaining a mapping relationship between a reference noise estimation result and a reference coding parameter;

Based on the noise estimation result and a mapping relationship between the reference noise estimation result and a reference encoding parameter, encoding parameters for audio encoding are determined.

In one implementation, the encoding parameter includes the number of channels; the processor 901 is specifically configured to:

Comparing the noise estimation result with a preset noise estimation result;

If the comparison result is that the noise estimation result is greater than the preset noise estimation result, determining the encoding parameter as the first channel number;

If the comparison result is that the noise estimation result is less than or equal to the preset noise estimation result, the encoding parameter is determined as the second channel number.

An embodiment of the present application further provides a computer storage medium in which program instructions are stored. When the program is executed, it may include some or all steps of the audio processing method in the corresponding embodiment of FIG. 2 or FIG. 4 or FIG. 5 .

It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

The embodiment of the present application also provides a computer program product or a computer program, which includes program instructions, and when the program instructions are executed by a processor, some or all of the steps in the above method can be implemented. For example, the program instructions are stored in a computer-readable storage medium. The processor of the computer device reads the program instructions from the computer-readable storage medium, and the processor executes the program instructions, so that the computer device performs the steps performed in the embodiments of the above methods.

The above is a detailed introduction to an audio processing method and medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims (15)

1. A method of audio processing, the method comprising:

acquiring a first audio signal of a listener; the listener and the sounding party have established communication connection;

Performing noise estimation on the first audio signal, determining a noise estimation result of the first audio signal, and sending the noise estimation result of the first audio signal to the sounding party; the noise estimation result is used for indicating the intensity of noise in the first audio signal;

Receiving the coded signal sent by the sounding party, performing audio decoding on the coded signal to obtain a decoded signal, and performing audio playing on the decoded signal; and the coding signal is obtained by performing audio coding on the second audio signal by using the coding parameters determined by the noise estimation result after the sounding party acquires the second audio signal of the sounding party.

2. The method of claim 1, wherein the performing noise estimation on the first audio signal, determining a noise estimation result of the first audio signal, comprises:

Carrying out framing treatment on the first audio signal to obtain M audio frames;

traversing each audio frame in the M audio frames, and if traversing to an mth audio frame in the M audio frames, performing Fourier transform processing on the mth audio frame to obtain a power spectrum of the mth audio frame;

And carrying out noise estimation on the mth audio frame based on the power spectrum of the mth audio frame, determining a noise estimation result of the mth audio frame, and taking the noise estimation result of the mth audio frame as a noise estimation result of a first audio signal.

3. The method of claim 2, wherein the noise estimating the mth audio frame based on the power spectrum of the mth audio frame, determining the noise estimation result of the mth audio frame, comprises:

dividing the power spectrum of the mth audio frame into K sub-bands, and determining the sub-band power spectrum of each sub-band in the K sub-bands;

carrying out noise estimation on the corresponding sub-bands based on the sub-band power spectrums of the sub-bands, and determining a noise estimation result of the corresponding sub-bands;

And carrying out noise estimation on the mth audio frame based on the noise estimation results corresponding to the sub-bands contained in the mth audio frame, and determining the noise estimation result of the mth audio frame.

4. A method according to claim 3, wherein said noise estimating the respective sub-band based on the sub-band power spectrum of the respective sub-band, determining the noise estimation result of the respective sub-band, comprises:

Performing time-frequency domain smoothing processing on the sub-band power spectrum of the kth sub-band aiming at the kth sub-band in the K sub-bands to obtain a smoothed sub-band power spectrum of the kth sub-band;

acquiring a preset period, and determining a minimum subband power spectrum of a kth subband for an mth audio frame based on the smooth subband power spectrum of the kth subband and the preset period;

determining a speech probability of speech being present in a kth sub-band of the mth audio frame based on the smoothed sub-band power spectrum and the minimum sub-band power spectrum;

And determining the noise estimation result of the kth sub-band of the m-th audio frame based on the voice probability, the sub-band power spectrum of the kth sub-band of the m-th audio frame and the noise estimation result of the kth sub-band of the m-1 th audio frame.

5. The method of claim 4, wherein performing time-frequency domain smoothing on the subband power spectrum of the kth subband to obtain a smoothed subband power spectrum of the kth subband, comprises:

acquiring sub-band power spectrums respectively corresponding to each adjacent sub-band in r adjacent sub-bands adjacent to the kth sub-band;

Carrying out frequency domain smoothing on the sub-band power spectrum of the kth sub-band based on the sub-band power spectrums respectively corresponding to the adjacent sub-bands to obtain a smoothed sub-band power spectrum of the kth sub-band on a frequency domain;

Acquiring a smooth sub-band power spectrum of a kth sub-band of the m-1 th audio frame;

and carrying out time domain smoothing processing based on the smooth sub-band power spectrum of the kth sub-band of the m-1 audio frame and the smooth sub-band power spectrum of the kth sub-band on the frequency domain to obtain the smooth sub-band power spectrum of the kth sub-band on the time domain, and taking the smooth sub-band power spectrum of the kth sub-band on the time domain as the smooth sub-band power spectrum of the kth sub-band.

6. The method of claim 4, wherein the predetermined period is T; the determining a minimum subband power spectrum for the kth subband of the mth audio frame based on the smooth subband power spectrum of the kth subband and the preset period comprises:

Performing residual taking processing on the k and the T, and determining a processing result;

if the processing result is a specified value, acquiring a minimum sub-band power spectrum of the previous period, and taking the minimum sub-band power spectrum of the previous period and a minimum value in a smooth sub-band power spectrum of a kth sub-band of the mth audio frame as a minimum sub-band power spectrum of the kth sub-band of the mth audio frame;

And if the processing result is not the appointed value, acquiring the minimum sub-band power spectrum of the kth sub-band of the m-1 th audio frame, and taking the minimum value in the smooth sub-band power spectrum of the kth sub-band and the minimum sub-band power spectrum of the kth sub-band of the m-1 th audio frame as the minimum sub-band power spectrum of the kth sub-band of the m-1 th audio frame.

7. The method of claim 4, wherein the determining the speech probability of speech being present in the kth sub-band of the mth audio frame based on the smoothed sub-band power spectrum and the minimum sub-band power spectrum comprises:

determining a signal-to-noise ratio based on the smoothed sub-band power spectrum and the minimum sub-band power spectrum;

Comparing the signal-to-noise ratio with a threshold value, and determining an initial voice probability of voice existing in a kth sub-band of the mth audio frame based on a comparison result;

The voice probability of the voice in the kth sub-band of the (m-1) th audio frame is obtained, and the voice probability of the voice in the kth sub-band of the (m-1) th audio frame is determined based on the voice probability corresponding to the kth sub-band of the (m-1) th audio frame and the initial voice probability.

8. The method of claim 7, wherein determining an initial speech probability for speech to be present in a kth sub-band of the mth audio frame based on the comparison result comprises:

If the comparison result is that the signal-to-noise ratio is greater than the threshold value, determining the initial voice probability of the voice existing in the kth sub-band of the mth audio frame as a first probability;

And if the comparison result is that the signal-to-noise ratio is smaller than or equal to the threshold value, determining the initial voice probability of the voice existing in the kth sub-band of the mth audio frame as a second probability.

9. The method of claim 3, wherein the determining the noise estimation result of the mth audio frame based on the noise estimation result corresponding to each sub-band included in the mth audio frame comprises:

acquiring a preset sub-band range, and determining N reference sub-bands with frequency ranges within the preset sub-band range from K sub-bands contained in the mth audio frame;

And acquiring a noise estimation result of an m-1 th audio frame, carrying out noise estimation on the m audio frame based on the noise estimation result of the m-1 th audio frame and the noise estimation results corresponding to all the reference sub-bands in the N reference sub-bands, and determining the noise estimation result of the m audio frame.

10. The method of claim 9, wherein the determining the noise estimate for the mth audio frame based on the noise estimate for the mth-1 audio frame and the noise estimates for each of the N reference subbands comprises:

for an nth reference sub-band in the N reference sub-bands, acquiring the number of frequency points in the nth reference sub-band, and determining a noise estimation result of the nth reference sub-band based on the number of frequency points in the nth reference sub-band and a noise estimation result corresponding to the nth reference sub-band;

Determining an initial noise estimation result of the mth audio frame based on noise estimation results of each of the N reference subbands;

And carrying out noise estimation on the m-th audio frame based on the noise estimation result of the m-1-th audio frame and the initial noise estimation result of the m-th audio frame, and determining the noise estimation result of the m-th audio frame.

11. The method of claim 2, wherein said determining a subband power spectrum for each of said K subbands comprises:

For a kth sub-band in the K sub-bands, acquiring the number of frequency points in the kth sub-band;

and determining the sub-band power spectrum of each sub-band in the K sub-bands based on the number of the frequency points in the K sub-bands and the frequency domain value corresponding to each frequency point in the K sub-bands.

12. A method of audio processing, the method comprising:

Receiving a noise estimation result sent by a listener; the noise estimation result is obtained by carrying out noise estimation on a first audio signal of the listener after the listener acquires the first audio signal, and the listener and the sounding party are in communication connection;

determining coding parameters for audio coding based on the noise estimation result;

and if the second audio signal of the sounding party is obtained, performing audio coding on the second audio signal by utilizing the coding parameters to obtain a coding signal corresponding to the second audio signal, and transmitting the coding signal to the listening party.

13. The method of claim 12, wherein the encoding parameters include one or more of an encoding rate and a sampling rate; the determining coding parameters for audio coding based on the noise estimation result includes:

Obtaining a mapping relation between a reference noise estimation result and a reference coding parameter;

and determining coding parameters for audio coding based on the noise estimation result and the mapping relation between the reference noise estimation result and the reference coding parameters.

14. The method of claim 12, wherein the encoding parameters include a number of channels; the determining coding parameters for audio coding based on the noise estimation result includes:

comparing the noise estimation result with a preset noise estimation result;

If the comparison result is that the noise estimation result is larger than the preset noise estimation result, determining the coding parameter as a first channel number;

And if the comparison result is that the noise estimation result is smaller than or equal to the preset noise estimation result, determining the coding parameter as a second channel number.

15. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein program instructions, which when executed, are adapted to carry out the method according to any of claims 1-14.