JP7318064B2 - Adaptive management of casting requests and/or user input in rechargeable devices - Google Patents

Description

The limited charge capacity of batteries incorporated in portable electronic devices is a challenge for such portable electronic devices, especially given that each subsystem of such devices relies on at least some amount of energy to function. May affect usability. Additionally, when a battery-powered device provides access to an automated assistant, the device is tasked with constantly processing audio and/or other data for detection of ringing signals that invoke the automated assistant. In that case, energy resources may be even more limited. If the battery-powered assistant device contains a device system-on-chip (SoC), the device SoC will be responsible for other subsystems (e.g., network processor, digital signal processor (DSP), etc.) when the user is interacting with the automated assistant. , etc.). For example, a significant amount of battery charge may be spent by the processor performing audio processing, which may include removing various audio data artifacts such as echo, static, and/or other noise. .

Serving or streaming content from one device for rendering on another device is sometimes referred to as "casting." A battery-powered portable electronic device capable of responding to casting requests can expend a significant amount of battery charge when operating to constantly process casting requests from local network devices. For example, a battery-powered device that accepts casting-related pings and/or requests to "cast" media to the battery-powered device may use the device SoC to process the data contained by the incoming request. . However, when such requests become more frequent and/or redundant, using the device SoC to process the requests can severely limit the charging capacity of the battery-powered device. As a result, battery-powered devices are still able to render casted data, but the available casting time is limited as a result of how often the device SoC is asked to handle casting-related requests. is reduced.

The implementations described herein provide access to an automated assistant and/or one or more interfaces for rendering casted data provided by a separate computing device while processing a cast request. and/or to rechargeable devices that adaptively manage user input. Rechargeable devices are generally limited by having a finite power source, such as a battery, that can be depleted by operating the rechargeable device to frequently process cast requests and user input (e.g., speech such as call phrases). can be In order to extend the time between charges and otherwise waste computational resources, rechargeable devices employ a variety of different subsystem operating schemes adapted to manage such requests and inputs. can be used.

For example, in some implementations, a rechargeable device includes a first processor, such as a digital signal processor (DSP), and a device system-on-chip (SoC) for handling various inputs depending on the operating mode of the rechargeable device. ) and a second processor such as The mode of operation may be that the device SoC is powered down, or otherwise the device SoC is in another mode of operation (e.g., a mode of operation in which an automated assistant can actively interact with the user via the rechargeable device). It can be in one of several modes of operation, such as a sleep mode, that consumes less power than if it were operating in full. While the rechargeable device is operating in sleep mode, the DSP may be powered on to monitor user input to the rechargeable device with permission from the user. As an example, a rechargeable device may include one or more microphones, and DSP is provided by one or more of the microphones when the rechargeable device is operating in sleep mode. Any output (eg, output characterizing speech from the user to the microphone) can be monitored. The DSP runs a speech recognition model (e.g., a calling phrase model) to determine whether the user has given an utterance corresponding to a calling phrase (e.g., "Assistant...") to invoke an automated assistant. can let When the DSP, using the speech recognition model, determines that the user has given a call phrase to summon the automated assistant, the DSP can cause the device SoC to initialize for further processing. For example, the device SoC may initialize with respect to a certain period of "wake time" to wait for further instructions and/or input from the user.

The amount of time a device SoC remains active may change over time depending on various characteristics associated with interactions between one or more users and the automated assistant. The amount of time can be adapted and/or determined to reduce waste of computational resources and power that would otherwise run the speech recognition model in the device SoC. It may be spent on letting For example, the device SoC may support a speech recognition model operated by a DSP as opposed to a speech recognition model operated by a DSP (e.g., a second call phrase model and/or other voice activity detector). A separate speech recognition model (eg, the first call phrase model and/or the voice activity detector) may be operated that may require more computational resources and/or power than the others. Therefore, by adapting the amount of "wake time" of the device SoC, unnecessary battery power consumption is maintained while still ensuring that the rechargeable device is able to provide automated assistant functionality in an effective manner. Consumption can be prevented.

In some implementations, a separate speech recognition model run by the device SoC is a speech recognition model of the user's voice based on data provided by the DSP, such as audio data generated in response to the user giving an initial utterance. characteristics can be determined. Based on the determined voice characteristics, the device SoC may select wake times during which the device SoC remains operational to proceed with processing any subsequent input from the user. By way of example, the first user generally and /or may be delayed by a few seconds (eg, 3 seconds) on average. The device SoC may allow for this delay and choose a wake time for the device SoC that does not significantly exceed the user's average delay. For example, the selected wake time for the device SoC can be, but is not limited to, (wake time = (user determined average delay) x (1 + N)), where: N" is any number such as, but not limited to, 0.2, 0.5. The same or different wake times may be selected for different users who typically and/or on average have a delay of several seconds (eg, 2 seconds) between giving a call phrase and giving a command phrase. In this way, rechargeable devices, including DSPs and device SoCs, can adaptively manage "wake times" per user to ensure responsiveness without wasting power and/or computational resources.

In some implementations, a speech recognition model operated by the DSP and another speech recognition model operated by the device SoC both determine whether the user has given a specific call phrase to invoke the automated assistant. can be used to determine However, a speech recognition model run by a DSP may apply less stringent standards for determining whether a user has given a call phrase than the standards enforced by another speech recognition model run by the device SoC. . In other words, a speech recognition model can be associated with a first accuracy threshold for determining whether a particular utterance corresponds to a calling phrase; corresponds to the call phrase can be associated with a second accuracy threshold. By way of example, a first accuracy threshold may be met when a user gives a particular utterance that is determined to contain at least part of a call phrase but also some amount of background noise. Certain utterances, however, have a higher degree of correlation between the particular utterance and the call-phrase than the second accuracy threshold -- at least as much as the degree of correlation to meet the first accuracy threshold. , may not meet the second accuracy threshold.

In some implementations, the DSP uses less power, less data, fewer channels of audio, lower sampling rate audio, and/or lower than the device SoC uses by another speech recognition model. Quality audio can be used to run speech recognition models. For example, the DSP may receive a single channel of audio data when the user speaks to the rechargeable device, and the device SoC may receive multiple channels of audio data when the user speaks to the rechargeable device. channels can be received. Additionally or alternatively, the DSP can operate using an average amount of power when using the speech recognition model and the device SoC uses a different speech recognition model. When powered, it is possible to operate using more than the average amount of power.

In some implementations, the degree of correlation determined by the DSP and/or device SoC is used to select the amount of wake time that the device SoC remains active to process further input from the user. obtain. For example, a first wake time may be selected by the device SoC when the device SoC determines a first degree of correlation between the utterance and the call phrase. However, when the device SoC determines a second degree of correlation between another utterance and the calling phrase, and the second degree of correlation is greater than the first degree of correlation, the device SoC determines the first may select a second wake time that is longer than the wake time of . In this way, the amount of time the device SoC remains active waiting in anticipation of further input from the user is determined by the interaction between the utterance from the user and the call phrase used to invoke the automated assistant. It can be adapted according to the degree of accuracy and/or correlation between. This can save computational resources in rechargeable devices by avoiding standard "wake times" that do not distinguish between any content and/or context of user input.

This can be particularly beneficial if the correlation between the utterances from the user and the calling phrase does not reach the correlation required to call the automated assistant. This means that a "near miss" (i.e., an utterance with a correlation that is almost but not quite enough to invoke the assistant) is far from the correlation required to invoke the assistant. This is because it is more likely to have arisen from an actual attempt to invoke an automated assistant than it is) and thus more likely to be followed by the user attempting to invoke the assistant again. Keeping the SoC active longer when a "sad failure" is detected may allow the device to process subsequent call attempts with shorter latencies. In other words, the device SoC may determine that the utterance is less than correlated with the call phrase by a certain degree, and thus the device SoC may determine that the utterance is less than A certain amount of time in seconds that is selected based on and/or proportional to the time). However, when the device SoC determines that another utterance is less (e.g., less than a greater degree) correlated with the call phrase, the device SoC is much faster to save power and computational resources. can be shut down to

When speech is detected by the rechargeable device and the device SoC is initialized for further processing, there may be differences in the clock settings of the clock running in the DSP and another clock running in the device SoC. . To further eliminate the waste of computational resources involved in waiting and/or responding to speech to be received at the rechargeable device, the device SoC is generated and timestamped at the DSP. Time synchronization may be performed in the DSP and/or device SoC to process the audio data. Such time synchronization can be particularly useful, for example, when the SoC is outputting audio when speech is received. Indeed, without time synchronization, processing captured audio data to remove audio output by the SoC from data corresponding to speech can be problematic.

In some implementations, time synchronization may be performed by the device SoC using one or more timestamps generated at the device SoC and one or more other timestamps generated at the DSP. As an example, a DSP may use a first clock to generate a first timestamp corresponding to a local time associated with the DSP. Additionally, the DSP may generate a second timestamp, for example, when the DSP causes the device SoC to initialize in response to the DSP determining that the user provided the call phrase. Upon receiving a signal (e.g., wake and/or interrupt command) from the DSP, the device SoC can generate a third timestamp using the second clock, The 3 timestamps may correspond to local time relative to the device SoC.

To perform time synchronization, the device SoC uses the first time stamp, the second time stamp, and the third time stamp to generate a time offset, and then converts the audio data generated in the DSP to A time offset can be used when processing. In some implementations, the device SoC determines an average value of the first timestamp and the second timestamp, then a delta corresponding to the difference between the average value and the third timestamp. value can be determined. The difference value may then be used when the device SoC is processing audio data, such as when performing echo cancellation. During echo cancellation, the device SoC can use the difference value to remove instances of audio being output by the rechargeable device from audio recorded by the microphone. As an example, when the device SoC is generating audio output corresponding to music playback, and the user speaks into the microphone while the music is playing, the audio data characterizing the utterances is transferred to the device SoC to remove instances of music playback. can be processed by Furthermore, this process of removing instances of music playback from the audio data can be accurately performed using the difference values determined by the device SoC and/or DSP, thereby allowing the device SoC's " Allows the "wake time" to be determined from accurate data. In other words, timestamps generated by the DSP may be transformed to correlate with timestamps generated by the device SoC for the purpose of performing certain audio processes such as echo cancellation. Additionally or alternatively, timestamps generated by the device SoC may be transformed to correlate with timestamps generated by the DSP for purposes of executing their audio processes.

In some implementations, the rechargeable device has one or more interfaces that should render audio, visual, haptic, and/or any other kind of output in response to casting requests from another computing device. can contain. However, while such casting requests may be provided by cell phones and/or other rechargeable devices such as laptop computers, the computing device providing the casting request may use the available power in the rechargeable device. It is possible to give such requests without consideration. To handle frequent cast requests while still eliminating wasted rechargeable power, the rechargeable device can offload the processing of certain requests to subsystems of the rechargeable device that are not the device SoC. For example, the WiFi chip of the rechargeable device may be tasked with handling certain requests received over the local area network (LAN) to which the rechargeable device and casting device are connected. In some implementations, the WiFi chip may process certain cast requests while the device SoC remains in sleep mode to avoid wasting power and computational resources. Requests left to the WiFi chip for processing that do not invoke the device SoC for further processing can be casting requests that specify one or more specific ports. Additionally or alternatively, the WiFi chip may be tasked with processing mDNS broadcast data without invoking the device SoC.

As an example, a user may operate a music application on their cellular device to stream music, and while music is playing, the user may initiate casting of music to the rechargeable device. can. A cellular device can send casting requests that may include mDNS broadcast data to a variety of different devices that are connected to a LAN with rechargeable devices. The rechargeable device can receive casting requests when the rechargeable device is operating through sleep mode, in which the device SoC is asleep, off, or otherwise. It is a lower power mode than if the rechargeable device were not working due to sleep mode. The WiFi chip of the rechargeable device may first process the casting request to determine if the casting request specifies a particular port and/or contains particular properties.

The WiFi chip may avoid calling the device SoC to respond to the casting request when the casting request specifies specific ports that correspond to one or more predetermined ports. Rather, the WiFi chip can rely on cached data stored in the WiFi chip's memory to generate response data to send back to the cellular device over the LAN. Additionally or alternatively, the WiFi chip may avoid calling the device SoC if the mDNS broadcast data included with the casting request specifies certain parameters of the casting request. For example, mDNS broadcast data provided by a cellular device may indicate that an audio playback service is being requested and/or that a particular application has initiated a casting request. The WiFi chip's cached data may indicate that the rechargeable device supports audio playback services and/or specific applications based on previous interactions with one or more other devices. Therefore, based on the available cached data, the WiFi chip may use the cached data to generate a response to the cellular device without calling the device SoC for further information. In this way, the rechargeable device can reduce the number of times the device SoC would otherwise be initialized for processing, thereby allowing the rechargeable power source (e.g., one or more batteries and/or capacitors) and waste of computational resources.

The above description is provided as an overview of some implementations of the disclosure. Further discussion of those and other implementations are presented in more detail below.

Other implementations employ one or more processors (e.g., central processing units) to perform methods such as one or more of the methods described above and/or elsewhere herein. CPUs), graphics processing units (GPUs), and/or tensor processing units (TPUs)). Still other implementations are operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein. It may include a system of one or more computers and/or one or more robots including one or more processors.

It should be understood that all combinations of the above concepts and the additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein.

FIG. 10 illustrates a user controlling a first client device to broadcast media to separate client devices without causing all client devices on the local network to transition out of their sleep state. FIG. 2B illustrates a user controlling a first client device to broadcast media to separate client devices without waking all client devices on the local network from their sleep states. FIG. 2 is a diagram of a scenario in which a device SoC is left in sleep mode according to implementations discussed herein; FIG. 2 is a diagram of a scenario in which a device SoC is transitioned out of sleep mode according to implementations discussed herein; FIG. 2 is a diagram of a scenario in which a device SoC is transitioned out of sleep mode according to implementations discussed herein; 1 is a diagram of a client device capable of determining wake times for the device SoC based on one or more properties associated with interactions between a user and the client device; FIG. Generate a wake time for the device SoC, generate a time offset between the DSP's clock and the device SoC's clock, and/or use the WiFi chip to respond to casting requests without bringing the device SoC out of sleep mode. Figure 1 shows a system for use; 1 illustrates a method for initializing a particular processor of a computing device for a selected amount of time based on one or more characteristics of interaction between the user and the computing device; FIG. FIG. 4 illustrates a method for processing audio data using determined time offsets corresponding to differences in operation between a first processor and a second processor; FIG. 3 illustrates a method for providing response data to a broadcasting device using a WiFi chip included in a battery-powered computing device; 1 is a block diagram of an exemplary computer system; FIG.

FIG. 1A illustrates a diagram 100 of a user 124 controlling a first client device 134 to broadcast media to separate client devices without causing all client devices on the local network to transition out of their sleep state. indicates First client device 134 may be a computing device such as cellular phone 126 and/or any other device capable of casting media to another device. First client device 134 can provide access to automated assistant 136 that can be invoked by input to assistant interface 138 . The first client device 134 can also include one or more applications 144 that can access media that can be cast from the first client device 134 to another client device. In some implementations, first client device 134 can cast media to separate client devices in response to input from user 124 to automated assistant 136 . For example, in response to a user examining other devices available for casting media related to application 144, first client device 134 may be configured to connect to multiple different client devices over a local area network. Can send mDNS data.

The mDNS data broadcast by first client device 134 may be sent to second client device 102 and/or third client device 112 over a local area network, such as a WiFi network. For example, mDNS data 130 can be sent from first client device 134 to second client device 102, and mDNS data 132 can be sent from first client device 134 to third client device 112. It is possible to be In some implementations, second client device 102 may be a portable computing device powered by portable power source 110 . Additionally, the third client device 112 may be powered by a portable power source and/or any other power source such as power supplied by a utility service. Second client device 102 and third client device 112 may operate in sleep mode as each device receives its respective mDNS data. In other words, since the devices are operating in sleep mode, the WiFi chips available in each device can process mDNS data without waking the respective device out of sleep. For example, device SoC 108 and device SoC 118 may operate in sleep mode (as indicated by the gradient fill pattern) when WiFi chip 106 and WiFi chip 116 receive and respond to mDNS data. In some implementations, the computing device consumes at least less power than if the device SoC of the computing device were powered down or otherwise the device SoC was operating according to another mode of operation. can be considered to be in "sleep mode".

WiFi chip 106 of second client device 102 may process mDNS data 130 using cached data available in memory 140 of WiFi chip 106 . Additionally, the WiFi chip 116 of the third client device 112 can process the mDNS data 132 using cached data available within the memory 142 of the WiFi chip 116 . The mDNS data broadcast by the first client device 134 may include the application associated with the broadcast, the port for sending the broadcast, the service being requested by the first client device 134, and/or the computing device performing the casting. Any other characteristics that may be specified during initialization can be specified.

FIG. 1B shows a diagram 150 of second client device 102 and third client device 112 responding to mDNS data sent to each client device. While generating and transmitting response data, each of device SoC 108 and device SoC 118 can remain in sleep mode, thereby wasting power and computational resources. Response data 148 may indicate whether the second client device 102 includes one or more characteristics requested by the first client device 134 and response data 146 may indicate whether the third client device 112 contains one or more features requested by the first client device 134 . In response to first client device 134 receiving response data 148 and response data 146 over the local network, first client device 134 provides a graphical interface identifying one or more client devices that satisfy the request. can be provided. User 124 can then select one of the client devices to cast the media. For example, when the application 144 requests a client device to cast and the user is presented with a list of client devices to select for casting, the user may select the first client device 134 .

Depending on the choice, the application 144 can communicate directly with the first client device 134 over the local network, or the application can communicate separately to render specific media data by the first client device 134. It is possible to communicate with a separate server to have the server communicate instructions to the first client device 134 . In some implementations, first client device 134 may be a standalone speaker device 122 and/or display device capable of rendering audio and/or visual data. Alternatively or additionally, third client device 112 may be a display device such as a computer monitor and/or television. The second client device 102 and the third client device 112 can each include a digital signal processor, which is an automated assistant when the respective device SoC is operating in sleep mode. can monitor each device interface that accesses the Additionally, the client device's digital signal processor (DSP), WiFi chip, device SoC, and/or any other subsystems may operate with any of the implementations discussed herein.

FIG. 2A shows a diagram 200 in which a user 220 gives an utterance 218 to a client device 202 and causes the client device 202 to process the utterance 218 using a digital signal processor without causing the device SoC 208 to transition out of sleep mode. Client device 202 can be a computing device 222 operated by power source 210, which can include a rechargeable power source such as a battery, capacitor, and/or any other rechargeable energy source. When the client device 202 is operating with the device SoC 208 in sleep mode, the user 220 can provide an utterance 218 that causes the client device 202 to transition the device SoC 208 out of sleep mode. It can be different from the calling phrase. For example, user 220 may give utterance 218 “Hello...” that may be received at one or more microphones connected to client device 202 .

A microphone connected to client device 202 can provide output in response to user 220 giving speech 218 . Although the device SoC 208 operates in sleep mode, the digital signal processor DSP 204 of the client device 202 can call the client device 202 to perform one or more different actions. The output of the microphone may be monitored to determine if the user provided one of the call phrases. In some implementations, DSP 204 may process audio data 212 that characterizes speech 218 by a process that utilizes a lower sampling rate than the sampling rate used by device SoC 208 to process the audio data. Alternatively or additionally, the DSP 204 generates audio data based on output from a smaller number of microphones compared to the number of microphones used to generate the audio processed by the device SoC 208. 212 can be processed. In other words, DSP 204 may utilize fewer channels of audio data compared to the amount of channels utilized by device SoC 208 . Utilizing lower sampling rates and/or fewer channels can be computationally efficient and minimize power consumption (and consequent battery consumption). Alternatively or additionally, DSP 204 may access first model 214 to process audio data 212 to determine if user 220 provided a call phrase.

The first model 214 can be different than the second model 216 used by the device SoC 208 to determine if the user 220 said the call phrase. For example, first model 214 can be a model trained to determine whether audio data characterizes a call phrase. The correspondence between the audio data and the call phrase can be characterized as one or more values, and the threshold degree of similarity between the audio data and the call phrase is determined by the second model 216 can be lower than another threshold magnitude corresponding to . In other words, it may be determined that the utterance satisfies the threshold of the first model 214 but not the threshold of the second model 216, but the utterance satisfies the second model 216 and satisfies the first model 214. It cannot be determined that there is no

In various implementations, the second model 216 is larger (in terms of bits) than the first model 214 and has a larger input dimension (eg, for processing more channels of audio data). and/or have a larger amount of trained nodes. As a result, processing audio data using the second model 216 may be more computationally expensive than processing audio data using the first model 214 . However, in some implementations, processing the audio data utilizing the second model 216 may result in the second model 216 being larger, the more channels of audio data being processed, the Accurate samples and/or higher sampling rates of audio data are processed, which may result in a more accurate determination of whether user 220 said a call phrase. Therefore, the DSP 204 can utilize the more efficient first model 214 to determine whether the audio data passes the "initial check" for the presence of the call phrase, and the "initial check" Only if it passes can the SoC 208 and the less efficient (but more accurate) second model 216 be utilized. This is more efficient in terms of resources than utilizing only the SoC 208 and the second model 216.

In some implementations, DSP 204 may process audio data at a different bit depth than the bit depth at which device SoC 208 processes audio data. For example, the DSP 204 captures audio data as 24-bit audio, but converts the audio data to 16-bit audio data and then converts the audio data to 16-bit audio data when determining whether the audio data characterizes a call phrase given by the user. Bit audio data can be used. DSP 204 may cause the captured 24-bit audio data to be transferred to device SoC 208 when DSP 204 determines that the 16-bit audio data characterizes the call phrase. The device SoC 208 can then process the 24-bit audio data rather than converting to different bit depths to process the transferred audio data.

In response to user 220 providing utterance 218, DSP 204 processes audio data 212 using first model 214, utterance 218 does not correspond to one of the one or more call phrases. can be determined. Accordingly, the DSP 204 may avoid waking the device SoC 208 from sleep for further processing. In this way, the device SoC 208 does not need to be initialized frequently to further process the audio data 212 and can remain in sleep mode. This allows client device 202 to conserve energy provided by power source 210 and computing resources available at client device 202 .

FIG. 2B shows a diagram 230 in which the user gives an utterance 234 to the client device 202 causing the DSP 204 of the client device 202 to wake up the device SoC 208 of the client device 202 . Speech 234 can be captured by one or more microphones of client device 202, which can be a computing device 222 powered by a portable and/or rechargeable power source 210. is. Initially, device SoC 208 may operate in sleep mode to save power and computational resources. While device SoC 208 is operating in sleep mode, DSP 204 may operate to detect when user 220 gives an utterance corresponding to one or more call phrases.

As an example, the user 220 can give an utterance 234, such as "Assistant," which can correspond to a call phrase that can wake the device SoC 208 from sleep to the DSP 204 when detected by the DSP 204. To detect call phrases, DSP 204 may convert output from one or more microphones of client device 202 into audio data 232 . DSP 204 can use first model 214 to process audio data 232 to determine if utterance 234 corresponds to a calling phrase. When the DSP 204 determines that the utterance 234 corresponds to the call phrase, the DSP 204 sends a command to the device SoC 208 to wake up the device SoC 208 or otherwise transition the device SoC 208 out of sleep mode. can do.

When DSP 204 transitions device SoC 208 from sleep mode to operational mode, DSP 204 may also send audio data to device SoC 208 for further processing. The device SoC 208 can then process the audio data using the second model 216 to see if the utterance 234 corresponds to the call phrase. When device SoC 208 determines that utterance 234 did not correspond to a call phrase, device SoC 208 can transition back to sleep mode to conserve computational resources and power. Alternatively or additionally, when device SoC 208 determines that utterance 234 does not correspond to a call phrase, but DSP 204 determines that utterance 234 does It can remain active or awake for a period of time in anticipation of further input. In some implementations, the wake time may be based on the degree of correlation between the utterance 234 and the calling phrase, the identification of the user's 220 voice, and/or any other implementation features discussed herein. . As an example, the wake time may be determined based on a comparison between the degree of correlation detected by device SoC 208 and a threshold degree of correlation. For example, when the degree of correlation detected by the device SoC 208 is 0.87 and the threshold degree of correlation is 0.9, the wake time of the device SoC 208 may be set to X period of time. However, if the degree of correlation detected by the device SoC 208 is 0.79 and the threshold degree of correlation is 0.9, the wake time of the device SoC 208 can be set to a period of time Y, Y is shorter than X.

FIG. 2C shows a diagram 240 in which an utterance 244 is given by the user 220, causing the DSP 204 of the computing device 202 to transition the device SoC 208 out of sleep mode and further initialize the automated assistant for further operation. Client device 202 can be a computing device 222 that includes one or more different interfaces for interacting with the automated assistant. To initialize the automated assistant, user 220 may provide a call phrase that is detected by DSP 204 when device SoC 208 is in sleep mode. The call phrase may be included in speech 244 that is processed using audio data 242 and first model 214 when detected by DSP 204 . DSP 204 may provide a wake command to device SoC 208 when DSP 204 determines using the first model that audio data 242 characterizes a call phrase.

In response to device SoC 208 receiving the wake command, device SoC 208 can process audio data corresponding to utterance 244 using second model 216 . Based on processing the audio data using the second model 216, the device SoC 208 may determine that the utterance 244 included a call phrase. Thus, based on the device SoC 208 determining that the user 220 provided the calling phrase, the device SoC 208 initializes the automated assistant locally and/or calls the automated assistant via the server device. can be given a network request to initialize the . For example, device SoC 208 may send data to WiFi chip 106 of client device 202 to initialize an automated assistant. Data may be sent to the automated assistant server over a network, such as the Internet, such that subsequent requests from the user may be sent to the automated assistant server via the client device 202. In some implementations, the automated assistant can be hosted on the client device 202, and thus a request from the user 220 for the automated assistant to perform a particular action is processed on the client device 202. can be processed. By allowing device SoC 208 to sleep to conserve power and other resources, and wake from sleep to validate specific utterances from user 220, client device 202 saves computational and power resources. can be saved, which can be particularly advantageous for client devices 202 that operate using a rechargeable power source 210 .

FIG. 3 shows a diagram 300 of a client device 302 that can determine wake times for the device SoC 308 based on one or more properties associated with interactions between a user 320 and the client device 302. FIG. User 320 can interact with client device 302 to invoke an automated assistant to perform one or more different functions. For example, client device 302 may be a stand-alone speaker device 322 capable of rendering audio, such as music, and/or controlling various other client devices connected to a common network with client device 302. is possible. The client device 302 may be controlled by multiple different users who speak and/or interact with the client device 302 differently. To accommodate such differences between users while still avoiding wasting power and computational resources, the client device 302 uses the device SoC to limit the amount of time that the device SoC 308 monitors input from the user 320 . A wake time 324 of 308 can be determined.

As an example, the user 320 may give an utterance 318 such as "Assistant, could you..." and then pause briefly to consider how to continue the utterance. User 320 may have habits or histories that indicate such intervals while interacting with client device 302 . Thus, data characterizing previous interactions between the user 320 and the client device can be used to determine how long to monitor for further input from the user 320 without wasting client device 302 resources. . For example, in response to utterance 318, DSP 304 of client device 302 may process audio data 312 that characterizes utterance 318 to determine whether audio data 312 characterizes a call phrase, such as "Assistant." When DSP 304 determines that utterance 318 includes a call phrase, DSP 304 may communicate with device SoC 308 to cause device SoC 308 to transition from sleep mode to operational mode. In some implementations, the DSP 304 may also send audio data 312 to the device SoC 308 to confirm that the user 320 gave the call phrase.

In some implementations, when device SoC 308 determines that user 320 did give the call phrase, device SoC 308 may further process audio data 312 to identify the user who gave utterance 318 . For example, device SoC 308 may access a voice identification model, with permission from the user, to identify one or more voice characteristics encompassed by audio data 312 . Based on the vocal characteristics contained by the audio data 312, the device SoC 308 may rank one or more different users according to whether the utterances 318 correspond to their particular vocal characteristics. . The highest ranked user can then be selected as the user who gave the utterance 318, and the device SoC 308 determines the wake time 324 based on identifying the highest ranked user. be able to. Alternatively or additionally, a user can be selected by device SoC 308 using one or more models that can be used to generate a prediction of the origin of utterance 318 . Alternatively or additionally, audio data 312 can be processed using one or more models that can also be used to generate wake times 324 .

In response to determining that the user 320 provided the call phrase, the device SoC 308 can communicate with the WiFi chip 306 to initialize the automated assistant over a wide area network such as the Internet. However, in some implementations, device SoC 308 may initiate a local device automated assistant that communicates with client device 302 over a local area network. While the automated assistant is initializing, device SoC 308 may monitor one or more interfaces of client device 302 for an amount of time at least equal to wake time 324 . When wake time 324 elapses, device SoC 308 can return to sleep mode and DSP 304 can take over monitoring output from one or more interfaces of client device 302 .

In some implementations, wake time 324 may be based on a determined degree of correlation between utterance 318 and the calling phrase. For example, device SoC 308 and/or DSP 304 may generate a value that characterizes the degree of correlation between utterance 318 and the call phrase. The amount of wake time 324 can decrease as the degree of correlation increases, and the amount of wake time 324 can increase as the degree of correlation decreases. In other words, the device SoC 308 determines that the utterance 318 falls within a 10% tolerance of the threshold for confirming that the utterance 318 contains a call phrase, and the wake time 324 is 1 minute. It is possible. However, when device SoC 308 determines that utterance 318 does contain a call phrase and thus meets the threshold, wake time 324 may be set to 5 seconds. Note that the wake time can be any amount of milliseconds, seconds, minutes, and/or any other time value on which processor operations can be based. For example, an utterance that correlates more closely with the calling phrase may result in a wake time that has fewer total milliseconds than a wake time resulting from a different utterance that correlates less closely with the calling phrase.

Figure 4 illustrates generating a wake time for the device SoC 444, generating a time offset between the DSP 442 clock and the device SoC 444 clock, and/or without transitioning the device SoC 444 out of sleep mode. A system 400 is shown for operating a computing device 418 to conserve computing resources by using a WiFi chip 434 to respond to casting requests. Automated assistant 404 may operate as part of an assistant application provided on one or more computing devices, such as computing device 418 and/or server device 402 . The user is automated through an assistant interface, which can be a microphone, camera, touch screen display, user interface, and/or any other device capable of providing an interface between the user and the application. The assistant 404 can be interacted with.

For example, a user may verbally enter the assistant interface to cause the automated assistant 404 to perform functions (e.g., provide data, control peripheral devices, access agents, generate input and/or output, etc.). Automated assistant 404 may be initialized by providing physical, textual, and/or graphical input. Computing device 418 can be a display panel that includes a touch interface for receiving touch input and/or gestures to allow a user to control applications on computing device 418 via the touch interface. may include a display device that is In some implementations, computing device 418 may not have a display device, thereby providing audible user interface output without providing graphical user interface output. In addition, computing device 418 may provide a user interface, such as a microphone, for receiving spoken natural language input from a user. In some implementations, computing device 418 may include a touch interface, may not have a camera, but may optionally include one or more other sensors.

Computing device 418 and/or other computing device 434 may communicate with server device 402 over a network 440, such as the Internet. Additionally, computing device 418 and other computing device 434 may communicate with each other via a local area network (LAN), such as a WiFi network. Computing device 418 can offload computing tasks to server device 402 to conserve computing resources on computing device 418 . For example, server device 402 can host automated assistant 404 and computing device 418 can transmit input received at one or more assistant interfaces 420 to server device 402 . However, in some implementations, automated assistant 404 may be hosted as client automated assistant 422 on computing device 418 .

In various implementations, all or some aspects of automated assistant 404 may be implemented on computing device 418 . In some of those implementations, aspects of automated assistant 404 are performed by automated assistant 422 on a client of computing device 418 and interface with server device 402 to perform other aspects of automated assistant 404. I take the. The server device 402 may optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or some aspects of automated assistant 404 are performed by client automated assistant 422 of computing device 418, client automated assistant 422 includes the computing device 418 operating system and It can be an application that is separate (e.g., installed "on top of" the operating system)--or alternatively is implemented directly by the operating system of computing device 418 (e.g., the operating system's However, it can be considered an application integrated with the operating system).

In some implementations, automated assistant 404 and/or client automated assistant 422 have multiple different modules to process input and/or output for computing device 418 and/or server device 402. can include an input processing engine 406 that can use the For example, input processing engine 406 may include speech processing module 408 that can process audio data received at assistant interface 420 to identify text contained in the audio data. Audio data may be sent from the computing device 418 to the server device 402 , for example, to conserve computing resources of the computing device 418 .

A process for converting audio data to text may include speech recognition algorithms that may use neural networks and/or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data is parsed by the data analysis module 410 and made available to automated assistants as text data that can be used to generate and/or identify command phrases from the user. obtain. In some implementations, the output data provided by the data analysis module 410 is used by the user to identify specific actions and/or routines that may be performed by the automated assistant 404 and/or applications or routines that may be accessed by the automated assistant 404. It can be provided to the parameter module 412 to determine if it has provided the corresponding input to the agent. For example, assistant data 416 can be stored on server device 402 and/or stored on computing device 418 as client data 432, automated assistant 404 and/or client automated assistant. It may contain data defining one or more actions that may be performed by 422 and the parameters required to perform the action.

In some implementations, a computing device may include a WiFi chip 434 that may include at least one or more portions of memory 436 and/or broadcast engine 438 . Broadcast engine 438 receives broadcast data from one or more other client devices over network 440 and generates response data using cached data stored in memory 436. can be done. WiFi chip 434 may store data characterizing available services, applications, hardware features, and/or any other properties and/or capabilities that may be associated with computing device 418 . When the computing device 418 is operating in sleep mode in which the device SoC 444 consumes less power and/or computational resources than when the computing device 418 is operating in wake mode , the WiFi chip 434 may respond to casting requests from other client devices without causing the device SoC 444 to transition out of sleep mode.

For example, when a request from a client device is received at WiFi chip 434 and the request identifies a target service that is also characterized by data stored in memory 436, broadcast engine 438 retrieves cached data from memory 436. can be used to generate the response data and provide the response data to the client device. Once the client device has selected the computing device 418 to use the target service, the client device can send commands to the computing device 418, the WiFi chip 434 processes the commands, and the device SoC The 444 can be transitioned from wake mode to run mode. However, to determine whether the computing device 418 is capable of providing a particular service, initializing a particular application, and/or otherwise providing a service to a requesting client device. If the broadcast engine 438 determines that the memory 436 does not contain enough data, the WiFi chip 434 can communicate with the device SoC 444 to process the request. In this case, the device SoC 444 can generate response data and provide the response data to the WiFi chip 434, which can transmit the response data to the client device.

In some implementations, computing device 418 includes one or more assistant interfaces 420 that can provide access to client automated assistant 422 and/or automated assistant 404 . A user may provide one or more different types of input to invoke client automated assistant 422 and/or automated assistant 404 . Such input can include spoken input, which can be processed by digital signal processor 442 when device SoC 444 is operating in sleep mode. One or more speech recognition models 440 available at the computing device 418 determine whether the audio data characterizing the spoken input contains a call phrase for initializing the automated assistant. can be used for Additionally, one or more speech recognition models 440 may be used by the wake time engine 448 to determine the amount of time the device SoC 444 should remain awake to detect subsequent input from the user. . In some implementations, the amount of wake time may be based on the degree of similarity between the user's utterance and the calling phrase for calling the automated assistant. Alternatively or additionally, the amount of wake time may be based on audio processing engine 430 processing the audio data corresponding to the utterance to identify the user who gave the utterance. For example, the audio processing engine 430 uses the client data 432 and/or the assistant data 416 to communicate between the user and the automated assistant, such as how long the user typically pauses during an interaction with the automated assistant. can determine characteristics of interactions between The wake time engine 448 can use this information to generate wake times for the device SoC 444 during certain interactions between the user and the automated assistant.

Additionally or alternatively, power engine 426 of computing device 418 may determine the estimated charge of power source 446 and communicate the estimated amount of charge and/or operating time to wake time engine 448 . The amount of charge and/or amount of operating time estimated by the power engine may be used by wake time engine 448 to determine wake time for device SoC 444 . For example, when a user generally pauses longer than the average user when interacting with an automated assistant, and the power supply 446 is fully charged, the wake time engine 448 estimates at least otherwise. An extended wake time may be allocated compared to the wake time allocated if the battery was less than 50% charged. Alternatively or additionally, when the user is interacting with the computing device 418 for generally shorter intervals than the average user and the power source 446 is fully charged, the wake time engine 448 may at least can be allocated shorter wake times relative to extended wake times to conserve power.

In some implementations, computing device 418 may include time offset engine 424 for determining offsets between clocks used by computing device 418 . For example, DSP 442 may run a first clock and device SoC 444 may run a second clock that may be offset from the first clock during operation of computing device 418 . This offset can affect operation in audio processing engine 430 , particularly when audio processing engine 430 is performing echo cancellation on spoken input to assistant interface 420 .

In some implementations, the offset between a first clock on which DSP 442 operates and a second clock on which device SoC 444 operates may be determined using timestamps. A timestamp may correspond to a pair of clock values including a clock value captured using a first clock and another clock value captured using a second clock. When the DSP 442 is working to determine if a call phrase has been detected, and the device SoC 444 is in sleep mode, the DSP 442 clocks a clock corresponding to the "wake" time when the call phrase is detected. values can be recorded. When the DSP 442 transitions the device SoC 444 out of sleep mode, timestamps may be recorded using the first clock and the second clock. However, to determine the "wake" time expressed relative to the second clock, the second clock value of the timestamp is the determined time between the first clock and the second clock. It can be "scaled" and/or otherwise adjusted according to the offset.

The time offset may be determined using a first timestamp and a second timestamp that may be recorded when both device SoC 444 and DSP 442 are both not in sleep mode. The first timestamp can correspond to the first pair of clock values and the second timestamp can correspond to the second pair of clock values. A first DSP clock value of the first pair of clock values may be subtracted from a second DSP clock value of the second pair of clock values to generate a first clock difference value. . Further, the clock values of the first SoC of the first pair of clock values are subtracted from the clock values of the second SoC of the second pair of clock values to generate the second clock difference value. he can Then, when the DSP 442 wakes the device SoC 444 from sleep to determine when a call phrase has been received, the mapping between the first clock difference value and the second clock difference value is can be used. For example, the ratio of the second clock difference value to the first clock difference value can be determined, the ratio being used by the DSP clock to determine the corresponding device SoC clock value. Values can be multiplied. For example, when DSP 442 wakes device SoC 444 from sleep, the DSP's clock value corresponding to the time the user gave the call phrase may be provided to device SoC 444 . The device SoC 444 can then map the DSP's clock value to the device SoC's clock value to determine when a call phrase was given by the user relative to the device SoC's clock. This value can then be used during processing of the audio data, such as during echo cancellation, to analyze the content of the audio data (eg, to identify the natural language content of utterances from the user).

FIG. 5 illustrates a method 500 for initializing a particular processor of a computing device for a selected amount of time based on one or more characteristics of interaction between a user and the computing device. . Method 500 may be performed by one or more processors, applications, and/or any other device and/or module capable of providing an interface between a user and an automated assistant. Method 500 may include an act of determining 502 whether the first processor has detected speech from the user. The first processor may be operational while the second processor is operating in sleep mode. Sleep mode causes the second processor to consume less power and/or less computational resources than operating modes in which one or more applications, such as an automated assistant, are actively running by the second processor. It is possible to be in a mode where In some implementations, the first processor can be a digital signal processor and the second processor can be the device SoC. Both the first processor and the second processor may be incorporated into a computing device operating using a rechargeable power source such as a battery, capacitor, and/or any other rechargeable power source. .

Method 500 may proceed to operation 504 when the first processor detects speech. Otherwise, when speech has not been detected by the first processor, the first processor continues to monitor one or more microphones of the computing device to determine whether the user has provided speech. can be done. Act 504 may include determining by the first processor whether the utterance includes the particular call phrase. The computing device may be operable to transition the second processor out of sleep mode when a particular one of the one or more different call phrases is provided to the computing device by the user. The invocation phrase can be, for example, "Assistant" and/or any other phrase that can be used to initialize the application. Method 500 may proceed from operation 504 to operation 508 when the first processor determines that the utterance includes a call phrase.

Act 508 may include transitioning the second processor from sleep mode to operational mode. Operation 508 may be performed by the first processor in response to the first processor identifying the call phrase. However, method 500 may proceed from operation 504 to operation 506 when the first processor determines that the utterance does not include a call phrase. Act 506 may include avoiding transitioning the second processor from sleep mode to operational mode. In other words, since the first processor does not detect the call phrase within the utterance, the first processor returns to operation 502 to determine if another utterance has been detected.

Method 500 may proceed from operation 508 to operation 510, which may include providing audio data from the first processor to the second processor. The audio data may correspond to utterances given to the computing device by a user. In some implementations, a first processor may operate a first call-phrase model to determine whether an utterance contains a call-phrase, while a second processor determines whether the utterance contains a call-phrase. A second call phrase model can be run to determine whether the The first model can accommodate a lower threshold for identifying correspondence between utterances and call-phrases, while the second call-phrase model can accommodate the correspondence between utterances and call-phrases. It is possible to accommodate a higher threshold than that of the first model for determining correspondence. Thus, when the second processor receives the audio data, the second processor can use the second call phrase model to determine whether the utterance includes the call phrase.

Method 500 may include an optional act 512 of determining, by the second processor, the degree to which the audio data characterizes the call phrase. The degree to which the audio data characterizes the calling phrase can be one or more metrics that quantify one or more similarities between the audio data and the calling phrase. In this manner, one or more metrics may later be used to make decisions about how the computing device should subsequently operate. For example, a value characterizing the degree to which the audio data characterizes the call phrase should be used to operate the second processor before transitioning to sleep mode and back (if no other audio data is passed to the second processor for processing). It can be used to determine the amount of time to run the mode.

In some implementations, the method 500 may include an optional act 514 of determining, by the second processor, vocal characteristics contained by the audio data. A second processor can operate a voice identification model that can be used, with permission from one or more users, to identify the user who gave the utterance (eg, the user's corresponding user profile). . For example, each user of a computing device may speak with a different and/or unique voiceprint, and based on these differences, a voice identification model may rank corresponding predictions of which user gave the utterance. can be determined. The user corresponding to the highest ranking may be selected as the user who gave the utterance to the computing device. User identification using the voice identification model may be used, with the user's permission, to determine the amount of time the second processor is to operate in active mode rather than sleep mode. The amount of time may be based on previous interactions between one or more users and the automated assistant. For example, a larger amount of time can be selected for users who generally have a delay between giving a call phrase and giving a subsequent command, while A smaller amount of time can be selected for another user with generally no delay between giving subsequent commands of .

Method 500 may further include operation 516 operating in an operational mode with the second processor for at least an amount of time based on one or more characteristics of the interaction between the user and the computing device. For example, in some implementations the amount of time may be based on the degree to which the audio data characterizes the calling phrase. Alternatively or additionally, the amount of time may be based on the identity of the user who provided the one or more vocal characteristics and/or utterances embodying the audio data. Alternatively or additionally, the amount of time may be the time of day, the number of available computing devices, network strength, the number of users within a particular proximity of a computing device, and/or the user may be based on one or more contextual features corresponding to the interaction between the user and the computing device, such as any other feature that may be associated with the interaction between the user and the computing device.

FIG. 6 illustrates a method 600 of processing audio data using determined time offsets corresponding to operational differences between a first processor and a second processor. Method 600 may be performed by one or more processors, applications, and/or any other device or module capable of processing audio data. The first processor and second processor identified in method 600 may be powered by a portable power source, such as a battery and/or capacitor, and embedded in a computing device that provides access to automated assistants. The method 600 may include an act of determining 602 by the first processor whether the speech was received at the computing device and/or another device in communication with the computing device. In particular, the first processor may process output from one or more microphones to determine whether a user has spoken to one or more microphones. When the first processor determines that the speech has not been received, the first processor monitors the output of the one or more microphones to determine if the speech has been received by the one or more users. can continue.

The method 600 may proceed from operation 602 to operation 604 when the first processor determines that speech has been detected. Act 604 may include determining whether the utterance included a calling phrase. The first processor may determine whether the utterance included the call phrase by using a first call phrase model that may be executed by the first processor. In particular, a first call phrase model may be used to analyze the output of one or more microphones to determine whether the utterance contained the call phrase. Method 600 may proceed from operation 604 to operation 608 when it is determined that the utterance included a call phrase.

Act 608 may include transitioning the second processor from sleep mode to operational mode. Operation 608 may be initiated by the first processor in response to determining that the utterance included the calling phrase. The method 600 may proceed from operation 604 to operation 606 when the first processor determines that the utterance did not include a call phrase. Act 606 may include avoiding transitioning the second processor from the sleep mode to the operational mode, and instead of transitioning the second processor out of sleep mode, the method 600 continues until the subsequent utterance is 1 We can return to operation 602 to detect if one or more microphones have been presented.

Method 600 may further include an act 610 of causing, by the second processor, the computing device to render audio output using the audio output data. Audio output may be provided by one or more interfaces that connect to the computing device. For example, a computing device may include one or more speakers for emitting audio, and/or a computing device communicates with another computing device that includes one or more speakers. Is possible. Audio output data may be based on data received over a network to which the computing device is connected and/or may be based on data stored in memory of the computing device. For example, the audio output can be music rendered using audio data corresponding to music stored in a memory device of the computing device. Audio output data may include or be associated with time data indicating when a portion of the audio was rendered by the computing device and/or output by one or more speakers.

Method 600 may proceed to operation 612 with determining whether a call phrase was detected using the second processor. In some implementations, the first processor can be a digital signal processor and the second processor can be the device SoC. A first processor may run a first speech recognition model and a second processor may run a second speech recognition model. The first speech recognition model can have a lower threshold for determining whether the utterance contains the calling phrase, and the second speech recognition model determines whether the utterance contains the calling phrase. It is possible to have a higher threshold for In some implementations, the first processor may process lower quality audio data than the audio data processed by the second processor. For example, the first processor may monitor the output of one or more microphones of the computing device at a lower sampling rate than the sampling rate at which the second processor monitors the one or more microphones. Alternatively or additionally, the first processor may monitor a smaller total number of audio channels compared to the number of audio channels monitored by the second processor. For example, a first processor may monitor a single microphone to determine if speech is given by the user, and a second processor monitors speech and/or call phrases given by the user. It is possible to monitor more than one microphone to determine if

A second processor may monitor the output of one or more microphones while the audio output is rendered by the computing device. The method 600 may proceed from operation 612 to operation 614 when the second processor determines that the call phrase was provided by the user. When the second processor has not determined that the call phrase was provided by the user, the second processor can continue to monitor the output of one or more microphones of the computing device. Act 614 may include determining, by the second processor, a time offset between the time data and audio input data characterizing the call phrase detected by the second processor. In some implementations, the time offset may be based on the difference between the clock behavior characteristics of the clock of the first processor and another clock of the second processor. However, in some implementations, the first processor and the second processor may operate with a single clock.

The method 600 may further include an operation 616 of processing the audio input data by the second processor using the time offsets to facilitate removal of one or more features of at least the audio input data. In particular, the time offset can be used during echo cancellation to remove features in the rendered audio output from the audio input applied to one or more microphones. By accounting for the time offset between the first processor and the second processor, errors that would otherwise become apparent during echo cancellation can be removed. This can lead to shorter latency between the user giving an utterance and the automated assistant responding to the utterance. Further, because the computing device operates from a rechargeable power source, the operating time for each full charge of the power source can be extended by reducing latency and total operating time for at least the second processor.

FIG. 7 illustrates a method 700 for providing response data to a broadcast device using a WiFi chip included in a battery-powered computing device. A method may be performed by one or more applications, processors, and/or any other device or module capable of processing network data. Method 700 may include an act of determining 702 whether mDNS broadcast data was received at the WiFi chip. When it is determined that mDNS broadcast data was received at the WiFi chip, method 700 can proceed to operation 704 . Act 704 may include determining whether a particular target port is identified by the mDNS broadcast data. When no mDNS broadcast data is received at the WiFi chip in act 702, the WiFi chip continues to monitor network traffic to determine if any packets of data received at the WiFi chip correspond to mDNS broadcast data. can be done.

Method 700 can proceed from operation 704 to operation 706 when the mDNS broadcast data identifies a particular target port, such as a designated port for casting media between client devices. Act 706 may include determining whether the cached data stored in the WiFi chip's memory characterizes one or more characteristics of the mDNS broadcast data. When the mDNS broadcast data does not identify a particular target port, method 700 can proceed from operation 704 to operation 702, where the WiFi chip can continue to monitor network traffic.

In some implementations, at operation 706, the WiFi chip may compare the mDNS broadcast data with cached data stored in the WiFi chip's memory. For example, a WiFi chip may store packets of data previously provided over the network and/or data generated in response to packets received over the network. For example, a WiFi chip may have previously responded to a cast request from another broadcasting device by indicating that the computing device containing the WiFi chip contains an application that is also contained in another broadcasting device. Alternatively or additionally, data stored in memory of the WiFi chip may indicate whether one or more services may be used by the computing device with the broadcast request. Alternatively or additionally, the data stored in the WiFi chip's memory may be indicative of one or more hardware characteristics of the computing device. Alternatively or additionally, the WiFi chip can determine whether cached data stored by the WiFi chip characterizes one or more characteristics associated with mDNS broadcast data. In this way, the WiFi chip can respond to requests broadcast over the network without waking up another processor in a computing device such as the device SoC.

Method 700 can proceed from operation 706 to operation 708, which can include generating response data based on the cached data. Operation 708 may be performed when the WiFi chip has cached data characterizing one or more characteristics associated with mDNS broadcast data. For example, when cached data identifies an application that is the subject of an mDNS broadcast, the WiFi chip may generate response data to indicate to the broadcasting device that the computing device does contain that particular application. In this way, the computing device does not need to wake up another processor to respond to the broadcast data, thereby wasting computational and/or power resources that may be limited for battery-powered devices. lose.

The method 700 may proceed from operation 706 to operation 710 when the WiFi chip's cached data does not characterize one or more characteristics associated with the mDNS broadcast data. Operation 710 may include transitioning device SoC of the computing device from a first mode of operation to a second mode of operation. In some implementations, the first mode of operation can be a mode in which the device SoC is executing fewer processes than the second mode of operation. Alternatively or additionally, the first mode of operation may correspond to less power consumption by the device SoC as compared to power consumption of the device SoC when operating in the second mode of operation.

Method 700 may proceed from operation 708 and/or operation 710 to operation 712 . Act 712 may include causing the computing device to transmit broadcast response data and/or other response data. Other response data may be generated at least in part by device SoC when operation 712 is performed. For example, when the cached data does not identify a particular feature, such as a service related to mDNS broadcast data, the device SoC uses data accessible to the device SoC to determine one or more services related to mDNS broadcast data. It can be used to generate other response data that can be characterized. In some implementations, cached data is stored by the WiFi chip and/or device SoC when the WiFi chip is tasked with transmitting data that was otherwise inaccessible through the WiFi chip's memory. can be updated. In this way, subsequent queries or requests from other client devices over the network can be answered by the WiFi chip without waking up the device SoC, thereby saving power and computational resources. Eliminate waste.

FIG. 8 is a block diagram of an exemplary computer system 810. As shown in FIG. Computer system 810 generally includes at least one processor 814 that communicates with several peripheral devices via a bus subsystem 812 . These peripheral devices may include, for example, storage subsystem 824 including memory 825 and file storage subsystem 826 , user interface output device 820 , user interface input device 822 , and network interface subsystem 816 . Input and output devices allow user interaction with computer system 810 . Network interface subsystem 816 provides an interface to external networks and is coupled to corresponding interface devices of other computer systems.

User interface input devices 822 may include pointing devices such as keyboards, mice, trackballs, touch pads, or graphics tablets, scanners, touch screens integrated into displays, audio input devices such as voice recognition systems, microphones, and/or May include other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for entering information into computer system 810 or communication networks.

User interface output devices 820 may include non-visual displays such as display subsystems, printers, fax machines, or audio output devices. A display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide non-visual presentation, such as through an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from computer system 810 to a user or another machine or computer system.

Storage subsystem 824 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may perform selected aspects of the methods 500, 600, 700 and/or the first client device 134, second client device 102, third client device 112, implements one or more of the client device 202, client device 302, server device 402, computing device 418, and/or any other engine, module, chip, processor, application, etc. contemplated herein may contain logic to

These software modules are typically executed by processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 includes primary random access memory (RAM) 830 for storage of instructions and data during program execution, and read only memory (ROM) 832 where fixed instructions are stored. may contain several memories including The file storage subsystem 826 can provide persistent storage for program and data files and can be a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable drive. It may contain possible media cartridges. Modules implementing the functionality of a particular implementation may be stored by file storage subsystem 826 in storage subsystem 824 or on other machines that may be accessed by processor 814 .

Bus subsystem 812 provides a mechanism for allowing the various components and subsystems of computer system 810 to communicate with each other as intended. Although bus subsystem 812 is shown schematically as a single bus, alternate implementations of the bus subsystem may use multiple buses.

Computer system 810 can be of various types including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 810 shown in FIG. 8 is intended only as a specific example, intended to illustrate some implementations. Many other configurations of computer system 810 having more or fewer components than the computer system shown in FIG. 8 are possible.

In situations where the systems described herein may collect or utilize personal information about a user (or "participant" as often referred to herein), the user may , controls whether a program or feature collects user information (e.g., information about a user's social networks, social activities or activities, occupation, user preferences, or user's current geographic location); A user may be given the opportunity to control if and/or how potentially relevant content should be received from a content server. Also, certain data may be processed in one or more ways before the data is stored or used such that personally identifiable information is removed. For example, the user's identity cannot be determined about the user, or the user's geographic location (such as city, zip code, or state) when geolocation information is obtained. level) and thus may be treated such that the user's specific geographic location cannot be determined. Accordingly, users may be able to control how information is collected and/or used about them.

Although several implementations have been described and illustrated herein, various other implementations may be used to perform the functions and/or obtain one or more of the results and/or advantages described herein. means and/or structures may be utilized and each such change and/or modification is considered to be within the scope of the implementations described herein. More broadly, all parameters, dimensions, materials and configurations described herein are intended to be exemplary and actual parameters, dimensions, materials and/or configurations may vary depending on the teachings used. depending on the particular application or applications being used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. Thus, the above-described implementations are presented by way of example only, and within the scope of the appended claims and equivalents thereof, implementations may be practiced otherwise than as specifically described and claimed. It should be understood that there is Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein. Further, any combination of two or more such features, systems, articles, materials, kits, and/or methods shall not be construed as mutually exclusive of such features, systems, articles, materials, kits, and/or methods. otherwise it is within the scope of this disclosure.

In some implementations, the method is processing the output of the microphone in a first processor of the computing device, the output corresponding to speech provided to the microphone by the user, the computing device: It is described as including operations such as processing, including a second processor operating in sleep mode when an utterance is given by a user. The method may further include determining at the first processor whether the output corresponds, at least in part, to a call phrase for calling an automated assistant accessible via the computing device. The method includes transitioning, by the first processor, a second processor from a sleep mode to an operational mode when the first processor determines that the output corresponds at least partially to a call phrase; to a second processor by a first processor; determining by a second processor the degree to which the data characterizes the call phrase based on the data received from the first processor; determining by the second processor an amount of wake time during which the second processor remains in the operational mode based on the degree characterizing the and causing the second processor to operate in the active mode for at least the amount of wake time.

In some implementations, the method transfers additional data characterizing a distinct utterance from the user or another user to the first processor when the second processor is operating in the operational mode for at least an amount of wake time. It may further include receiving at a second processor from the processor and having the automated assistant respond to the separate utterances based on the additional data by the second processor. In some implementations, a first processor operates a first speech recognition model and a second processor operates a second speech recognition model different from the first speech recognition model. In some implementations, a first speech recognition model is associated with a first accuracy threshold for determining another degree to which the data characterizes the call phrase, and a second speech recognition model is associated with the data characterizing the call phrase. Associated with a second accuracy threshold, different from the first accuracy threshold, for determining the degree of characterization of the phrase. In some implementations, the second accuracy threshold is met by a greater degree of correlation between the spoken input and the calling phrase, and the greater degree of correlation is The degree of correlation to satisfy a threshold is used as a criterion.

In some implementations, the first processor is a digital signal processor (DSP), the second processor is a device system-on-chip (SoC), and the computing device is includes one or more batteries that provide power to the first processor and the second processor. In some implementations, determining the amount of wake time during which the second processor remains in operational mode includes specifying a previously determined amount of wake time designated for the second processor. Including, the previously determined amount of wake time is based on one or more interactions between the user and the automated assistant before the user gives the speech. In some implementations, the method includes, by the first processor, transitioning the second processor from the sleep mode to the operational mode when the first processor determines that the output does not at least partially correspond to the calling phrase. It can further include avoiding.

In some implementations, the method converts features of the user's voice characterized by the output of the microphone into data characterizing the output of the microphone when the first processor determines that the output corresponds at least partially to the calling phrase. Determining an amount of wake time during which the second processor remains in the operating mode may further include determining by the second processor based on the user's voice characterized by the output of the microphone. further based on the characteristics of

In other implementations, the method is processing the output of the microphone in a first processor of the computing device, the output corresponding to speech provided to the microphone by the user, the computing device processing the speech is described as including operations such as processing, including a second processor operating in sleep mode when a is provided by a user. In some implementations, the method further comprises determining at the first processor whether the output corresponds, at least in part, to a call phrase for calling an automated assistant that can be accessed via the computing device. can contain. In some implementations, the method comprises transitioning a second processor from a sleep mode to an operational mode by the first processor when the first processor determines that the output corresponds at least in part to the calling phrase; determining, by a second processor, a voice characteristic characterized by the output of the microphone; and determining, by a second processor, an amount of wake time during which the second processor remains in an operating mode based on the vocal characteristic characterized by the output. and operating the second processor in the operating mode for at least the amount of wake time based on determining the amount of wake time for the second processor. can contain.

In some implementations, the method is such that after the second processor operates through the operating mode, when the second processor is subsequently operating through the sleep mode, another output from the microphone invokes an automated assistant. determining in a first processor of the computing device that the other input corresponds at least partially to the calling phrase of transitioning a second processor from a sleep mode to an operational mode by the first processor; determining another voice characteristic characterized by another output from the microphone to a second voice characteristic based on the another output; determining by the processor and determining by the second processor another amount of wake time during which the second processor remains in the operating mode based on voice characteristics characterized by the different output; , the different amount of wake time is different than the amount of wake time, and based on determining the amount of wake time for the second processor, for at least the different amount of wake time; operating the second processor according to the operating mode.

In some implementations, the second processor operates a voice feature model when determining whether speech was presented to the microphone by the user and/or a separate user. In some implementations, the computing device includes one or more batteries that provide power to the first processor and the second processor when the second processor is operating according to the operating mode. In some implementations, the amount of wake time is based on one or more interactions between the user and the automated assistant before the user gives a speech.

In still other implementations, the method is such that the input to the computing device's microphone corresponds, at least in part, to a calling phrase for calling an automated assistant that can be accessed via the computing device's processor. It is described as including operations such as determining by. The method may further include causing the processor to transition another processor of the computing device from a sleep mode to an operational mode based on the input to the microphone. The method transfers first data characterizing audio output provided by the computing device via one or more speakers in communication with the computing device to another processor after the other processor transitions from sleep mode to operational mode. generating by a processor, wherein the first data includes first time data characterizing a time when another processor generated the first data; another input is a microphone of the computing device and generating, by the processor, second data characterizing another input to the microphone of the computing device, the second data being the second data that the processor generates the second data generating, including second time data characterizing another time at which the other processor generated the first data and a time offset between the time at which the other processor generated the first data and another time at which the processor generated the second data; Determining by another processor, processing the second data using the time offset by another processor to advance deletion of one or more features of the audio output provided by one or more speakers. Based on processing the second data using the time offset, whether another input to the microphone corresponds to speech to invoke an automated assistant that may be accessed via the computing device. and determining by another processor. The method provides a response output by another processor to the automated assistant through an interface in communication with the computing device when the other input to the microphone is determined to correspond to an utterance to invoke the automated assistant. may further include causing.

In some implementations, processing the second data using the time offsets to advance the removal of one or more features of the audio output is an acoustic echo using the second data and the audio data. Including performing an acoustic echo cancellation (AEC) process. In some implementations, the time offset corresponds to a difference in clock behavior characteristics of a processor's clock and another clock of another processor. In some implementations, the time offset is based on the difference between a first clock value determined using a clock and a second clock value determined using another clock. In some implementations, the first clock value and the second clock value are determined when the separate processor is in operational mode. In some implementations, the time offset is determined by multiplying the ratio of the difference between the clock values by the time value corresponding to another time. In some implementations, a computing device includes one or more batteries that provide power to a processor and another processor when the other processor is operating according to the operating mode. In some implementations, the processor is a digital signal processor (DSP) and another processor is a device system-on-chip (SoC).

In yet another implementation, the method includes receiving multicast Domain Name System (mDNS) broadcast data from the broadcasting device at a WiFi chip of a computing device, the computing device including the device system-on-chip (SoC) operating in the first mode of operation when the WiFi chip of the mobile device receives the mDNS broadcasted data. . The method comprises determining by the WiFi chip based on the mDNS broadcast data whether the target port identified by the mDNS broadcast data corresponds to a particular port that can be accessed via the computing device. It can contain more. The method uses WiFi when the device SoC is operating in a first mode of operation when the target port identified by the mDNS broadcast data corresponds to a specific port that can be accessed via the computing device. Accessing cached broadcast device data stored in a chip-accessible memory device based on a target port corresponding to a particular port, and specifying the cached broadcast device data by mDNS broadcast data. determining based on the cached broadcast device data stored in memory whether the cached broadcast device data characterizes one or more characteristics of the broadcast device; Characterizing the feature may further include generating response data based on the cached broadcast device data and transmitting the response data to the broadcast device.

In some implementations, the method includes when the target port identified by the mDNS broadcast data corresponds to a specific port that can be accessed via the computing device, and when the cached broadcast device data is the broadcast device's transitioning the device SoC from the first mode of operation to the second mode of operation based on the cached broadcast device data not characterizing the one or more characteristics when the one or more characteristics are not characterized wherein the second mode of operation is associated with higher power consumption by the device SoC as compared to power consumption of the device SoC when operating in the first mode of operation; .

In some implementations, the computing device includes one or more batteries that provide power to the WiFi chip and the device SoC when the device SoC is operating according to the second mode of operation. In some implementations, determining whether the cached broadcast device data characterizes the one or more characteristics of the broadcast device is determined by determining whether the cached broadcast device data is mDNS broadcast data from the broadcast device. Including determining whether to identify the application that initiated the transmission. In some implementations, determining whether the cached broadcast device data characterizes one or more characteristics of the broadcast device identifies the service for which the cached broadcast device data is requested by the broadcast device. including determining whether In some implementations, the method includes when the target port identified by the mDNS broadcast data corresponds to a specific port that can be accessed via the computing device, and when the cached broadcast device data is the broadcast device's having the device SoC generate other response data based on the mDNS broadcast data when not characterizing the one or more characteristics; and transmitting the other response data to the broadcasting device by the WiFi chip. It can contain more.

100 figures
102 second client device
106 WiFi chip
108 device SoCs
110 portable power
112 Third client device
116 WiFi Chip
118 device SoCs
122 stand-alone speaker device
124 users
126 cellular phone
130 mDNS data
132 mDNS data
134 first client device
136 Automated Assistant
138 Assistant Interface
140 memory
142 memory
144 applications
146 response data
148 response data
150 figure
200 figures
202 client devices
204 Digital Signal Processor DSP
208 device SoCs
210 power supply
212 audio data
214 first model
216 second model
218 Utterance
220 users
222 Computing Devices
230 figure
232 audio data
234 Utterance
240 figure
242 audio data
244 Utterance
300 figures
302 client device
306 WiFi chip
308 device SoCs
312 audio data
318 Utterance
320 users
322 stand-alone speaker device
324 wake time
400 system
402 Server Device
404 Automated Assistant
406 Input Processing Engine
408 Audio Processing Module
410 Data Analysis Module
412 Parameter Module
416 Assistant Data
418 Computing Devices
420 assistant interface
422 Client Automated Assistant
424 hour offset engine
426 Power Engine
430 audio processing engine
432 Client Data
434 WiFi chips and other computing devices
436 memory
438 Broadcast Engine
440 network, speech recognition model
444 device SoCs
446 power supply
448 Wake Time Engine
500 ways
600 ways
810 computer system
812 Bus Subsystem
814 processor
816 network interface subsystem
820 User Interface Output Device
822 user interface input device
824 storage subsystem
825 memory
826 file storage subsystem
830 main random access memory (RAM)
832 Read Only Memory (ROM)

Claims (15)

receiving multicast Domain Name System (mDNS) broadcast data from a broadcasting device at a WiFi chip of a computing device from said broadcasting device;
said computing device comprises a device system-on-chip (SoC) operating in a first mode of operation when said WiFi chip of said computing device receives said mDNS broadcasted data;
determining by the WiFi chip based on the mDNS broadcast data whether a target port identified by the mDNS broadcast data corresponds to a specific port that can be accessed via the computing device; and,
when the target port identified by the mDNS broadcast data corresponds to the specific port that can be accessed via the computing device;
cached broadcast device data stored in a memory device accessible to the WiFi chip when the device SoC is operating in the first mode of operation; accessing based on
determining whether the cached broadcast device data characterizes one or more characteristics of the broadcast device specified by the mDNS broadcast data based on the cached broadcast device data stored in the memory device; a step of determining
when the cached broadcast device data characterizes one or more characteristics of the broadcast device;
generating response data based on the cached broadcast device data;
and sending said response data to said broadcast device. when the target port specified by the mDNS broadcast data corresponds to the specified port that can be accessed via the computing device, and the cached broadcast device data is one of the broadcast devices or When not characterizing multiple features,
transitioning the device SoC from the first mode of operation to a second mode of operation based on the cached broadcast device data not characterizing the one or more characteristics; is associated with higher power consumption by the device SoC as compared to power consumption of the device SoC when operating in the first mode of operation. Method. 3. The computing device of claim 2, wherein the computing device includes one or more batteries that provide power to the WiFi chip and the device SoC when the device SoC is operating according to the second mode of operation. Method. determining whether the cached broadcast device data characterizes the one or more characteristics of the broadcast device;
2. The method of claim 1, comprising determining whether the cached broadcast device data identifies an application that initiated transmission of the mDNS broadcast data from the broadcast device. determining whether the cached broadcast device data characterizes the one or more characteristics of the broadcast device;
5. The method of claim 4, comprising determining whether the cached broadcast device data identifies a service requested by the broadcast device. determining whether the cached broadcast device data characterizes the one or more characteristics of the broadcast device;
2. The method of claim 1, comprising determining whether the cached broadcast device data identifies a service requested by the broadcast device. when the target port specified by the mDNS broadcast data corresponds to the specified port that can be accessed via the computing device, and the cached broadcast device data is one of the broadcast devices or When not characterizing multiple features,
causing the device SoC to generate other response data based on the mDNS broadcast data;
and sending the other response data to the broadcast device by the WiFi chip. A portable computing device,
one or more speakers;
one or more batteries;
a device system-on-chip (SoC) at least selectively powered by the one or more batteries;
a WiFi chip at least selectively powered by said one or more batteries;
at least selectively executing WiFi chip instructions stored by the WiFi chip,
receiving multicast Domain Name System (mDNS) broadcast data from a broadcasting device over a local network, comprising:
the device SoC is operating in a first mode of operation when the WiFi chip receives the mDNS broadcast data;
determining based on the mDNS broadcast data whether a target port identified by the mDNS broadcast data corresponds to a particular port that may be accessed via the portable computing device;
when the target port identified by the mDNS broadcast data corresponds to the specific port that can be accessed via the portable computing device;
cached broadcast device data stored in a memory device accessible to the WiFi chip when the device SoC is operating in the first mode of operation; accessing based on
determining whether the cached broadcast device data characterizes one or more characteristics of the broadcast device specified by the mDNS broadcast data based on the cached broadcast device data stored in the memory device; a step of determining
when the cached broadcast device data characterizes one or more characteristics of the broadcast device;
generating response data based on the cached broadcast device data;
A portable computing device that performs the step of transmitting the response data to the broadcast device over the local network. When the WiFi chip executes the stored WiFi chip instructions,
when the target port specified by the mDNS broadcast data corresponds to the specified port that can be accessed via the portable computing device, and the cached broadcast device data is one of the broadcast devices; or when not characterizing multiple features,
transitioning the device SoC from the first mode of operation to a second mode of operation based on the cached broadcast device data not characterizing the one or more characteristics; is associated with higher power consumption by the device SoC as compared to power consumption of the device SoC when operating in the first mode of operation. portable computing device. 10. The portable computing device of claim 9, wherein the one or more batteries power the device SoC only when the device SoC is operating in the second mode of operation. In determining whether the cached broadcast device data characterizes the one or more characteristics of the broadcast device, the WiFi chip:
9. The portable computing device of claim 8, determining whether the cached broadcast device data identifies an application that initiated transmission of the mDNS broadcast data from the broadcast device. In determining whether the cached broadcast device data characterizes the one or more characteristics of the broadcast device, the WiFi chip:
12. The portable computing device of claim 11, determining whether the cached broadcast device data identifies a service requested by the broadcast device. In determining whether the cached broadcast device data characterizes the one or more characteristics of the broadcast device, the WiFi chip:
9. The portable computing device of claim 8, determining whether the cached broadcast device data identifies a service requested by the broadcast device. When the WiFi chip executes the stored WiFi chip instructions,
when the target port specified by the mDNS broadcast data corresponds to the specified port that can be accessed via the portable computing device, and the cached broadcast device data is one of the broadcast devices; or when not characterizing multiple features,
causing the device SoC to generate other response data based on the mDNS broadcast data;
9. The portable computing device of claim 8, further comprising transmitting the other response data to the broadcast device over the local network. When executed by a WiFi chip in a computing device, said WiFi chip:
receiving multicast Domain Name System (mDNS) broadcast data from a broadcasting device at a WiFi chip of a computing device from said broadcasting device;
said computing device comprises a device system-on-chip (SoC) operating in a first mode of operation when said WiFi chip of said computing device receives said mDNS broadcasted data;
determining by the WiFi chip based on the mDNS broadcast data whether a target port identified by the mDNS broadcast data corresponds to a specific port that can be accessed via the computing device; and,
when the target port identified by the mDNS broadcast data corresponds to the specific port that can be accessed via the computing device;
cached broadcast device data stored in a memory device accessible to the WiFi chip when the device SoC is operating in the first mode of operation; accessing based on
determining whether the cached broadcast device data characterizes one or more characteristics of the broadcast device specified by the mDNS broadcast data based on the cached broadcast device data stored in the memory device; a step of determining
when the cached broadcast device data characterizes one or more characteristics of the broadcast device;
generating response data based on the cached broadcast device data;
and transmitting the response data to the broadcast device.

Images