KR20200110788A - Devices, methods, and graphical user interfaces for system-wide behavior for 3D models - Google Patents

Abstract

A computer system having a display creation component, one or more input devices, and one or more cameras receives a request to display a virtual object within a first user interface area that includes a field of view of the one or more cameras. In response to the request, upon determining that the object placement criteria are not met, the representation of the virtual object is a first set of visual properties and a first orientation independent of which part of the physical environment is displayed within the field of view of one or more cameras. Is displayed. Upon determination that the object placement criteria are satisfied, the representation of the virtual object is displayed with a second set of visual attributes distinct from the first set of visual attributes and in a second orientation corresponding to the plane.

Description

Devices, methods, and graphical user interfaces for system-wide behavior for 3D models

The present invention relates generally to electronic devices that display virtual objects, including, but not limited to, electronic devices that display virtual objects in a variety of contexts.

The development of computer systems for augmented reality has increased considerably in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices, such as touch-sensitive surfaces, for computer systems and other electronic computing devices are used to interact with virtual/augmented reality environments. Exemplary touch-sensitive surfaces include touchpads, touch-sensitive remote controls and touch screen displays. Such surfaces are used to manipulate user interfaces on the display and objects within it. Exemplary user interface objects include digital images, video, text, icons, control elements such as buttons, and other graphics.

However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g. applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient and , Limited. For example, using a sequence of inputs to orient and position a virtual object within an augmented reality environment is tedious, creates a significant cognitive burden on the user, and impairs the experience in a virtual/augmented reality environment. In addition, these methods take longer than necessary and energy is wasted. This latter consideration is particularly important in battery-operated devices.

Accordingly, there is a need for computer systems having improved methods and interfaces for interacting with virtual objects. Such methods and interfaces, optionally, supplement or replace conventional methods for interacting with virtual objects. Such methods and interfaces reduce the number, size, and/or type of inputs from the user and create a more efficient human-machine interface. In the case of battery-operated devices, such methods and interfaces save power and increase the time between battery charges.

The deficiencies and other problems associated with interfaces for interacting with virtual objects (e.g., user interfaces for augmented reality (AR) and related non-AR interfaces) are reduced or eliminated by the disclosed computer systems. . In some embodiments, the computer system includes a desktop computer. In some embodiments, the computer system is portable (eg, a notebook computer, tablet computer, or handheld device). In some embodiments, the computer system includes a personal electronic device (eg, a wearable electronic device such as a watch). In some embodiments, the computer system has (and/or communicates with) a touchpad. In some embodiments, the computer system has (and/or communicates with the touch-sensitive display) a touch-sensitive display (also known as a “touch screen” or “touch screen display”). In some embodiments, a computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory to perform a number of functions. In some embodiments, the user interacts with the GUI through gestures and/or stylus and/or finger contacts on the touch-sensitive surface in part. In some embodiments, functions optionally include playing games, image editing, drawing, presenting, word processing, spreadsheet creation, making phone calls, video conferencing, sending email, instant messaging, athletic support, Digital photography, digital video recording, web browsing, digital music playback, note taking, and/or digital video playback. Executable instructions for performing these functions are, optionally, included in a non-transitory computer readable storage medium or other computer program product configured for execution by one or more processors.

According to some embodiments, the method is performed in a computer system having a display, a touch-sensitive surface, and one or more cameras. The method includes displaying a representation of the virtual object within a first user interface area on the display. The method further includes detecting a first input by contact at a location on the touch-sensitive surface corresponding to the representation of the virtual object on the display, while displaying the first representation of the virtual object in the first user interface area on the display. Includes. The method further includes, in response to detecting the first input by contact, the display of at least a portion of the first user interface area of the one or more cameras, according to a determination that the first input by contact meets the first criteria. Displaying a second user interface area on the display, including replacing with a representation of the field of view; And continuously displaying a representation of the virtual object while switching from displaying the first user interface area to displaying the second user interface area.

According to some embodiments, the method is performed in a computer system having a display, a touch-sensitive surface, and one or more cameras. The method includes displaying a first representation of the virtual object within a first user interface area on the display. The method further comprises, while displaying the first representation of the virtual object in a first user interface area on the display, a first input by a first contact at a location on the touch-sensitive surface corresponding to the first representation of the virtual object on the display. And detecting. The method further comprises, in response to detecting a first input by the first contact and in accordance with a determination that the input by the first contact meets the first criteria, within a second user interface area different from the first user interface area. And displaying the representation of the virtual object. The method further includes detecting a second input while displaying a second representation of the virtual object within a second user interface area; And in response to detecting the second input, in response to a determination that the second input corresponds to a request to manipulate the virtual object in the second user interface area, based on the second input, a second input of the virtual object in the second user interface area. 2 change the display properties of the expression; In accordance with the determination that the second input corresponds to a request to display the virtual object in the augmented reality environment, displaying a third representation of the virtual object with a representation of the field of view of the one or more cameras.

According to some embodiments, a method is performed in a computer system having a display and a touch-sensitive surface. The method includes in response to a request to display the first user interface, displaying a first user interface with a representation of the first item. The method also displays a representation of the first item with a visual indication indicating that the first item corresponds to each of the first virtual three-dimensional objects, according to the determination that the first item corresponds to each virtual three-dimensional object. It includes the step of. The method also includes displaying a representation of the first item without a visual indication according to the determination that the first item does not correspond to each virtual three-dimensional object. The method also includes, after displaying the representation of the first item, receiving a request to display a second user interface comprising the second item. The method also includes, in response to a request to display the second user interface, displaying the second user interface with a representation of the second item. The method also displays a representation of the second item with a visual indication indicating that the second item corresponds to each second virtual three-dimensional object according to the determination that the second item corresponds to each virtual three-dimensional object. It includes the step of. The method also includes displaying a representation of the second item without a visual indication according to the determination that the second item does not correspond to each virtual three-dimensional object.

According to some embodiments, a method is performed in a computer system having a display generating component, one or more input devices, and one or more cameras. The method includes receiving a request to display a virtual object within a first user interface area that includes at least a portion of the field of view of one or more cameras. The method further comprises, in response to a request to display the virtual object in the first user interface area, displaying a representation of the virtual object over at least a portion of the field of view of one or more cameras included in the first user interface area through a display generating component. And the field of view of the one or more cameras is a view of the physical environment in which the one or more cameras are located. Displaying the representation of the virtual object, depending on the determination that the object placement criteria are not met-the object placement criteria require that the placement position for the virtual object be identified within the field of view of one or more cameras in order for the object placement criteria to be met. -Displaying a representation of the virtual object with a first set of visual properties and in a first orientation independent of which portion of the physical environment is displayed within the field of view of the one or more cameras; And in a second set of visual attributes distinct from the first set of visual attributes and in a second orientation corresponding to a plane within the physical environment detected within the field of view of the one or more cameras, upon determination that the object placement criteria are met. And displaying the representation of the virtual object.

According to some embodiments, a method is performed in a computer system having a display generating component, one or more input devices, one or more cameras, and one or more attitude sensors for detecting changes in the attitude of a device including one or more cameras. do. The method includes receiving a request to display an augmented reality view of the physical environment within a first user interface area that includes a representation of the field of view of one or more cameras. The method also displays, in response to receiving a request to display an augmented reality view of the physical environment, a representation of the field of view of one or more cameras, and upon determining that calibration criteria are not met for an augmented reality view of the physical environment , Displaying a calibration user interface object that is dynamically animated according to movement of one or more cameras in a physical environment, and displaying the calibration user interface object includes, while displaying the calibration user interface object, one or more postures. Detecting, through the sensors, a change in attitude of one or more cameras in the physical environment; And in response to detecting a change in the attitude of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object according to the detected change in the attitude of the one or more cameras in the physical environment. do. The method also includes detecting that calibration criteria are met while displaying a calibration user interface object that moves on the display in accordance with the detected change in the attitude of one or more cameras in the physical environment. The method also includes responsive to detecting that calibration criteria are met, stopping displaying the calibration user interface object.

According to some embodiments, a method is performed in a computer system having a display generating component and one or more input devices including a touch-sensitive surface. The method includes displaying, by a display generating component, a representation of a first perspective of a virtual three-dimensional object within a first user interface area. The method further provides for displaying a portion of the virtual 3D object that is not visible from the first view of the virtual 3D object while displaying the representation of the first viewpoint of the virtual 3D object in the first user interface area on the display. And detecting a first input corresponding to a request to rotate the virtual three-dimensional object with respect to the display. The method further comprises, in response to detecting the first input, determined based on the magnitude of the first input, according to a determination that the first input corresponds to a request to rotate the three-dimensional object about the first axis, and Rotating the virtual three-dimensional object about the first axis by an amount constrained by the limit on movement that limits the rotation of the virtual three-dimensional object by rotation above the threshold amount about the first axis; According to the determination that the first input corresponds to a request to rotate the 3D object about a second axis different from the first axis, the virtual 3D object is transferred to the second axis by an amount determined based on the size of the first input. And, for inputs having a magnitude greater than each threshold, the device rotates the virtual three-dimensional object about the second axis by a rotation greater than the threshold amount.

In accordance with some embodiments, a method is performed in a computer system having a display generating component and a touch-sensitive surface. The method includes a first object manipulation behavior performed in response to inputs meeting the first gesture recognition criteria and a second object manipulation behavior performed in response to inputs meeting the second gesture recognition criteria through a display generating component. And displaying a first user interface area including a user interface object associated with a plurality of object manipulation behaviors including. The method further includes detecting movement of one or more contacts across the touch-sensitive surface while displaying the first user interface area, detecting a first portion of the input to the user interface object, and detecting the one or more While the contacts are detected on the touch-sensitive surface, evaluating movement of the one or more contacts relative to both the first gesture recognition criteria and the second gesture recognition criteria. The method also includes, in response to detecting the first portion of the input, updating the appearance of the user interface object based on the first portion of the input, wherein the updating the appearance of the user interface object comprises: Change the appearance of the user interface object according to the first object manipulation behavior based on the first portion of the input, according to a determination that the first portion of the first portion satisfies the first gesture recognition criteria before meeting the second gesture recognition criteria; Updating second gesture recognition criteria by increasing a threshold for the second gesture recognition criteria; And according to a determination that the input meets the second gesture recognition criteria before meeting the first gesture recognition criteria, change the appearance of the user interface object according to the second object manipulation behavior based on the first portion of the input; Updating the first gesture recognition criteria by increasing the threshold for the first gesture recognition criteria.

According to some embodiments, a method is performed in a computer system having a display generating component, one or more input devices, one or more audio output generators, and one or more cameras. The method includes, via a display generation component, displaying a representation of the virtual object within a first user interface area comprising a representation of the field of view of one or more cameras, wherein the displaying step is captured within the field of view of one or more cameras. And maintaining a first spatial relationship between the plane detected in the physical environment and the representation of the virtual object. The method also includes detecting movement of the device that adjusts the field of view of one or more cameras. The method also includes, in response to detecting movement of the device that adjusts the field of view of the one or more cameras, a first spatial between the virtual object and the plane detected within the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted. According to the determination that the display of the representation of the virtual object in the first user interface area is adjusted according to the relationship, and the movement of the device causes the virtual object exceeding the threshold amount to move outside the displayed portion of the field of view of one or more cameras, one And generating a first audio alert through the above audio output generators.

According to some embodiments, the electronic device comprises a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally an intensity of contacts with a touch-sensitive surface. One or more sensors, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, optionally one or more attitude sensors for detecting changes in attitude A memory storing one or more processors, one or more programs; The one or more programs are configured to be executed by one or more processors, and the one or more programs include instructions for performing the operations of or causing the performance of any of the methods described herein. According to some embodiments, the computer-readable storage medium comprises a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally a touch-sensitive surface, and Having one or more sensors for detecting the intensities of the contacts of, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more attitude sensors. Stored instructions that, when executed by the electronic device, cause the device to perform the operations of any of the methods described herein or cause the performance of those operations. According to some embodiments, detecting intensities of contacts with a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally a touch-sensitive surface. One or more sensors, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, optionally one or more attitude sensors, a memory, and one or more stored in the memory A graphical user interface on an electronic device having one or more processors for executing programs is updated in response to inputs, as described in any of the methods described herein. It includes one or more of the elements displayed in any way. According to some embodiments, the electronic device comprises: a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally of contacts with a touch-sensitive surface. One or more sensors for detecting intensities, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more for detecting changes in attitude Attitude sensors; And means for performing or causing the performance of any of the methods described herein. According to some embodiments, detecting intensities of contacts with a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally a touch-sensitive surface. Having one or more sensors for, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more attitude sensors for detecting changes in attitude. An information processing apparatus for use in an electronic device includes means for performing the operations of any of the methods described herein or for causing the performance of such operations.

Accordingly, one or more sensors for detecting the intensities of contacts with display generating components, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally a touch-sensitive surface. Electronic devices having one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more attitude sensors, may have virtual objects in various contexts. Improved methods and interfaces for display are provided, thereby increasing the effectiveness, efficiency and user satisfaction with such devices. Such methods and interfaces may complement or replace conventional methods for displaying virtual objects in a variety of contexts.

For a better understanding of the various described embodiments, reference should be made to the specific details for carrying out the following invention in connection with the following drawings in which like reference numerals indicate corresponding parts throughout the drawings.
1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display, in accordance with some embodiments.
1B is a block diagram illustrating example components for event processing, in accordance with some embodiments.
1C is a block diagram illustrating a tactile output module according to some embodiments.
2 illustrates a portable multifunction device with a touch screen, in accordance with some embodiments.
3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, in accordance with some embodiments.
4A illustrates an exemplary user interface for a menu of applications on a portable multifunction device, in accordance with some embodiments.
4B illustrates an exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display, in accordance with some embodiments.
4C-4E illustrate examples of dynamic intensity thresholds in accordance with some embodiments.
4F-4K illustrate a set of sample tactile output patterns in accordance with some embodiments.
5A-5A illustrate exemplary user interfaces for displaying a representation of a virtual object while switching from displaying a first user interface area to displaying a second user interface area, according to some embodiments. .
6A-6AJ illustrate a first representation of a virtual object in a first user interface area, a second representation of a virtual object in a second user interface area, and a representation of a field of view of one or more cameras, according to some embodiments. Together illustrate example user interfaces for displaying a third representation of a virtual object.
7A to 7E, 7F1, 7F2, 7G1, 7G2, and 7H to 7P display items with visual indications indicating that the item corresponds to a virtual three-dimensional object, according to some embodiments. Illustrates exemplary user interfaces for doing so.
8A-8E are flowcharts of a process for displaying a representation of a virtual object while switching from displaying a first user interface area to displaying a second user interface area, in accordance with some embodiments.
9A-9D illustrate a first representation of a virtual object in a first user interface area, a second representation of a virtual object in a second user interface area, and a representation of a field of view of one or more cameras, according to some embodiments. Together are flow charts of a process for displaying a third representation of a virtual object.
10A-10D are flow diagrams of a process for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object, in accordance with some embodiments.
11A-11V illustrate example user interfaces for displaying a virtual object having different visual properties depending on whether object placement criteria are met, according to some embodiments.
12a to 12d, 12e-1, 12e-2, 12f-1, 12f-2, 12g-1, 12g-2, 12h-1, 12h-2, 12i-1 12I-2, 12J, 12K-1, 12K-2, 12L-1, and 12L-2 are dynamically animated according to movement of one or more cameras of the device, according to some embodiments. Illustrates exemplary user interfaces for displaying a calibration user interface object.
13A-13M illustrate exemplary user interfaces for restricting rotation of a virtual object about an axis, in accordance with some embodiments.
14A-14Z for increasing the second threshold size of movement required for the second object manipulation behavior, according to a determination that the first threshold size of movement is satisfied for the first object manipulation behavior, according to some embodiments. Illustrates exemplary user interfaces.
14AA to 14A are for increasing the second threshold size of movement required for the second object manipulation behavior according to the determination that the first threshold size of movement is satisfied for the first object manipulation behavior, according to some embodiments. Illustrates flowcharts illustrating operations.
15A-15AI illustrate exemplary user interfaces for generating an audio alert upon determining that movement of the device causes the virtual object to move outside the displayed field of view of one or more device cameras, according to some embodiments. do.
16A-16G are flow charts of a process for displaying a virtual object having different visual properties depending on whether object placement criteria are met, according to some embodiments.
17A-17D are flow diagrams of a process for displaying a calibration user interface object that is dynamically animated according to movement of one or more cameras of the device, according to some embodiments.
18A-18I are flow diagrams of a process for constraining rotation of a virtual object about an axis, in accordance with some embodiments.
19A-19H are for increasing the second threshold size of movement required for the second object manipulation behavior, according to a determination that the first threshold size of movement is satisfied for the first object manipulation behavior, according to some embodiments. These are the flow charts of the process.
20A-20F are flow diagrams of a process for generating an audio alert upon determining that movement of the device causes the virtual object to move outside of the displayed field of view of one or more device cameras, in accordance with some embodiments.

Virtual objects are graphical representations of three-dimensional objects in a virtual environment. Additions not available in the augmented reality environment (e.g., the physical world) from interacting with the virtual objects to display the virtual objects in the context of an application user interface (e.g., a two-dimensional application user interface that does not display an augmented reality environment). Conventional methods of transitioning to being displayed in the context of the environment in which the view of the physical world is augmented with supplementary information providing information to the user (e.g., the size of a virtual object for an actual or desired appearance in an augmented reality environment) It often requires a number of distinct inputs (eg, a sequence of gestures and button presses, etc.) to achieve the intended result (adjusting the location, and/or orientation). Additionally, conventional input methods often relate to virtual objects that may be placed in an augmented reality environment and/or the time required to activate one or more device cameras to capture a view of the physical world (e.g. The delay between receiving a request to display an augmented reality environment and displaying the augmented reality environment is involved due to the time required to analyze and characterize the view of the physical world (detecting planes and/or surfaces in the rendered view) do. Embodiments of the present specification (e.g., display a virtual object in an augmented reality environment by allowing the user to provide an input to switch from displaying the virtual object in the context of the application user interface to displaying the virtual object in an augmented reality environment). By allowing the user to change the display properties of the virtual object (e.g., in a three-dimensional staging environment) before doing so, by providing an indication that allows the user to easily identify virtual objects throughout the system across multiple applications. , Rotation of the displayed virtual object around an axis by providing an animated calibration user interface object to indicate the movement of the device required for calibration by changing the visual properties of the object while determining the placement information for the object. By constraining, by increasing the threshold size of movement for the second object manipulation behavior when the threshold size of movement is satisfied for the first object manipulation behavior, and to indicate that the virtual object has moved outside the displayed field of view By providing audio alerts), it provides an intuitive way for users to display and/or interact with virtual objects in various contexts.

The systems, methods, and GUIs described herein improve user interface interactions with virtual/augmented reality environments in a number of ways. For example, they make it easier to display the virtual object in an augmented reality environment and, in response to different inputs, adjust the appearance of the virtual object for display in the augmented reality environment.

In the following, FIGS. 1A-1C, 2, and 3 provide a description of exemplary devices. 4a, 4b, 5a to 5at, 6a to 6aj, 7a to 7p, 11a to 11v, 12a to 12l, 13a to 13m, 14a to 14z, and 15A-15Ai illustrate example user interfaces for displaying virtual objects in various contexts. 8A-8E illustrate a process for displaying a representation of a virtual object while switching from displaying a first user interface area to displaying a second user interface area. 9A to 9D are a first representation of a virtual object in a first user interface area, a second representation of a virtual object in a second user interface area, and a third representation of a virtual object together with a representation of the field of view of one or more cameras Illustrates the process for displaying. 10A-10D illustrate a process for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object. 16A-16G illustrate a process for displaying a virtual object having different visual properties depending on whether the object placement criteria are met. 17A-17D illustrate a process for displaying a calibration user interface object that is dynamically animated according to movement of one or more cameras of the device. 18A-18I illustrate a process for constraining rotation of a virtual object about an axis. 14A to 14A and 19A to 19H show a process for increasing the second threshold size of movement required for the second object manipulation behavior, according to the determination that the first threshold size of movement is satisfied for the first object manipulation behavior. Illustrate. 20A-20F illustrate a process for generating an audio alert upon determining that movement of the device causes the virtual object to move outside of the displayed field of view of one or more device cameras. 5A to 5A, 6A to 6A, 7A to 7P, 11A to 11V, 12A to 12L, 13A to 13M, 14A to 14Z, and 15A to 15AI Interfaces are in FIGS. 8A to 8E, 9A to 9D, 10A to 10D, 14A to 14A, 16A to 16G, 17A to 17D, 18A to 18I, 19A to 19H, And the processes of FIGS. 20A-20F.

Example devices

Now, the embodiments will be referred to in detail, and examples of the embodiments are illustrated in the accompanying drawings. In the detailed description that follows, many specific details are set forth to provide a thorough understanding of the various described embodiments. However, it will be apparent to those skilled in the art that various described embodiments may be practiced without these specific details. In other examples, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

In some examples, the terms first, second, etc. are used herein to describe various elements, but it will also be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but unless the context clearly indicates otherwise, they are not the same contact.

The terms used in the description of the various embodiments described herein are for the purpose of describing specific embodiments only, and are not intended to be limiting. As used in the description of the various described embodiments and in the appended claims, the singular forms ("a", "an", and "the") are intended to include the plural as well, unless the context clearly indicates otherwise. Is intended. Further, it will be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated items listed. The terms “include”, “including”, “comprise”, and/or “comprising”, as used herein, refer to the recited features, integers Specifies the presence of elements, steps, actions, elements, and/or components, but the presence of one or more other features, integers, steps, actions, elements, components, and/or groups thereof Or it will be further understood that it does not exclude addition.

As used herein, the term "if" is, optionally, in response to "when" or "upon" or "determining, depending on the context ( in response to determining)" or "in response to detecting". Similarly, the phrases "when it is determined to" or "when [the condition or event mentioned] is detected" is, optionally, "when determining ..." or "in response to determining ... It is interpreted to mean "on detecting [the mentioned condition or event]" or "in response to detecting [the mentioned condition or event].

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and/or music player functions. Exemplary embodiments of portable multifunction devices limit iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, CA, USA Include without. Other portable electronic devices such as laptop or tablet computers with touch-sensitive surfaces (eg, touch screen displays and/or touchpads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device but a desktop computer having a touch-sensitive surface (eg, a touch screen display and/or a touchpad).

In the discussion that follows, an electronic device is described that includes a display and a touch-sensitive surface. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices such as a physical keyboard, mouse and/or joystick.

Devices typically support a variety of applications such as one or more of the following: note-taking applications, drawing applications, presentation applications, word processing applications, website creation applications, disc authoring applications, spreadsheet applications, game applications, phone applications, and video Conferencing applications, email applications, instant messaging applications, athletic support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and/or digital video player applications.

Various applications running on the device, optionally, use at least one universal physical user interface device, such as a touch-sensitive surface. The one or more functions of the touch-sensitive surface as well as the corresponding information displayed on the device are, optionally, adjusted and/or changed from one application to the next and/or within each application. In this way, the universal physical architecture of the device (such as a touch-sensitive surface) supports a variety of applications, optionally using user interfaces that are intuitive and transparent to the user.

Attention is now directed to embodiments of portable devices with touch-sensitive displays. 1A is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 in accordance with some embodiments. The touch-sensitive display system 112 is sometimes referred to as a “touch screen” for convenience, and sometimes simply as a touch-sensitive display. Device 100 includes memory 102 (optionally including one or more computer readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF Circuitry 108, audio circuitry 110, speaker 111, microphone 113, input/output (I/O) subsystem 106, other input or control devices 116, and external ports 124 ). Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors for detecting intensities of contacts on device 100 (e.g., a touch-sensitive surface such as touch-sensitive display system 112 of device 100). 165). The device 100 optionally includes a touch pad 355 of the device 300 or a touch-sensitive display system 112 of the device 100 for generating tactile outputs on the device 100. And one or more tactile output generators 167 (for generating tactile outputs on the same touch-sensitive surface). These components optionally communicate via one or more communication buses or signal lines 103.

Device 100 is only an example of a portable multifunction device, and device 100 is, optionally, having more or fewer components than shown, optionally, combining two or more components, or optionally component It should be understood that they have different configurations or arrangements. The various components shown in FIG. 1A are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing and/or application specific integrated circuits.

The memory 102 optionally includes a high speed random access memory, and optionally, one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices And non-volatile memory such as. Access to memory 102 by other components of device 100 such as CPU(s) 120 and peripherals interface 118 is optionally controlled by memory controller 122.

The peripherals interface 118 may be used to couple the input and output peripherals of the device to the CPU(s) 120 and memory 102. One or more processors 120 drive or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions for device 100 and to process data.

In some embodiments, peripherals interface 118, CPU(s) 120 and memory controller 122 are, optionally, implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

The radio frequency (RF) circuit unit 108 receives and transmits RF signals, also referred to as electromagnetic signals. The RF circuitry 108 converts electrical signals to/from electromagnetic signals, and communicates with communication networks and other communication devices via the electromagnetic signals. The RF circuit unit 108, optionally, includes, but is not limited to, an antenna system, an RF transceiver, one or more amplifiers, tuners, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, a memory, and the like. It includes well-known circuitry for performing these functions. The RF circuitry 108 is, optionally, networks such as the Internet, intranet, and/or wireless network, also referred to as the World Wide Web (WWW), such as a cellular telephone network, a wireless local area network (LAN) and/or a MAN (metropolitan area network), and communicates with other devices by wireless communication. Wireless communication, optionally, GSM (Global System for Mobile Communications), EDGE (Enhanced Data GSM Environment), HSDPA (high-speed downlink packet access), HSUPA (high-speed uplink packet access), EV-DO (Evolution, Data-Only), HSPA, HSPA+, DC-HSPA (Dual-Cell HSPA), LTE (long term evolution), NFC (near field communication), W-CDMA (wideband code division multiple access), CDMA (code division multiple access) ), TDMA (time division multiple access), Bluetooth, Wi-Fi (Wireless Fidelity) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and/or IEEE 802.11n ), voice over Internet Protocol (VoIP), Wi-MAX, protocol for email (e.g., Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP)) ), SIMPLE (Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions), IMPS (Instant Messaging and Presence Service)), and/or Short Message Service (SMS), or communication protocols not yet developed at the filing date of this document. , Including any other suitable communication protocol, but Use any of a number of communication standards, protocols and technologies, but not limited to these.

The audio circuit unit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit unit 110 receives audio data from the peripheral device interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into sound waves that can be heard by humans. The audio circuit unit 110 also receives an electrical signal converted from sound waves by the microphone 113. The audio circuit unit 110 converts the electrical signal into audio data and transmits the audio data to the peripheral device interface 118 for processing. Audio data is, optionally, fetched from and/or transmitted to memory 102 and/or RF circuitry 108 by peripherals interface 118 from memory 102 and/or RF circuitry 108. In some embodiments, the audio circuitry 110 also includes a headset jack (eg, 212 in FIG. 2 ). The headset jack is between the audio circuitry 110 and removable audio input/output peripherals such as output-only headphones, or a headset having both an output (eg, one or both ear headphones) and an input (eg, a microphone) Provides an interface.

I/O subsystem 106 couples input/output peripherals on device 100 with peripherals interface 118, such as touch-sensitive display system 112 and other input or control devices 116. I/O subsystem 106 optionally includes one or more input controllers for display controller 156, light sensor controller 158, intensity sensor controller 159, haptic feedback controller 161 and other input or control devices. 160). One or more input controllers 160 receive/transmit electrical signals from/to other input or control devices 116. Other input or control devices 116 optionally include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some other embodiments, the input controller(s) 160 is optionally coupled (or coupled to any of) a pointer device such as a keyboard, infrared port, USB port, stylus, and/or mouse. Not). One or more buttons (eg, 208 in FIG. 2) optionally include an up/down button for volume control of the speaker 111 and/or microphone 113. The one or more buttons optionally include a push button (eg, 206 in FIG. 2 ).

The touch-sensitive display system 112 provides an input interface and an output interface between a device and a user. The display controller 156 receives and/or transmits electrical signals from/to the touch-sensitive display system 112. The touch-sensitive display system 112 displays a visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as “graphics”). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term "affordance" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to inputs directed towards a graphical user interface object). . Examples of user-interactive graphical user interface objects include, without limitation, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accept input from a user based on haptic and/or tactile contact. Touch-sensitive display system 112 and display controller 156 (along with any associated modules and/or sets of instructions in memory 102) contact (and contact) on touch-sensitive display system 112 Detect any movement or interruption of the user interface objects (e.g., one or more soft keys, icons, web pages or images) displayed on the touch-sensitive display system 112 ) To interact with. In some embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

The touch sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light-emitting polymer display) technology, or LED (light-emitting diode) technology, but other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally include one or more of capacitive, resistive, infrared and surface acoustic wave technologies, as well as other proximity sensor arrays or touch-sensitive display system 112. Detect contact and any movement or interruption thereof using any of a plurality of touch sensing technologies, currently known or to be developed in the future, including, but not limited to, other factors for determining contact points. In some embodiments, a projected mutual capacitance sensing technology, such as those found on the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, CA, is used. .

The touch sensitive display system 112 optionally has a video resolution in excess of 100 dpi. In some embodiments, the touch screen video resolution exceeds 400dpi (eg, 500dpi, 800dpi, or higher). The user optionally contacts the touch-sensitive display system 112 using any suitable object or accessory such as a stylus, finger, or the like. In some embodiments, the user interface is designed to operate using finger-based contacts and gestures, which may be less precise than stylus-based input due to the larger contact area of the finger on the touch screen. In some embodiments, the device converts the coarse finger-based input into a precise pointer/cursor location or command to perform the actions desired by the user.

In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) to activate or deactivate certain functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike a touch screen, does not display a visual output. The touchpad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

Device 100 also includes a power system 162 for supplying power to various components. Power system 162, optionally, a power management system, one or more power sources (e.g., batteries, alternating current (AC)), recharge system, power failure detection circuit, power converter or inverter, power status indicator ( For example, light emitting diodes (LEDs)), and any other components associated with the generation, management and distribution of power within portable devices.

Device 100 also optionally includes one or more optical sensors 164. 1A shows a light sensor coupled with a light sensor controller 158 in I/O subsystem 106. The optical sensor(s) 164 optionally includes charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. The optical sensor(s) 164 receives light from the surrounding environment, projected through one or more lenses, and converts the light into data representing an image. In conjunction with the imaging module 143 (also referred to as a camera module), the optical sensor(s) 164 optionally captures still images and/or video. In some embodiments, the optical sensor is located on the back of the device opposite the touch-sensitive display system 112 on the front of the device 100, so that the touch screen is used as a viewfinder for still and/or video image acquisition. To be used. In some embodiments, another light sensor is used on the front of the device to obtain an image of the user (e.g., for a selfie, for the user to videoconference while watching other videoconference participants on a touch screen, etc.) Is located in

Device 100 also optionally includes one or more contact intensity sensors 165. 1A shows a contact intensity sensor coupled with intensity sensor controller 159 in I/O subsystem 106. Contact strength sensor(s) 165, optionally, one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (Eg, sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). The contact intensity sensor(s) 165 receives contact intensity information (eg, pressure information or a proxy for pressure information) from the surrounding environment. In some embodiments, at least one contact intensity sensor is collocated with or proximate to the touch-sensitive surface (eg, touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back of the device 100, opposite the touch screen display system 112 located on the front of the device 100.

Device 100 also optionally includes one or more proximity sensors 166. 1A shows proximity sensor 166 coupled with peripherals interface 118. Alternatively, proximity sensor 166 is coupled with input controller 160 in I/O subsystem 106. In some embodiments, the proximity sensor turns off and disables the touch-sensitive display system 112 when the multifunction device is positioned near the user's ear (eg, when the user is making a phone call).

Device 100 also optionally includes one or more tactile output generators 167. 1A shows a tactile output generator coupled with a haptic feedback controller 161 in an I/O subsystem 106. In some embodiments, the tactile output generator(s) 167 is one or more electroacoustic devices such as speakers or other audio components and/or a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or And electromechanical devices that convert energy into linear motion, such as other tactile output generating components (eg, a component that converts electrical signals into tactile outputs on a device). The tactile output generator(s) 167 receives tactile feedback generation instructions from the haptic feedback module 133 and generates tactile outputs on the device 100 that can be sensed by the user of the device 100 . In some embodiments, the at least one tactile output generator is located with or proximate the touch-sensitive surface (e.g., touch-sensitive display system 112), and optionally, vertically (e.g., , Generating a tactile output by moving in/out of the surface of the device 100) or laterally (eg, back and forth in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch-sensitive display system 112 located on the front of the device 100.

Device 100 also optionally includes one or more accelerometers 168. 1A shows the accelerometer 168 coupled with the peripherals interface 118. Alternatively, accelerometer 168 is optionally coupled with input controller 160 in I/O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. Device 100 optionally, in addition to the accelerometer(s) 168, a magnetometer (not shown), and a GPS ( Or a GLONASS or other global navigation system) receiver (not shown).

In some embodiments, software components stored in memory 102 include operating system 126, communication module (or set of instructions) 128, contact/motion module (or set of instructions) 130, graphics module ( Or a set of instructions) 132, a haptic feedback module (or set of instructions) 133, a text input module (or set of instructions) 134, a GPS module (or set of instructions) 135, and applications. (Or sets of instructions) 136. In addition, in some embodiments, memory 102 stores device/global internal state 157 as shown in FIGS. 1A and 3. The device/global internal state 157, if present, may include an active application state indicating which applications are currently active; A display state indicating which applications, views, or other information occupy various areas of the touch-sensitive display system 112; Sensor status, including information obtained from various sensors of the device and other input or control devices 116; And one or more of location and/or position information regarding the location and/or posture of the device.

Operating system 126 (e.g., an embedded operating system such as iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or VxWorks) includes general system tasks (e.g., memory management, storage device control, power management, etc.) It includes various software components and/or drivers for controlling and managing the system, and facilitates communication between various hardware and software components.

The communication module 128 enables communication with other devices through one or more external ports 124, and also provides various methods for processing data received by the RF circuitry 108 and/or external ports 124. Contains software components. The external port 124 (eg, USB, FIREWIRE, etc.) is configured to couple directly to other devices or indirectly through a network (eg, Internet, wireless LAN, etc.). In some embodiments, the external port is the same, similar, and/or compatible with the 30-pin connector used on some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, CA, USA. It is a possible multi-pin (eg 30-pin) connector. In some embodiments, the external port is the same or similar to the Lightning connector used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, CA, USA. Or a compatible Lightning connector.

The contact/motion module 130 optionally includes contact with the touch-sensitive display system 112 (along with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). Is detected. The contact/motion module 130 determines whether a contact has occurred (e.g., detecting a finger-down event), the strength of the contact (e.g., force or pressure of the contact, or Determining the force or pressure of the contact), determining whether there is movement of the contact to track movement across the touch-sensitive surface (e.g., one or more finger-dragging events ), and determining whether contact has stopped (e.g., detecting a finger-up event or contact interruption) (e.g., by a finger , Or by a stylus) various software components for performing various operations related to detection of a contact. The contact/motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point represented by a series of contact data, optionally, determines the speed (magnitude), velocity (magnitude and direction), and/or acceleration (change in magnitude and/or direction) of the contact point. Includes doing. These actions are optionally applied to single contacts (eg, single finger contacts or stylus contacts) or to multiple simultaneous contacts (eg “multi-touch”/multiple finger contacts). In some embodiments, the contact/motion module 130 and the display controller 156 detect contact on the touchpad.

The contact/motion module 130 optionally detects a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (eg, different motions, timings, and/or intensities of detected contacts). Thus, the gesture is, optionally, detected by detecting a specific contact pattern. For example, detecting a finger tap gesture can detect a finger-down event and then finger-down at the same location (or substantially the same location) as the finger-down event (e.g., at the location of the icon). It involves detecting an up (liftoff) event. As another example, detecting a finger swipe gesture on a touch-sensitive surface detects a finger-down event followed by one or more finger-drag events, followed by a finger-up (liftoff). ) Including detecting the event. Similarly, tap, swipe, drag, and other gestures are detected for the stylus, optionally by detecting a specific contact pattern for the stylus.

In some embodiments, detecting a finger tap gesture depends on the length of time between detecting a finger-down event and a finger-up event, but between detecting a finger-down event and a finger-up event. Is independent of the strength of the finger contact. In some embodiments, the tap gesture is, independently of whether the intensity of the finger contact during the tap satisfies a given intensity threshold (greater than the nominal contact-detection intensity threshold), such as a light tap or deep tap intensity threshold. It is detected according to a determination that the length of time between the down event and the finger-up event is less than a predetermined value (eg, less than 0.1, 0.2, 0.3, 0.4 or 0.5 seconds). Thus, the finger tap gesture may satisfy specific input criteria that do not require that the characteristic strength of the contact meet a given intensity threshold in order for the specific input criteria to be satisfied. For clarity, finger contact in a tap gesture typically needs to meet a nominal contact-detection intensity threshold (less than this no contact is detected) in order for a finger-down event to be detected. A similar analysis applies to detecting a tap gesture by a stylus or other contact. If the device is capable of detecting a finger or stylus contact hovering over the touch-sensitive surface, the nominal contact-detection intensity threshold does not optionally correspond to a physical contact between the finger or stylus and the touch-sensitive surface.

The same concepts apply in a similar way to other types of gestures. For example, a swipe gesture, a pinch gesture, a depinch gesture, and/or a long press gesture is optionally independent of the strengths of the contacts included in the gesture, or a contact that performs the gesture. The (s) are detected based on satisfaction of criteria that do not require that intensity thresholds be reached in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts; The pinch gesture is detected based on movement of the two or more contacts towards each other; The de-pinch gesture is detected based on movement of the two or more contacts away from each other; The long press gesture is detected based on the duration of the contact on the touch-sensitive surface with less than a threshold amount of movement. As such, the statement that certain gesture recognition criteria do not require that the strength of the contact(s) meet the respective strength threshold in order for certain gesture recognition criteria to be satisfied, is that the specific gesture recognition criteria ) May be satisfied if it does not reach the respective intensity threshold, meaning that it may also be satisfied in situations where one or more of the contacts in the gesture reaches or exceeds the respective intensity threshold. In some embodiments, the tap gesture is based on a determination that finger-down and finger-up events are detected within a predefined period, regardless of whether the contact is above or below their respective intensity threshold for a predefined period of time. And the swipe gesture is detected based on a determination that the contact movement is greater than a predefined size, even if the contact exceeds the respective intensity threshold at the end of the contact movement. Implementations in which the detection of a gesture is affected by the strength of the contacts performing the gesture (e.g., the device detects a long press more quickly when the strength of the contact exceeds the strength threshold, or the strength of the contact is more Even when high, it delays detection of tap input), the detection of such gestures, as long as the criteria for recognizing a gesture in situations where the contact does not reach a certain intensity threshold can be met (e.g. Even if the time it takes to do it changes), it does not require the contacts to reach a certain intensity threshold.

The contact intensity thresholds, duration thresholds, and movement thresholds are, in some situations, the same input element or so that multiple different interactions with the same input element can provide a richer set of user interactions and responses. It is combined in a variety of different combinations to create heuristics for distinguishing two or more different gestures towards the area. A statement that a particular set of gesture recognition criteria does not require that the strength of the contact(s) meet its respective strength threshold in order for certain gesture recognition criteria to be met, includes a contact whose gesture has an strength above its respective strength threshold. Does not preclude simultaneous evaluation of different intensity dependent gesture recognition criteria to identify different gestures with criteria that are met when doing so. For example, in some situations, first gesture recognition criteria for a first gesture-this does not require the strength of the contact(s) to meet the respective strength threshold in order for the first gesture recognition criteria to be met- Is in competition with the second gesture recognition criteria for the second gesture, which depends on the contact(s) reaching their respective intensity threshold. In such competitions, the gesture is optionally not recognized as meeting the first gesture recognition criteria for the first gesture if the second gesture recognition criteria for the second gesture are first satisfied. For example, if a contact reaches its respective intensity threshold before the contact moves by a predefined amount of movement, a deep pressing gesture rather than a swipe gesture is detected. Conversely, if the contact moves by a predefined amount of movement before the contact reaches its respective intensity threshold, a swipe gesture is detected rather than a deep pressing gesture. Even in such situations, the first gesture recognition criteria for the first gesture still do not require the strength of the contact(s) to meet the respective strength threshold in order for the first gesture recognition criteria to be satisfied, which means that the contact If the gesture remained below the respective intensity threshold until the end of (e.g., a swipe gesture with a contact that did not increase to an intensity above the respective intensity threshold), the gesture would have been recognized as a swipe gesture by the first gesture recognition criteria. Because it is. As such, certain gesture recognition criteria that do not require the strength of the contact(s) to meet their respective strength thresholds in order for certain gesture recognition criteria to be satisfied are, (A) in some situations (e.g., the tap gesture Case) ignore the intensity of the contact in relation to the intensity threshold and/or (B) in some situations (e.g., in the case of a long-press gesture competing with a deep-press gesture for recognition) specific gesture recognition criteria corresponding to the input Prior to recognizing the gesture, if a competing set of strength-dependent gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to a strength-dependent gesture (e.g., for a press-and-hold gesture). It will still depend on the strength of the contact relative to the intensity threshold in the sense that the gesture recognition criteria will fail.

The graphics module 132 includes components for changing the visual effect (e.g., brightness, transparency, saturation, contrast, or other visual property) of the displayed graphic. It includes various known software components for rendering and displaying. As used herein, the term “graphic” includes, without limitation, text, web pages, icons (eg, user interface objects including soft keys), digital images, videos, animations, etc. , Includes any object that can be displayed to the user.

In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphic module 132 receives from applications, etc., one or more codes specifying a graphic to be displayed, along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data to the display controller 156. Print.

The haptic feedback module 133 commands commands to generate tactile outputs using the tactile output generator(s) 167 at one or more locations on the device 100 in response to user interactions with the device 100. (E.g., instructions used by haptic feedback controller 161) various software components.

Optionally, the text input module 134, which is a component of the graphics module 132, may include various applications (e.g., contacts 137, email 140, IM 141), browser 147, and text input. Soft keyboards for entering text in any other application).

The GPS module 135 determines the location of the device and uses this information for use in various applications (e.g., to the phone 138 for use in location-based dialing, the camera ( 143), and to applications that provide location-based services such as weather widgets, local yellow page widgets and map/navigation widgets).

Applications 136, optionally, include the following modules (or sets of instructions), or a subset or superset thereof:

연락처 모듈(137)(때때로 주소록 또는 연락처 목록으로 지칭됨);

Contacts module 137 (sometimes referred to as an address book or contact list);

전화 모듈(138);

Telephone module 138;

화상 회의 모듈(139);

Video conferencing module 139;

이메일 클라이언트 모듈(140);

Email client module 140;

인스턴트 메시징(IM) 모듈(141);

Instant messaging (IM) module 141;

운동 지원 모듈(142);

Exercise support module 142;

정지 및/또는 비디오 이미지들을 위한 카메라 모듈(143);

A camera module 143 for still and/or video images;

이미지 관리 모듈(144);

Image management module 144;

브라우저 모듈(147);

Browser module 147;

캘린더 모듈(148);

A calendar module 148;

날씨 위젯(149-1), 주식 위젯(149-2), 계산기 위젯(149-3), 알람 시계 위젯(149-4), 사전 위젯(149-5), 및 사용자에 의해 얻어지는 다른 위젯들뿐 아니라 사용자-생성 위젯들(149-6) 중 하나 이상을 선택적으로 포함하는 위젯 모듈들(149);

Weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user. As well as widget modules 149 that selectively include one or more of user-generated widgets 149-6;

사용자-생성 위젯들(149-6)을 만들기 위한 위젯 생성기 모듈(150);

A widget generator module 150 for making user-generated widgets 149-6;

검색 모듈(151);

Search module 151;

비디오 재생기 모듈 및 음악 재생기 모듈로 선택적으로 구성된 비디오 및 음악 재생기 모듈(152);

A video and music player module 152 optionally configured with a video player module and a music player module;

메모 모듈(153);

Memo module 153;

지도 모듈(154); 및/또는

Map module 154; And/or

온라인 비디오 모듈(155).

Online video module 155.

Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption. Includes, digital rights management, voice recognition and voice duplication.

Along with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphic module 132 and the text input module 134, the contact module 137, including the following, address book or contact Contains executable instructions to manage a list (e.g., stored in memory 102 or in application internal state 192 of contacts module 137 in memory 370): name(s) to address book Adding; Deleting name(s) from the address book; Associating phone number(s), email address(s), physical address(s), or other information with a name; Associating an image with a name; Sorting and sorting names; Providing phone numbers and/or email addresses to initiate and/or facilitate communication by phone 138, video conference 139, email 140 or IM 141, and the like.

RF circuit unit 108, audio circuit unit 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, and Together with the text input module 134, the phone module 138 inputs a sequence of characters corresponding to the phone number, accesses one or more phone numbers in the address book 137, modifies the entered phone number, and Contains executable commands to dial the phone number of the phone, have a conversation, and disconnect or hang up when the conversation is complete. As noted above, wireless communication optionally uses any of a plurality of communication standards, protocols and technologies.

RF circuit unit 108, audio circuit unit 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, optical sensor(s) 164, optical sensor controller ( 158), a contact module 130, a graphics module 132, a text input module 134, a contact list 137, and a phone module 138, the video conferencing module 139 interacts with the user according to user instructions. Contains executable instructions for initiating, conducting, and ending a video conference between one or more other participants.

Together with the RF circuitry 108, the touch sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132 and the text input module 134, the email client module 140 Includes executable instructions for creating, sending, receiving, and managing email in response to the instructions. Together with the image management module 144, the email client module 140 makes it very easy to create and send an email with still or video images taken with the camera module 143.

With the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132 and the text input module 134, the instant messaging module 141, Enter a sequence of characters corresponding to an instant message, modify previously entered characters (e.g., short text message service (SMS) or multimedia message service (MMS) for phone-based instant messages) An executable that allows you to send individual instant messages, receive instant messages, and view received instant messages using protocols or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messages. Contains instructions. In some embodiments, the sent and/or received instant messages optionally include graphics, photos, audio files, video files, and/or other attachments as supported by MMS and/or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., XMPP, SIMPLE, APNs, or IMPS). Transmitted messages) both.

RF circuit unit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, text input module 134, GPS module 135, map module 154 ), and with the video and music player module 152, the exercise support module 142 devises exercises (eg, with time, distance, and/or calorie consumption goals); Communicate with athletic sensors (in sports devices and smart watches); Receive motion sensor data; Calibrate sensors used to monitor movement; Select and play music for exercise; Includes executable instructions to display, store, and transmit athletic data.

Touch-sensitive display system 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact module 130, graphic module 132, and image management module 144 and Together, the camera module 143 captures still images or video (including a video stream) and stores them in the memory 102 and/or modifies the properties of the still image or video, and/or ) To delete a still image or video from).

With the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphic module 132, the text input module 134 and the camera module 143, the image management module 144 is stopped. And/or arranging, modifying (eg, editing) video images, or otherwise manipulating, labeling, deleting, presenting (eg, in a digital slide show or album), and storing executable instructions. do.

Along with the RF circuit unit 108, the touch sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147, Execution to browse the Internet according to user instructions, including retrieving, linking to, receiving, and displaying web pages or parts thereof, as well as attachments and other files linked to web pages. Includes possible commands.

RF circuitry 108, touch sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser Along with module 147, calendar module 148 creates, displays, modifies, and stores calendars and data associated with calendars (e.g., calendar entries, to-do lists, etc.) according to user instructions. Contains executable instructions that cause

Together with the RF circuit unit 108, the touch sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134 and the browser module 147, the widget Modules 149 are selectively downloaded and used by the user (e.g., weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4) And dictionary widget 149-5), or mini-applications created by the user (eg, user-generated widget 149-6). In some embodiments, the widget includes a Hypertext Markup Language (HTML) file, Cascading Style Sheets (CSS) file, and a JavaScript file. In some embodiments, the widget includes an Extensible Markup Language (XML) file and a JavaScript file (eg, Yahoo! widgets).

Together with the RF circuit unit 108, the touch sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134 and the browser module 147, the widget The generator module 150 includes executable instructions that cause widgets to be created (eg, change a user-specified portion of a web page into a widget).

Along with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 is one or more Contains executable instructions that allow searching for text, music, sound, image, video, and/or other files in memory 102 that match search criteria (eg, one or more user-specific search terms).

Touch-sensitive display system 112, display system controller 156, contact module 130, graphic module 132, audio circuit unit 110, speaker 111, RF circuit unit 108, and browser module 147 ), the video and music player module 152 can download and play videos and executable instructions that allow a user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files. Include executable instructions to display, present, or otherwise play (eg, on the touch-sensitive display system 112, or wirelessly or on an external display connected through an external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

Along with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the memo module 153 can take notes according to user instructions, Contains executable instructions to create and manage to-do lists and the like.

RF circuit unit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module In conjunction with 147, the map module 154 may also provide maps and data associated with the maps (e.g., driving directions; stores and other points of interest at or near a specific location) according to user instructions. And other location-based data), including executable instructions for receiving, displaying, modifying, and storing data.

Touch-sensitive display system 112, display system controller 156, contact module 130, graphic module 132, audio circuit unit 110, speaker 111, RF circuit unit 108, text input module 134 ), email client module 140, and browser module 147, the online video module 155 allows a user to access online videos in one or more file formats such as H.264, browse them, (e.g. Receive, for example, by streaming and/or download), play (e.g., on the touch screen 112, or wirelessly or on an external display connected through an external port 124), and with a specific online video It contains executable instructions that allow you to send an email with a link to and otherwise manage. In some embodiments, instant messaging module 141 rather than email client module 140 is used to send a link to a particular online video.

Each of the modules and applications identified above is an executable instruction for performing one or more functions described above and methods described herein (e.g., computer-implemented methods and other information processing methods described herein). Corresponds to a set of things. These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. . In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. In addition, memory 102 optionally stores additional modules and data structures not described above.

In some embodiments, device 100 is a device in which operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad. By using the touch screen and/or touchpad as the primary input control device for operation of the device 100, the number of physical input control devices (such as push buttons, dials, etc.) on the device 100 is selectively reduced.

The predefined set of functions performed entirely through the touch screen and/or touchpad, optionally, includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 to a main, home or root menu from any user interface displayed on the device 100. In these embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.

1B is a block diagram illustrating example components for event processing, in accordance with some embodiments. In some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3)) includes event classifier 170 (e.g., in operating system 126) and individual application 136-1 (e.g., above). One of the applications 136, 137-155, 380-390).

The event classifier 170 receives event information, and determines an application 136-1 to which the event information is to be transmitted, and an application view 191 of the application 136-1. The event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 that indicates the current application view(s) displayed on the touch-sensitive display system 112 when the application is active or running. In some embodiments, the device/global internal state 157 is used by the event classifier 170 to determine which application(s) are currently active, and the application internal state 192 is applied to the event classifier 170. It is used to determine application views 191 to deliver event information by.

In some embodiments, the application internal state 192 is a user representing resume information to be used when application 136-1 resumes execution, information being displayed by application 136-1 or ready to be displayed. Among the interface state information, a state queue to enable the user to return to the previous state or view of the application 136-1, and a redo/undo queue of previous actions taken by the user. Include additional information, such as one or more.

The event monitor 171 receives event information from the peripheral device interface 118. The event information includes information about sub-events (eg, user touch on the touch-sensitive display system 112 as part of a multi-touch gesture). Peripheral interface 118 is from the I/O subsystem 106 or sensors, such as proximity sensor 166, accelerometer(s) 168, and/or microphone 113 (via audio circuitry 110). Send the information it receives. The information that the peripherals interface 118 receives from the I/O subsystem 106 includes information from the touch-sensitive display system 112 or touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response to this, the peripheral device interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (e.g., receiving an input exceeding a predetermined noise threshold and/or receiving an input for exceeding a predetermined duration). .

In some embodiments, event classifier 170 also includes a hit view determination module 172 and/or an active event recognizer determination module 173.

The hit view determination module 172 provides software procedures for determining where a sub-event has occurred within one or more views when the touch-sensitive display system 112 displays more than one view. Views are made up of controls and other elements that the user can see on the display.

Another aspect of a user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, where information is displayed and touch-based gestures occur. The application views (of each application) for which a touch is detected optionally correspond to a program of the application or program levels within the view hierarchy. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as appropriate inputs is optionally based at least in part on the hit view of the initial touch initiating the touch-based gesture Is determined.

The hit view determination module 172 receives information related to sub-events of a touch-based gesture. If the application has multiple views organized in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchy that should handle sub-events. In most situations, the hit view is the lowest level view in which the subevent being initiated (ie, the first subevent in the sequence of subevents forming the event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source that caused it to be identified as a hit view.

The active event recognizer determination module 173 determines which views or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of subevents. In other embodiments, the active event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively involved views, so that all actively involved views are It decides that the sequence should be received. In other embodiments, even if touch sub-events were entirely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively engaged views.

The event dispatcher module 174 dispatches event information to an event recognizer (eg, event recognizer 180). In embodiments including the active event recognizer determination module 173, the event dispatcher module 174 communicates the event information to the event recognizer determined by the active event recognizer determination module 173. In some embodiments, the event dispatcher module 174 stores event information in an event queue, which event information is retrieved by each event receiver module 182.

In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In still other embodiments, event classifier 170 is a standalone module, or part of another module that is stored in memory 102, such as contact/motion module 130.

In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which is a touch event occurring within each view of the application's user interface. Contains instructions for processing them. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, each application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 may be part of a separate module, such as a user interface kit (not shown) or a higher level object for which the application 136-1 inherits methods and other properties. to be. In some embodiments, each event handler 190 is one of the data updater 176, the object updater 177, the GUI updater 178, and/or the event data 179 received from the event classifier 170. It includes the above. The event handler 190 optionally updates the application internal state 192 by using or calling the data updater 176, the object updater 177, or the GUI updater 178. Alternatively, one or more of the application views 191 include one or more respective event handlers 190. Further, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included within each application view 191.

Each event recognizer 180 receives event information (eg, event data 179) from event classifier 170 and identifies an event from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event recognizer 180 also includes at least metadata 183 and a subset of event delivery instructions 188 (optionally including sub-event delivery instructions).

The event receiver 182 receives event information from the event classifier 170. The event information includes information about a sub-event, for example, a touch or a touch movement. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. If the sub-event is related to the motion of the touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, events include rotation of the device from one orientation to another (e.g., from portrait to landscape orientation, or vice versa), and the event information is the current orientation of the device (also referred to as device orientation). Contains the corresponding information about

The event comparator 184 compares the event information with predefined event or sub-event definitions, and based on the comparison, determines an event or sub-event, or determines or updates a state of an event or sub-event. In some embodiments, event comparator 184 includes event definitions 186. Event definitions 186 include definitions of events (eg, predefined sequences of sub-events), eg, event 1 (187-1), event 2 (187-2), and the like. In some embodiments, sub-events within event 187 include, for example, touch start, touch end, touch move, touch cancel, and multiple touches. In one example, the definition for event 1 187-1 is a double tap on the displayed object. The double tap is, for example, a first touch on a displayed object during a predetermined phase (touch start), a first liftoff during a predetermined phase (touch end), the displayed object during a predetermined phase. A second touch on the image (touch start), and a second liftoff during a predetermined phase (touch end). In another example, the definition for event 2 187-2 is dragging on the displayed object. Dragging includes, for example, a touch (or touch) on a displayed object during a predetermined phase, movement of the touch across the touch-sensitive display system 112, and lift-off of the touch (touch termination). In some embodiments, the event also includes information about one or more associated event handlers 190.

In some embodiments, event definition 187 includes a definition of an event for each user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view in which three user interface objects are displayed on the touch-sensitive display system 112, when a touch is detected on the touch-sensitive display system 112, the event comparator 184 is A hit test is performed to determine which of the interface objects are associated with the touch (sub-event). When each displayed object is associated with each event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, the event comparator 184 selects the object triggering the hit test and the event handler associated with the sub-event.

In some embodiments, the definition for each event 187 also includes delayed actions that delay delivery of event information until after it is determined whether the sequence of subevents corresponds to or does not correspond to the event type of the event recognizer. .

When each event recognizer 180 determines that a series of sub-events do not match any of the events in the event definitions 186, each event recognizer 180 may be unable to event, fail the event, or After entering the event end state, each event recognizer ignores subsequent sub-events of the touch-based gesture. In this situation, other event recognizers that remain active for the hit view, if any, continue to track and process sub-events of the ongoing touch-based gesture.

In some embodiments, each event recognizer 180 contains configurable attributes, flags, and/or lists that indicate how the event delivery system should perform sub-event delivery to actively engaged event recognizers. It includes metadata 183 to have. In some embodiments, metadata 183 includes configurable attributes, flags, and/or lists that indicate how the event recognizers interact with each other, or become interactive. In some embodiments, metadata 183 includes configurable attributes, flags, and/or lists that indicate whether sub-events are delivered to various levels in the view or program hierarchy.

In some embodiments, each event recognizer 180 activates an event handler 190 associated with the event when one or more specific subevents of the event are recognized. In some embodiments, each event recognizer 180 communicates event information associated with the event to the event handler 190. Activating the event handler 190 is separate from sending (and delaying sending) sub-events to each hit view. In some embodiments, event recognizer 180 sends a flag associated with the recognized event, and event handler 190 associated with the flag catches the flag and performs a predefined process.

In some embodiments, the event delivery instructions 188 include subevent delivery instructions that deliver event information about the subevent without activating the event handler. Instead, subevent delivery instructions deliver event information to event handlers associated with a series of subevents or to actively participating views. Event handlers associated with a series of sub-events or actively participating views receive event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used by application 136-1. For example, the data updater 176 updates a phone number used in the contact module 137 or stores a video file used in the video and music player module 152. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, the object updater 177 creates a new user interface object or updates the location of the user interface object. The GUI updater 178 updates the GUI. For example, the GUI updater 178 prepares display information for display on a touch-sensitive display and transmits it to the graphics module 132.

In some embodiments, event handler(s) 190 includes or accesses data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are contained within a single module of each application 136-1 or application view 191. In other embodiments, they are contained within two or more software modules.

The above discussion regarding event handling of user touches on a touch-sensitive display also applies to other types of user inputs for operating multifunction devices 100 with input devices, but all of which are initiated on touch screens. It must be understood that it is not. For example, mouse movements and mouse button presses selectively coordinated with single or multiple keyboard presses or hold; Contact movements such as tap, drag, scroll, etc. on the touch pad; Pen stylus inputs; Device movement; Verbal commands; Detected eye movements; Biometric inputs; And/or any combination thereof are optionally used as inputs corresponding to subevents defining the event to be recognized.

1C is a block diagram illustrating a tactile output module according to some embodiments. In some embodiments, the I/O subsystem 106 (e.g., haptic feedback controller 161 (FIG. 1A) and/or other input controller(s) 160 (FIG. 1A)) is shown in FIG. Includes at least some of the example components. In some embodiments, peripherals interface 118 includes at least some of the exemplary components shown in FIG. 1C.

In some embodiments, the tactile output module includes a haptic feedback module 133. In some embodiments, the haptic feedback module 133 provides user interface feedback from software applications on the electronic device (e.g., alerts and other notifications indicating the occurrence of events or performance of actions in user interfaces of the electronic device) and Tactile outputs for (feedback responsive to user inputs corresponding to the displayed user interfaces) are collected and combined. The haptic feedback module 133 includes a waveform module 123 (for providing waveforms used to generate tactile outputs), a mixer 125 (for mixing waveforms such as waveforms in different channels), A compressor 127 (for reducing or compressing the dynamic range of the waveforms), a low pass filter 129 (for filtering high-frequency signal components in the waveforms), and for adjusting the waveforms according to thermal conditions. ) Including one or more of the thermal controller 131. In some embodiments, the haptic feedback module 133 is included within the haptic feedback controller 161 (FIG. 1A ). In some embodiments, a separate unit of the haptic feedback module 133 (or a separate implementation of the haptic feedback module 133) is also included within the audio controller (e.g., audio circuitry 110, FIG. It is used to generate signals. In some embodiments, a single haptic feedback module 133 is used to generate waveforms for tactile outputs and generate audio signals.

In some embodiments, the haptic feedback module 133 also includes a trigger module 121 (e.g., a software application that initiates a process for determining a generated tactile output and generating a corresponding tactile output, an operating system, Or other software modules). In some embodiments, trigger module 121 generates trigger signals to initiate generation of waveforms (eg, by waveform module 123 ). For example, the trigger module 121 generates trigger signals based on preset timing criteria. In some embodiments, trigger module 121 receives trigger signals from outside of haptic feedback module 133 (e.g., in some embodiments, haptic feedback module 133 is outside of haptic feedback module 133 ). Receive trigger signals from a hardware input processing module 146 located at), trigger signals into a user interface element (e.g., an affordance in an application or an application icon) or a hardware input device (e.g., an intensity sensitive touch screen, Software applications that trigger actions (e.g., using trigger module 121) based on activation of an input surface or home button) or other components within haptic feedback module 133 (e.g., waveform module 123) To relay. In some embodiments, trigger module 121 also receives tactile feedback generation instructions (eg, from haptic feedback module 133 of FIGS. 1A and 3 ). In some embodiments, the trigger module 121 is a haptic feedback module 133 (or a trigger module 121 in the haptic feedback module 133) (e.g., from the haptic feedback module 133 of FIGS. 1A and 3 ). In response to receiving these tactile feedback instructions, trigger signals are generated.

Waveform module 123 receives trigger signals as input (e.g., from trigger module 121), and in response to receiving the trigger signals, waveforms (e.g., Figure 4F and Waveforms selected from a predefined set of waveforms designated for use by waveform module 123, such as the waveforms described in more detail below with reference to FIG. 4G are provided.

Mixer 125 receives waveforms as input (eg, from waveform module 123) and mixes the waveforms together. For example, when the mixer 125 receives two or more waveforms (e.g., a first waveform in a first channel and a second waveform that at least partially overlaps the first waveform in the second channel), the mixer 125 outputs a combined waveform corresponding to the summation of two or more waveforms. In some embodiments, mixer 125 may also (e.g., increase the scale of the particular waveform(s) and/or decrease the scale of the remainder of the waveforms) to convert the particular waveform(s) into Modify one or more of the two or more waveforms to accentuate them. In some situations, mixer 125 selects one or more waveforms to remove from the combined waveform (e.g., waveforms from more than three sources that were requested to be output simultaneously by tactile output generator 167 When present, the waveform from the oldest source is dropped).

Compressor 127 receives as input the waveforms (eg, the combined waveform from mixer 125) and modifies the waveforms. In some embodiments, compressor 127 reduces the waveforms (e.g., according to the physical specifications of the tactile output generators (167 in FIG. 1A or 357 in FIG. 3)) to reduce the tactile Let the outputs be reduced. In some embodiments, compressor 127 limits the waveforms, such as by forcing a predefined maximum amplitude for the waveforms. For example, compressor 127 reduces the amplitudes of the portions of the waveforms that exceed the predefined amplitude threshold while maintaining the amplitudes of the portions of the waveforms that do not exceed the predefined amplitude threshold. In some embodiments, compressor 127 reduces the dynamic range of the waveforms. In some embodiments, the compressor 127 dynamically reduces the dynamic range of the waveforms so that the combined waveforms can be applied to the performance specifications of the tactile output generator 167 (eg, force and/or movable mass displacement limits) Keep within.

The low-pass filter 129 receives waveforms (e.g., compressed waveforms from the compressor 127) as input, and filters (e.g., smoothes) the waveforms (e.g., removing high-frequency signal components from the waveforms). Or decrease). For example, in some cases, the compressor 127 exceeds the performance specifications of the tactile output generator 167 and/or tactile when the tactile outputs are generated according to the compressed waveforms, with compressed waveforms. It contains unrelated signals (eg, high-frequency signal components) that interfere with the generation of outputs. The low pass filter 129 reduces or removes such irrelevant signals in the waveforms.

Thermal controller 131 receives waveforms (e.g., filtered waveforms from low pass filter 129) as input, and (e.g., internal temperatures detected within device 100, e.g., haptic feedback controller ( Adjust the waveforms according to the thermal conditions of the device 100) based on the temperature of 161 and/or external temperatures detected by the device 100. For example, in some cases, the output of haptic feedback controller 161 varies with temperature (e.g., haptic feedback controller 161 is, in response to receiving the same waveforms, haptic feedback controller 161 is first It generates a first tactile output when at temperature, and a second tactile output when haptic feedback controller 161 is at a second temperature separate from the first temperature). For example, the magnitude (or amplitude) of tactile outputs can vary with temperature. To reduce the effect of temperature changes, the waveforms are modified (eg, the amplitude of the waveforms increases or decreases based on temperature).

In some embodiments, the haptic feedback module 133 (eg, trigger module 121) is coupled to the hardware input processing module 146. In some embodiments, the other input controller(s) 160 in FIG. 1A includes a hardware input processing module 146. In some embodiments, the hardware input processing module 146 is a hardware input device 145 (e.g., other input or control devices 116 in FIG. 1A, e.g., an intensity sensitive input surface such as an intensity sensitive touch screen). Or the home button). In some embodiments, the hardware input device 145 is any input device described herein, such as a touch-sensitive display system 112 (Fig. 1A), a keyboard/mouse 350 (Fig. 3), a touchpad. 355 (Fig. 3), one of the other input or control devices 116 (Fig. 1A), or a strength-sensitive home button. In some embodiments, the hardware input device 145 is not a touch-sensitive display system 112 (FIG. 1A), a keyboard/mouse 350 (FIG. 3), or a touchpad 355 (FIG. 3). It consists of a strength-sensitive home button. In some embodiments, in response to inputs from the hardware input device 145 (e.g., intensity-sensitive home button or touch screen), the hardware input processing module 146 One or more trigger signals are provided to the haptic feedback module 133 to indicate that an input, such as an input corresponding to a “click” of the home button (eg, “down click” or “up click”), has been detected. In some embodiments, the haptic feedback module 133 simulates haptic feedback of pressing the physical home button by providing waveforms corresponding to the "click" of the home button in response to an input corresponding to the "click" of the home button. do.

In some embodiments, the tactile output module includes a haptic feedback controller 161 (eg, haptic feedback controller 161 in FIG. 1A), which controls the generation of tactile outputs. In some embodiments, the haptic feedback controller 161 is coupled to a plurality of tactile output generators, for selecting one or more of the plurality of tactile output generators to generate tactile outputs. The waveforms are sent to the selected one or more tactile output generators. In some embodiments, the haptic feedback controller 161 includes tactile output requests corresponding to activation of the hardware input device 145 and tactile output requests corresponding to software events (e.g., a haptic feedback module ( 133) and prioritize tactile outputs corresponding to activation of hardware input device 145 over tactile outputs corresponding to software events (e.g., to prioritize tactile outputs corresponding to activation of hardware input device 145). , By increasing the scale of the particular waveform(s) and/or decreasing the scale of the other of the waveforms) to emphasize one or more of the two or more waveforms to emphasize the particular waveform(s) relative to the rest of the two or more waveforms. Modify it.

In some embodiments, as shown in FIG. 1C, the output of the haptic feedback controller 161 is coupled to the audio circuitry of the device 100 (e.g., the audio circuitry 110 of FIG. It is provided to the audio circuit part of the device 100. In some embodiments, the haptic feedback controller 161 provides both waveforms used to generate tactile outputs and audio signals used to provide audio outputs with the generation of tactile outputs. In some embodiments, the haptic feedback controller 161 modifies the audio signals and/or waveforms (used to generate tactile outputs), so that the audio signal (e.g., by delaying the audio signals and/or waveforms) Allow the outputs and tactile outputs to be synchronized. In some embodiments, the haptic feedback controller 161 includes a digital-to-analog converter used to convert digital waveforms into analog signals, the analog signals being an amplifier 163 and/or a tactile output generator 167 Received by ).

In some embodiments, the tactile output module includes an amplifier 163. In some embodiments, amplifier 163 receives waveforms (e.g., from haptic feedback controller 161), amplifies the waveforms, and then outputs the amplified waveforms to tactile output generator 167 (e.g., tactile output). To the generators (any of 167 in FIG. For example, the amplifier 163 converts the received waveforms into signal levels according to the physical specifications of the tactile output generator 167 (e.g., the signals transmitted to the tactile output generator 167 are converted to a haptic feedback controller ( 161), amplified with the voltage and/or current required by the tactile output generator 167 to generate tactile outputs to generate tactile outputs corresponding to the waveforms received from the amplified waveforms, and tactile the amplified waveforms. To the output generator 167. In response, the tactile output generator 167 generates tactile outputs (eg, by shifting the movable mass back and forth in one or more dimensions relative to the neutral position of the movable mass).

In some embodiments, the tactile output module includes a sensor 169, which is coupled to the tactile output generator 167. The sensor 169 may be of the tactile output generator 167 or one or more components of the tactile output generator 167 (e.g., one or more moving parts such as a membrane used to generate tactile outputs). Detect states or state changes (eg, mechanical position, physical displacement, and/or movement). In some embodiments, sensor 169 is a magnetic field sensor (eg, a Hall effect sensor) or other displacement and/or movement sensor. In some embodiments, the sensor 169 provides information (e.g., the location, displacement, and/or movement of one or more parts within the tactile output generator 167) to the haptic feedback controller 161 and provides a tactile output. Depending on the information provided by the sensor 169 about the state of the generator 167, the haptic feedback controller 161 can provide the haptic feedback controller 161 with waveforms output from the haptic feedback controller 161 (e.g. The waveforms transmitted to the red output generator 167) are adjusted.

2 illustrates a portable multifunction device 100 having a touch screen (eg, touch-sensitive display system 112, FIG. 1A), in accordance with some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In these embodiments, as well as other embodiments described below, the user may, for example, one or more fingers 202 (not shown to scale in the figure) or one or more stylus 203 (scale in the figure). Not shown as shown) makes it possible to select one or more of the graphics by making a gesture on the graphic. In some embodiments, the selection of one or more graphics occurs when the user stops contacting the one or more graphics. In some embodiments, the gesture may optionally be one or more taps of a finger in contact with the device 100, one or more swipes (left to right, right to left, upwards and/or downwards). And/or rolling (right-to-left, left-to-right, upwards and/or downwards). In some implementations or situations, if the graphic is inadvertently contacted, the graphic is not selected. For example, when the gesture corresponding to the selection is a tap, the swipe gesture of sweeping over the application icon selectively does not select the corresponding application.

Device 100 also optionally includes one or more physical buttons, such as a “home” or menu button 204. As described above, the menu button 204 is optionally used to navigate to any application 136 in the set of applications that are selectively running on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

In some embodiments, the device 100 includes a touch screen display, a menu button 204 (sometimes referred to as the home button 204), a push button 206 to power on/off the device and lock the device, Volume control button(s) 208, a subscriber identification module (SIM) card slot 210, a headset jack 212, and a docking/charging external port 124. The push button 206 optionally powers on/off the device by pressing the button and holding the button pressed for a predefined time interval; Locking/locking the device by pressing the button and releasing the button before a predefined time interval elapses; It is used to unlock the device or initiate the unlock process. In some embodiments, device 100 also accepts verbal input for activation or deactivation of some functions via microphone 113. Device 100 may also optionally provide a tactile output for a user of device 100 and/or one or more contact intensity sensors 165 to detect intensities of contacts on touch-sensitive display system 112. And one or more tactile output generators 167 for generating them.

3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, in accordance with some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., home or industrial Controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses for interconnecting these components. 320). Communication buses 320 optionally include circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between them. Device 300 includes an input/output (I/O) interface 330 that includes a display 340, which is typically a touch screen display. The I/O interface 330 also optionally includes a keyboard and/or mouse (or other pointing device) 350 and a touchpad 355, a tactile output for generating tactile outputs on the device 300. Generator 357 (e.g., similar to tactile output generator(s) 167 described above with reference to FIG. 1A), and sensors 359 (e.g., light sensor, acceleration sensor, proximity sensor, touch sensitive Sensor, and/or a contact intensity sensor similar to the contact intensity sensor(s) 165 described above with reference to FIG. 1A). Memory 370 includes high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; Optionally one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other nonvolatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU(s) 310. In some embodiments, memory 370 includes programs, modules, and data structures similar to programs, modules, and data structures stored in memory 102 of portable multifunction device 100 (FIG. 1A). , Or a subset of them. In addition, memory 370 optionally stores additional programs, modules and data structures not present in memory 102 of portable multifunction device 100. For example, the memory 370 of the device 300 is, optionally, a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388 ), and/or the spreadsheet module 390, while the memory 102 of the portable multifunction device 100 (FIG. 1A), optionally, does not store these modules.

Each of the previously identified elements in FIG. 3 is, optionally, stored in one or more of the memory devices described above. Each of the previously identified modules corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., sets of instructions) identified above need not be implemented as separate software programs, procedures, or modules, so in various embodiments, various subsets of these modules are selectively combined. Or otherwise rearranged. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Further, the memory 370 optionally stores additional modules and data structures not described above.

Attention is now directed to embodiments of user interfaces (“UI”) that are selectively implemented on the portable multifunction device 100.

4A illustrates an exemplary user interface 400 for a menu of applications on a portable multifunction device 100, in accordance with some embodiments. Similar user interfaces are optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

셀룰러 및 Wi-Fi 신호들과 같은 무선 통신(들)에 대한 신호 강도 표시자(들);

Signal strength indicator(s) for wireless communication(s) such as cellular and Wi-Fi signals;

시간;

time;

블루투스 표시자;

Bluetooth indicator;

배터리 상태 표시자;

Battery status indicator;

다음과 같은, 빈번하게 사용되는 애플리케이션들에 대한 아이콘들을 갖는 트레이(408):

Tray 408 with icons for frequently used applications, such as:

o An icon 416 for the phone module 138 labeled "Call", optionally including an indicator 414 of the number of missed calls or voicemail messages;

o An icon 418 for an email client module 140 labeled “mail” that optionally includes an indicator 410 of the number of unread emails;

o An icon 420 for the browser module 147 labeled “browser”; and

o An icon 422 for the video and music player module 152, labeled “music”; and

다음과 같은, 다른 애플리케이션들에 대한 아이콘들:

Icons for different applications, such as:

o An icon 424 for the IM module 141, labeled “Message”;

o An icon 426 for the calendar module 148, labeled “calendar”;

o An icon 428 for the image management module 144 labeled "Photo";

o Icon 430 for camera module 143, labeled "camera";

o Icon 432 for online video module 155, labeled “online video”;

o Icon 434 for stock widget 149-2, labeled "stock";

o An icon 436 for the map module 154, labeled "Map";

o Icon 438 for weather widget 149-1 labeled "Weather";

o Icon 440 for alarm clock widget 149-4, labeled "Clock";

o Icon 442 for exercise support module 142, labeled "Exercise Support";

o An icon 444 for the notes module 153, labeled “Notes”;

o An icon 446 for a settings application or module that provides access to settings for device 100 and its various applications 136.

It should be noted that the icon labels illustrated in FIG. 4A are only examples. For example, different labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to each application icon. In some embodiments, the label for a specific application icon is separate from the name of the application corresponding to the specific application icon.

4B shows an exemplary user interface on a device (e.g., device 300, FIG. 3) having a touch-sensitive surface 451 (e.g., tablet or touchpad 355, FIG. 3) that is separate from the display 450. Illustrate. While many of the following examples will be provided with reference to inputs on the touch screen display 112 (the touch-sensitive surface and display are combined), in some embodiments, the device is separate from the display, as shown in Figure 4B. Detect inputs on a personal touch-sensitive surface. In some embodiments, the touch-sensitive surface (eg, 451 in FIG. 4B) has a major axis (eg, 452 in FIG. 4B) that corresponds to a major axis (eg, 453 in FIG. 4B) on the display (eg, 450 ). According to these embodiments, the device may have a touch-sensitive surface 451 at locations corresponding to respective locations on the display (e.g., in FIG. 4B, 460 corresponds to 468, and 462 corresponds to 470). The contacts with (e.g., 460, 462 in FIG. 4B) are detected. In this way, user inputs (e.g., contacts 460, 462 and their movements) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4B) are independent of the display. Is used by the device to manipulate the user interface on the display of the multifunction device (eg, 450 in FIG. 4B). It should be understood that similar methods are, optionally, used for other user interfaces described herein.

Additionally, while the following examples are given primarily with reference to finger inputs (e.g., finger contacts, finger tap gestures, finger swipe gestures, etc.), in some embodiments, one or more of the finger inputs are It should be understood that input from the device (eg, mouse based input or stylus input) is substituted. For example, a swipe gesture is optionally replaced with a mouse click (eg, instead of a contact) and subsequent movement of the cursor along the path of the swipe (eg, instead of a movement of the contact). As another example, the tap gesture is optionally replaced by a mouse click while the cursor is positioned over the location of the tap gesture (eg, instead of stopping detecting the contact following detection of the contact). Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice are selectively used simultaneously, or mouse and finger contacts are selectively used simultaneously.

As used herein, the term “focus selector” refers to an input element representing the current portion of the user interface that is interacting with the user. In some implementations that include a cursor or other location marker, a touch-sensitive surface (e.g., FIG. 3) while the cursor is over a specific user interface element (e.g., button, window, slider or other user interface element). When an input (e.g., a pressing input) is detected on the touchpad 355 of FIG. 4B or on the touch-sensitive surface 451 of FIG. 4B, the cursor is a “focus selector” so that a specific user interface element is adjusted according to the detected input. Functions as In some implementations including a touch screen display (e.g., the touch-sensitive display system 112 of FIG. 1A or the touch screen of FIG. 4A) that enables direct interaction with user interface elements on the touch screen display, When an input (e.g., a pressing input by contact) is detected on the touch screen display at the location of a particular user interface element (e.g., button, window, slider or other user interface element), a particular user interface element is To be adjusted accordingly, the contact detected on the touch screen functions as a "focus selector". In some implementations, the focus is one of the user interface without moving the corresponding cursor or moving the contact (e.g., by using a tab key or an arrow key to move the focus from one button to another). Is moved to another area of the user interface from the area of In such implementations, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the particular form the focus selector has, the focus selector is generally used to convey the user's intended interaction with the user interface (e.g., by indicating to the device an element of the user interface the user wishes to interact with). Is a user interface element (or contact on a touch screen display) controlled by. For example, the position of the focus selector (e.g., cursor, contact, or selection box) above each button while a press input is detected on a touch-sensitive surface (e.g., touchpad or touch screen) is (on the display of the device). Unlike other user interface elements shown) it will indicate that the user is trying to activate each button.

As used in the specification and claims, the term “strength” of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (eg, finger contact or stylus contact) on a touch-sensitive surface. , Or a substitute for the force or pressure of a contact on a touch-sensitive surface. The intensity of the contact has a range of values that includes at least 4 distinct values and more typically includes several hundred (eg, at least 256) distinct values. The strength of the contact is selectively determined (or measured) using various approaches, and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (eg, weighted average or summed) to determine an estimated force of the contact. Similarly, the pressure sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and/or changes thereto, the capacitance of the touch-sensitive surface near the contact and/or changes thereto, and/or the touch-sensitive surface near the contact. The resistance of and/or changes thereto are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, alternative measurements for contact force or pressure are used directly to determine whether an intensity threshold has been exceeded (eg, the intensity threshold is described in units corresponding to the alternative measurements). In some implementations, alternative measurements for contact force or pressure are converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., an intensity threshold. Is the pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input would otherwise display affordances (e.g., on a touch-sensitive display) and/or (e.g., a touch-sensitive display, a touch-sensitive surface, or User access to additional device functions that may not be readily accessible by the user on a reduced size device with a limited footprint to receive user input (via physical/mechanical controls such as knobs or buttons) Makes it possible.

In some embodiments, the contact/motion module 130 is a set of one or more intensity thresholds to determine whether an action has been performed by a user (eg, to determine whether the user has “clicked” on an icon). Use In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators, it is possible to change the physical hardware of the device 100). Can be adjusted without). For example, the mouse “click” threshold of a trackpad or touch screen display can be set to any of a wide range of predefined thresholds without changing the trackpad or touch screen display hardware. Additionally, in some implementations, the user of the device may (eg, by adjusting individual intensity thresholds and/or by adjusting multiple intensity thresholds at once with a system level click “intensity” parameter) one of a set of intensity thresholds. Software settings are provided to adjust the anomaly.

As used in the specification and claims, the term “characteristic strength” of a contact refers to a property of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is, optionally, a predefined number of intensity samples, or (e.g., after detecting the contact, before detecting the liftoff of the contact, before or after detecting the start of the movement of the contact, Prior to detecting the end of the contact, before or after detecting an increase in the intensity of the contact, and/or before or after detecting a decrease in the intensity of the contact) a predetermined period (e.g.,) for a predefined event. 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds). The characteristic strength of the contact is, optionally, a maximum value of the contact intensities, a mean value of the contact intensities, an average value of the contact intensities, and a top 10 percentile value of the contact intensities. ), half of the maximum value of the intensities of the contact, 90 percent of the maximum value of the intensities of the contact, a value generated by low-pass filtering the intensity of the contact starting at a predefined time or over a predefined period. Is based on one or more of the like. In some embodiments, the duration of the contact is used to determine the feature strength (eg, when the feature strength is the average of the strength of the contact over time). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether the operation was performed by the user. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, as a result of a contact having a characteristic intensity that does not exceed the first intensity threshold, a first operation is performed, and a result of a contact having a characteristic intensity that exceeds the first intensity threshold but not exceeding the second intensity threshold, the second The second operation is performed, and as a result of the contact having a characteristic intensity exceeding the second intensity threshold, a third operation is performed. In some embodiments, a comparison between a characteristic strength and one or more strength thresholds is not used to determine whether to perform a first action or a second action, but rather whether to perform one or more actions (e.g., each option. It is used to determine whether to perform or suspend performing each operation).

In some embodiments, a portion of the gesture is identified to determine a characteristic strength. For example, the touch-sensitive surface may receive a continuous swipe contact (eg, a drag gesture) that transitions from the start position and reaches the end position (the intensity of the contact increases at this point). In this example, the characteristic strength of the contact at the end position may be based on only a portion of the continuous swipe contact (eg, only the portion of the swipe contact at the end position), not the entire swipe contact. In some embodiments, a smoothing algorithm may be applied to the swipe contact's intensities prior to determining the contact's characteristic strength. For example, the smoothing algorithm may optionally be one of an unweighted sliding-average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and/or an exponential smoothing algorithm. Contains one or more. In some situations, these smoothing algorithms remove narrow spikes or dips in the strengths of the swipe contact to determine the feature strength.

The user interface diagrams described herein optionally include one or more intensity thresholds (e.g., a touch detection intensity threshold (IT ₀ ), a light tap intensity threshold (IT _L ), a deep tap intensity threshold (IT _D )) (e.g., At least initially higher than IT _L ), and/or various intensity diagrams showing the current intensity of the contact on the touch-sensitive surface for one or more other intensity thresholds (e.g., intensity threshold lower than IT _L (IT _H )). These intensity diagrams are typically not part of the displayed user interface, but are provided to aid in the interpretation of the figures In some embodiments, the tap intensity threshold is that the device is a physical mouse button or trackpad. Corresponds to the intensity at which the device will perform actions typically associated with clicking a. In some embodiments, the deep pressing intensity threshold may cause different actions than actions typically associated with clicking a button or trackpad of a physical mouse. Corresponding to the intensity to be performed, in some embodiments, the contact is less than the light pressing intensity threshold (eg, and above the nominal contact detection intensity threshold IT ₀ (less than the contact no longer detected)). When detected as a characteristic intensity, the device will move the focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an action associated with the lightly press intensity threshold or the deeply press intensity threshold. On the one hand, these intensity thresholds are consistent among different sets of user interface drawings.

In some embodiments, the device's response to inputs detected by the device is dependent on criteria based on contact strength during input. For example, for some “press lightly” inputs, the strength of the contact that exceeds the first strength threshold during input triggers a first response. In some embodiments, the device's response to inputs detected by the device is dependent on criteria including both contact strength during input and time based criteria. For example, for some "press deep" inputs, the intensity of the contact that exceeds the second intensity threshold during input, which is greater than the first intensity threshold for light tapping, meets the first intensity threshold and the second intensity threshold The second response is triggered only when the delay time has elapsed between satisfying. This delay time is typically less than 200 ms (milliseconds) duration (e.g., 40, 100, or 120 ms depending on the magnitude of the second intensity threshold, where the delay time increases as the second intensity threshold increases). )to be. This delay time helps prevent accidental recognition of a deep press input. As another example, for some "press deep" inputs, there is a sensitivity reduction period that occurs after the time the first intensity threshold is met. During the sensitivity reduction period, the second intensity threshold is increased. This temporary increase in the second intensity threshold also helps to prevent accidental deep press input. For other deep press inputs, the response to detection of the deep press input does not depend on time-based criteria.

In some embodiments, one or more of the input intensity thresholds and/or corresponding outputs may be one or more factors such as user settings, contact motion, input timing, application execution, speed at which the intensity is applied, number of simultaneous inputs, user It varies based on history, environmental factors (eg, ambient noise), focus selector position, etc. Exemplary factors are described in US Patent Application Nos. 14/399,606 and 14/624,296, which are incorporated herein by reference in their entirety.

For example, FIG. 4C illustrates a dynamic intensity threshold 480 that changes over time based in part on the intensity of the touch input 476 over time. The dynamic intensity threshold 480 has two components, a first component 474 that decays over time after a predefined delay time p1 from when the touch input 476 is initially detected, and a time It is the sum of the second components 478 that follow the intensity of the touch input 476 over time. The initial high intensity threshold of the first component 474 reduces accidental triggering of a "press deep" response, but still allows an immediate "press deep" response if the touch input 476 provides sufficient intensity. The second component 478 reduces unintentional triggering of a "press deep" response by gradual fluctuations in intensity of the touch input. In some embodiments, when the touch input 476 meets the dynamic intensity threshold 480 (eg, at point 481 in FIG. 4C ), a “press deep” response is triggered.

4D illustrates another dynamic intensity threshold 486 (eg, intensity threshold I _D ). 4D also illustrates other two intensity thresholds, namely a first intensity threshold I _H and a second intensity threshold I _L. In FIG. 4D, the touch input 484 satisfies the first intensity threshold (I _H ) and the second intensity threshold (I _L ) before time p2, but the delay time (p2) passes through time 482. Until no response is provided. Also in Figure 4d, the dynamic intensity threshold 486 decays over time, with the decay being a predefined delay time p1 from time 482 (when the response associated with the second intensity threshold I _L is triggered). ) Starts at time 488 after elapsed. This type of dynamic intensity threshold is immediately after triggering a response associated with a lower intensity threshold, for example a first intensity threshold (I _H ) or a second intensity threshold (I _L ), or at the same time as the dynamic intensity threshold (I _D ) Reduce accidental triggering of the response associated with

4E illustrates another dynamic intensity threshold 492 (eg, intensity threshold I _D ). In FIG. 4E, the response associated with the intensity threshold I _L is triggered after a delay time p2 elapses from when the touch input 490 is initially detected. At the same time, the dynamic intensity threshold 492 attenuates after a predefined delay time p1 elapses from when the touch input 490 is initially detected. Therefore, when the intensity of the touch input 490 decreases after triggering the response associated with the intensity threshold I _L , and then the intensity of the touch input 490 increases without releasing the touch input 490, the touch input 490 A response associated with the intensity threshold I _D may be triggered (eg, at time 494) even when the intensity of) is less than another intensity threshold, eg, intensity threshold I _L.

Increase in the light press intensity threshold (IT _L) characteristics of the contact as between the pressing depth and lightly press intensity threshold (IT _L) from the intensity of the less intensity threshold (IT _D) intensity strength is sometimes referred to as "tap tap" input do. Deep press intensity threshold (IT _D) from the intensity of the pressing depth is less than the threshold intensity (IT _D) increases in the characteristics of the contact to the intensity of the intensity than is sometimes referred to as "deep press" type. Contact-detection intensity threshold (IT ₀₎ from the strength of the under contact-increasing properties of the contact of the century between detected intensity threshold (IT ₀₎ and lightly pressing intensity threshold (IT _L) intensity, and sometimes the contact on the touch surface Is referred to as detecting. Contact detection intensity threshold (IT ₀₎ contact detection intensity threshold value from the intensity of the excess (IT ₀₎ characteristic decrease in the strength of contact with the under strength is sometimes referred to as for detecting the lift-off of the touch from the touch surface. In some embodiments IT ₀ is 0 (zero). In some embodiments IT ₀ is greater than 0. In some embodiments, a shaded circle or ellipse is used to indicate the intensity of contact on the touch-sensitive surface. In some embodiments, an unshaded circle or ellipse is used to represent an individual contact on the touch-sensitive surface without specifying the intensity of the individual contact.

In some embodiments described herein, the one or more actions detect each pressing input performed with each contact (or multiple contacts) in response to detecting a gesture including each pressing input. Wherein each pressing input is detected based at least in part on detecting an increase in the intensity of a contact (or a plurality of contacts) above a pressing input intensity threshold. In some embodiments, the individual action is performed in response to detecting an increase in the intensity of the individual contact above the pressing input intensity threshold (eg, the individual action is a “down stroke” of the individual pressing input. )"). In some embodiments, the pressing input comprises an increase in the intensity of each contact above the pressing input intensity threshold and a subsequent decrease in the intensity of the contact below the pressing input intensity threshold, each action being each less than the pressing input intensity threshold. Is performed in response to detecting a subsequent decrease in the intensity of the contact (eg, each action is performed in the “up stroke” of each pressing input).

In some embodiments, the device employs an intensity hysteresis to avoid accidental inputs, sometimes referred to as “jitter,” where the device has a hysteresis intensity threshold ( For example, the hysteresis intensity threshold is an X intensity unit that is lower than the pressing input intensity threshold, or the hysteresis intensity threshold is 75%, 90% or any suitable ratio of the pressing input intensity threshold). As such, in some embodiments, the pressing input comprises an increase in the intensity of each contact above the pressing input intensity threshold and a subsequent decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the pressing input intensity threshold, each Is performed in response to detecting a subsequent decrease in the intensity of each contact below the hysteresis intensity threshold (each action is performed in the "up stroke" of each pressing input). Similarly, in some embodiments, the pressing input is an increase in the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the pressing input intensity threshold, and optionally, to an intensity below the hysteresis intensity. It is detected only when detecting a subsequent decrease in the intensity of the contact, and each operation is performed in response to detecting a pressing input (eg, an increase in the intensity of the contact or a decrease in the intensity of the contact according to the surrounding environment).

For convenience of explanation, the description of operations performed in response to a pressing input associated with a pressing input intensity threshold or in response to a gesture including a pressing input may optionally include an increase in the intensity of the contact exceeding the pressing input intensity threshold, and a hysteresis intensity threshold. An increase in the intensity of the contact from an intensity less than to an intensity above the pressing input intensity threshold, a decrease in the intensity of the contact below the pressing input intensity threshold, or a decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the pressing input intensity threshold. Triggered in response to detecting. Further, in examples in which the operation is described as being performed in response to detecting a decrease in the intensity of the contact below the pressing input intensity threshold, the action is, optionally, less than a hysteresis intensity threshold corresponding to and lower than the pressing input intensity threshold. Is performed in response to detecting a decrease in the intensity of the contact. As described above, in some embodiments, triggering of these responses also results in time-based criteria being met (e.g., a delay time elapses between the first intensity threshold being met and the second intensity threshold being met). It depends.

As used in the specification and claims, the term “tactile output” refers to the physical displacement of the device relative to its previous position, a component of the device (eg, a touch-sensitive surface) relative to another component of the device (eg, a housing). ), or the displacement of the component with respect to the center of mass of the device to be detected by the user using the user's tactile sense. For example, in a situation in which a device or a component of a device is in contact with a touch-sensitive user's surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement is caused by the user It will be interpreted as a tactile sensation corresponding to a perceived change in the physical properties of a component of the device. For example, movement of a touch-sensitive surface (eg, a touch-sensitive display or trackpad) is, optionally, interpreted by the user as a “down click” or “up click” of a physical actuator button. In some cases, the user will feel a tactile sensation such as a "down click" or "up click" even when there is no movement of a physical actuator button associated with a touch-sensitive surface that is physically depressed (e.g., displaced) by the user's movement. . As another example, movement of the touch-sensitive surface is, optionally, interpreted or sensed by the user as the "roughness" of the touch-sensitive surface, even when there is no change in the flatness of the touch-sensitive surface. While these interpretations of touch by the user will be susceptible to the user's individualized sensory perception, there are many touch sensory perceptions that are common to the majority of users. Thus, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "click up", "click down", "roughness"), unless otherwise stated, the generated tactile output is typical (or It corresponds to the physical displacement of the device or its component that will produce the described sensory perception for the user (average). Providing haptic feedback to the user using tactile outputs improves the operability of the device (e.g., by helping the user provide appropriate inputs and reducing user mistakes when actuating/interacting with the device) and It makes the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, the tactile output pattern determines the characteristics of the tactile output, such as the amplitude of the tactile output, the shape of the moving waveform of the tactile output, the frequency of the tactile output, and/or the duration of the tactile output. To be specified.

When tactile outputs with different tactile output patterns are generated by the device (e.g., via one or more tactile output generators that generate tactile outputs by moving a movable mass), the tactile outputs are When holding or touching, it can evoke different haptic sensations. The user's sense is based on user perception of the tactile output, but most users will be able to discern changes in the waveform, frequency, and amplitude of the tactile outputs generated by the device. Thus, the waveform, frequency and amplitude can be adjusted to indicate to the user that different operations have been performed. As such, a given environment (e.g., a user interface including graphical features and objects, a simulated physical environment with virtual boundaries and virtual objects, a real physical environment with physical boundaries and physical objects, and/or any of the above) Properties of objects (eg size, material, weight, stiffness, smoothness, etc.) in a combination of Behaviors (eg, oscillation, displacement, acceleration, rotation, expansion, etc.); And/or tactile outputs with tactile output patterns designed and/or selected and/or engineered to simulate interactions (e.g., collision, adhesion, repulsion, attraction, friction, etc.), in some situations. , It will provide useful feedback to users that reduces input errors and increases the efficiency of user operation of the device. Additionally, tactile outputs are optionally generated to correspond to a simulated physical characteristic, such as an input threshold or feedback that is not related to the selection of an object. Such tactile outputs will, in some situations, provide useful feedback to users that reduce input errors and increase the efficiency of user operation of the device.

In some embodiments, a tactile output with a suitable tactile output pattern serves as a cue for the occurrence of an event of interest in the user interface or behind a scene within the device. Examples of events of interest include activation of an affordance (e.g., a real or virtual button, or toggle switch) provided on the device or in the user interface, the success or failure of a requested action, reaching or reaching a boundary in the user interface. Traversing, entering a new state, switching input focus between objects, activating a new mode, reaching or traversing an input threshold, detecting or recognizing the type of input or gesture, and the like. In some embodiments, tactile outputs are provided to serve as a warning or alert for an impending event or outcome that will occur if a redirection or interruption input is not detected in a timely manner. Tactile outputs also enrich the user experience, in other contexts, and/or improve the accessibility of the device to users with visual impairment or motor neurological impairment or other accessibility needs, and/or user interface and/or Or it is used to improve the efficiency and function of the device. The tactile outputs optionally involve audio outputs and/or visible user interface changes, which further enhances the user's experience when the user interacts with the user interface and/or device, and enhances the user interface and/or Or to facilitate better transfer of information about the state of the device, reduce input errors and increase the efficiency of user operation of the device.

4F-4H illustrate one or more haptic feedbacks suitable for various purposes and in various scenarios, such as those described above and those described in connection with the user interfaces and methods discussed herein. It provides a set of sample tactile output patterns that can be used as is, individually or in combination, through transformations (eg, modulation, amplification, truncation, etc.). This example of a palette of tactile outputs shows how a set of three waveforms and eight frequencies can be used to create an array of tactile output patterns. In addition to the tactile output patterns shown in these figures, each of these tactile output patterns is optionally, for example, Fig. 4i, shown as variants with gains of 1.0, 0.75, 0.5, and 0.25, respectively. As shown for the pull tap 80 Hz, the pull tap 200 Hz, the minitap 80 Hz, the mini tap 200 Hz, the micro tap 80 Hz, and the micro tap 200 Hz in FIG. 4K, tactile output The amplitude is adjusted by changing the gain value for the pattern. 4I to 4K, changing the gain of the tactile output pattern changes the amplitude of the pattern without changing the frequency of the pattern or changing the shape of the waveform. In some embodiments, changing the frequency of the tactile output pattern also results in a lower amplitude, which some tactile output generators are limited by how much force can be applied to the movable mass, and thus the waveform This is because the higher frequency movements of the mass are constrained to lower amplitudes (e.g., 230 Hz, 270 Hz, And the peak amplitudes of the pull tap at 300 Hz are lower than the amplitudes of the pull tap at 80 Hz, 100 Hz, 125 Hz, and 200 Hz).

4F to 4K show tactile output patterns having a specific waveform. The waveform of the tactile output pattern represents the pattern of physical displacements with respect to a neutral position (eg, xzero) versus time that the movable mass goes through to generate a tactile output with that tactile output pattern. For example, the first set of tactile output patterns (e.g., "pull-tap" tactile output patterns) shown in FIG. 4F is an oscillation with 2 complete cycles each (e.g., starting and ending in a neutral position and being neutral). Oscillations traversing the location three times). The second set of tactile output patterns (e.g., "minitap" tactile output patterns) shown in FIG. 4G are oscillations each comprising one complete cycle (e.g., starting and ending in a neutral position, with 1 neutral position). Oscillations). The third set of tactile output patterns (e.g., "microtap" tactile output patterns) shown in FIG. 4H are oscillations (e.g., starting and ending in a neutral position, each containing 1/2 of a complete cycle) Oscillations that do not cross the location) The waveform of the tactile output pattern also includes a start buffer and an end buffer representing the gradual acceleration and deceleration of the movable mass at the beginning and end of the tactile output. The exemplary waveforms shown in FIGS. 4F-4K include xmin and xmax values representing the maximum and minimum limits of movement of the movable mass. In the case of larger electronic devices with larger movable masses, there may be larger or smaller minimum and maximum limits of movement of the mass. The examples shown in FIGS. 4F to 4K describe the movement of a mass in one dimension, but similar principles will also apply to the movement of a movable mass in two or three dimensions.

As shown in Figures 4F-4K, each tactile output pattern also has a corresponding characteristic frequency that affects the "pitch" of the haptic sensation, which haptic sensation is the user from the tactile output with that characteristic frequency. Felt by For continuous tactile output, the characteristic frequency represents the number of cycles (eg, cycles per second) completed within a given period by the movable mass of the tactile output generator. In the case of an individual tactile output, a separate output signal (e.g., having 0.5, 1, or 2 cycles) is generated and the characteristic frequency value is how much the movable mass is to produce a tactile output with that characteristic frequency. Specify whether you need to move quickly. As shown in Figures 4F-4H, for each type of tactile output (e.g., as defined by each waveform, such as pull tap, minitap, or microtap), the higher frequency value shifts. The faster movement(s) by the possible mass, and thus, in general, a shorter time to complete the tactile output (e.g., the time to complete the required number of cycle(s) for an individual tactile output + Corresponding to the start and end buffer times). For example, a pull tap having a characteristic frequency of 80 Hz takes longer to complete than a pull tap having a characteristic frequency of 100 Hz (e.g., 28.3 ms compared to 35.4 ms in FIG. 4F). Also, for given frequencies, a tactile output with more cycles of its waveform at each frequency takes longer to complete than a tactile output with fewer cycles of its waveform at the same each frequency. For example, a pull tap at 150 Hz takes longer to complete than a minitap at 150 Hz (e.g., 12.8 ms versus 19.4 ms), and a minitap at 150 Hz takes longer to complete than a microtap at 150 Hz. It takes (e.g. 9.4ms versus 12.8ms). However, for tactile output patterns with different frequencies, this rule may not apply (e.g., tactile outputs with more cycles but higher frequency have fewer cycles but tactile output patterns with lower frequency). It may take less time to complete than outputs, and vice versa). For example, at 300 Hz, a pull tap takes as long as a mini tap (eg, 9.9 ms).

As shown in Figs. 4F to 4K, the tactile output pattern also has a characteristic amplitude that affects the amount of energy contained in the tactile signal or the "intensity" of the haptic sensation, which haptic sensation has its characteristic amplitude Can be felt by the user through the tactile output with. In some embodiments, the characteristic amplitude of a tactile output pattern refers to an absolute or normalized value representing the maximum displacement of a movable mass from a neutral position when generating a tactile output. In some embodiments, the characteristic amplitude of the tactile output pattern is determined by various conditions (e.g., customized based on user interface situations and behaviors) and/or preconfigured metrics (e.g., input-based metrics, and /Or user interface based metrics), eg, by a fixed or dynamically determined gain factor (eg, a value of 0 to 1). In some embodiments, an input-based metric (e.g., an intensity change metric or an input-velocity metric) is a characteristic of the input (e.g., the rate of change of the characteristic intensity of the touch at the pressing input or the touch The speed of movement of the contact across the sensitive surface is measured. In some embodiments, a user interface based metric (e.g., a speed across a boundary metric) is a characteristic of a user interface element (e.g., a hidden or visible boundary in the user interface) during a user interface change that triggers the generation of a tactile output. The speed of movement of the element across it is measured. In some embodiments, the characteristic amplitude of the tactile output pattern may be modulated by a “envelope”, and the peaks of adjacent cycles may have different amplitudes, wherein one of the previously shown waveforms is When the output is being generated, it is further modified by multiplication by an envelope parameter that changes over time (eg, between 0 and 1) to gradually adjust the amplitude of the portions of the tactile output over time.

Only certain frequencies, amplitudes and waveforms are represented by the sample tactile output patterns in FIGS. 4F-4K for illustrative purposes, but tactile output patterns with different frequencies, amplitudes and waveforms are for similar purposes. Can be used. For example, waveforms with 0.5 to 4 cycles can be used. Other frequencies in the range of 60 Hz to 400 Hz may also be used.

User interfaces and associated processes

Now, having a display, a touch-sensitive surface, (optionally) one or more tactile output generators to generate tactile outputs, and (optionally) one or more sensors to detect the intensities of contacts with the touch-sensitive surface. Attention is drawn to embodiments of user interfaces (“UI”) and associated processes that may be implemented on an electronic device such as portable multifunction device 100 or device 300.

5A-5A illustrate exemplary user interfaces for displaying a representation of a virtual object while switching from displaying a first user interface area to displaying a second user interface area, according to some embodiments. . User interfaces in these figures are in FIGS. 8A to 8E, 9A to 9D, 10A to 10D, 16A to 16G, 17A to 17D, 18A to 18I, 19A to 19H, and FIG. It is used to illustrate the processes described below, including the processes of FIGS. 20A-20F. For ease of description, some of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is, optionally: an individual finger or stylus contact, or a representative point corresponding to the finger or stylus contact (e.g., the center of the individual contact, or a point associated with the individual contact), or It is the center of two or more contacts detected on the touch-sensitive display system 112. Optionally, however, similar operations may be performed separately from the display 450 in response to detecting contacts on the touch-sensitive surface 451 while displaying the user interfaces shown in the figure with the focus selector on the display 450. It is performed on a device with a touch-sensitive surface 451 of.

5A illustrates a real-world context in which the user interfaces described in connection with FIGS. 5B-5A are used.

5A illustrates a physical space 5002 in which the table 5004 is located. The device 100 is held in the user's hand 5006 by the user.

5B illustrates a messaging user interface 5008 displayed on display 112. The messaging user interface 5008 includes a message speech bubble 5010 including a received text message 5012, a message speech balloon 5014 including a transmitted text message 5016, and a virtual object (e.g., a virtual object) received in the message. Chair 5020) and a virtual object indicator 5022 indicating that the virtual chair 5020 is an observable object in an augmented reality view (e.g., within a representation of the field of view of one or more cameras of the device 100) It includes a speech balloon 5018. The messaging user interface 5008 also includes a message input area 5024 configured to display message input.

5C-5G illustrate an input that causes a portion of the messaging user interface 5008 to be replaced with the field of view of one or more cameras of the device 100. In FIG. 5C, contact 5026 of the device 100 with the touch screen 112 is detected. The characteristic strength of the contact is _greater than the contact detection intensity threshold IT ₀ and less than the hint press intensity threshold IT _H , as illustrated by the intensity level meter 5028. In FIG. 5D, an increase in the characteristic intensity of the contact 5026 exceeding the hint pressing intensity threshold IT _H as exemplified by the intensity level meter 5028 causes the area of the message speech bubble 5018 to increase, and the virtual The size of the chair 5020 is increased, and the messaging user interface 5008 begins to blur behind the message balloon 5018 (e.g., to provide the user with visual feedback of the effect of increasing the characteristic strength of the contact). I did. In FIG. 5E, the increase in the characteristic intensity of the contact 5026 above the light press intensity threshold IT _L as illustrated by the intensity level meter 5028 indicates that the message balloon 5018 is a platter. ) (5030), the size of the virtual chair 5020 is further increased, and the blurring of the messaging user interface 5008 behind the platter 5030 is increased. In FIG. 5F, an increase in the characteristic strength of the contact 5026 above the deep press intensity threshold IT _D as illustrated by the intensity level meter 5028 causes a portion of the messaging user interface 5008 to be increased. The tactile output generators 167 of the device 100 output a tactile output (as illustrated at 5032) to indicate that the criteria for replacing the field of view of one or more cameras of the device 100 have been met. do.

In some embodiments, the progress shown in FIGS. 5C-5E is reversible before the characteristic strength of the contact 5026 reaches the deep press strength threshold IT _D as shown in FIG. 5F. For example, reducing the characteristic intensity of contact 5026 after the increases illustrated in FIGS. 5D and/or 5E will cause the interface state corresponding to the reduced intensity level of contact 5026 to be displayed (e.g. , The interface as shown in FIG. 5e is shown according to the determination that the reduced characteristic strength of the contact exceeds the light pressing intensity threshold (IT _L ), and the interface as shown in FIG. 5d hints at the reduced characteristic strength of the contact. The interface as shown in FIG. 5C is shown according to the determination that the push strength threshold IT _H is exceeded, and the interface as shown in FIG. 5C is shown according to the determination that the reduced characteristic strength of the contact is less than the hint push strength threshold IT _H In some embodiments, reducing the characteristic strength of contact 5026 after the increases illustrated in FIGS. 5D and/or 5E will cause the interface as shown in FIG. 5C to be redisplayed.

5F-5J illustrate an animated transition, in which a portion of the messaging user interface is replaced with the field of view of one or more cameras (hereinafter “camera(s)”) of device 100 during the animated transition. 5F to 5G, the contact 5026 has been lifted off from the touch screen 112 and the virtual chair 5020 has rotated toward its final position in Fig. 5I. In FIG. 5G, the field of view 5034 of the camera(s) has started to fade into view within the platter 5030 (as indicated by dashed lines). In FIG.5H, the field of view 5034 of the camera(s) (e.g., showing a view of the physical space 5002 as captured by the camera(s)) completes fading into view within the platter 5030. I did. 5H to 5I, the virtual chair 5020 continues its rotation toward its final position in FIG. 5I. 5I, tactile output generators 167 are illustrated at 5036 to indicate that at least one plane (e.g., floor surface 5038) has been detected within the field of view 5034 of the camera(s). Same) and tactile output. Virtual chair 5020 is placed on the detected plane (eg, according to a determination by device 100 that the virtual object is configured to be placed in an upright orientation on a detected horizontal surface, such as floor surface 5038 ). The size of the virtual chair 5020 is continuously adjusted on the display 112 as a portion of the messaging user interface is converted into a representation of the field of view 5034 of the camera(s) on the display 112. For example, the scale of the virtual chair 5020 relative to the physical space 5002 as shown in the field of view 5034 of the camera(s) is the virtual chair 5020 within the field of view 5034 of the camera(s). And/or the detected size of objects (such as table 5004). In FIG. 5J, the virtual chair 5020 is displayed in its final position with a predefined orientation relative to the detected floor surface within the field of view 5034 of the camera(s). In some embodiments, the initial landing position of the virtual chair 5020 is a predefined position relative to the detected plane within the field of view of the camera(s), such as the center of an unoccupied area of the detected plane. In some embodiments, the initial landing position of the virtual chair 5020 is determined according to the lift-off position of the contact 5026 (e.g., the lift-off position of the contact 5026 is based on the criteria for transitioning to the augmented reality environment). After being satisfied in 5f, it may be different from the initial touchdown position of contact 5026 due to movement of contact 5026 across touch screen 112).

5K and 5L illustrate movement (eg, by the user's hand 5006) of device 100 to adjust the field of view 5034 of the camera(s). As the device 100 is moved relative to the physical space 5002, the displayed field of view 5034 of the camera(s) changes, and the virtual chair 5020 is the floor within the displayed field of view 5034 of the camera(s). It remains attached to the surface 5038 in the same position and in the same orientation.

5M-5Q illustrate inputs that cause movement of the virtual chair 5020 across the floor surface 5038 within the displayed field of view 5034 of the camera(s). In FIG. 5N, a contact 5040 with the touch screen 112 of the device 100 is detected at a position corresponding to the virtual chair 5020. 5N and 5O, virtual chair 5020 is dragged by contact 5040 as contact 5040 moves along the path indicated by arrow 5042. As the virtual chair 5020 is moved by the contact 5040, the size of the virtual chair 5020 is the virtual chair 5020 for the physical space 5002 as shown in the field of view 5034 of the camera(s). Is changed to maintain the scale For example, in FIGS. 5N-5P, the virtual chair 5020 is further from the device 100 from the foreground of the field of view 5034 of the camera(s) and the table 5004 within the field of view 5034 of the camera(s). ), the size of the virtual chair 5020 decreases (eg, so that the scale of the chair relative to the table 5004 in the field of view 5034 of the camera(s) is maintained). Moreover, as the virtual chair 5020 is moved by the contact 5040, the planes identified within the field of view 5034 of the camera(s) are highlighted. For example, the floor plane 5038 is highlighted in FIG. 5O. 5O and 5P, virtual chair 5020 continues to be dragged by contact 5040 as contact 5040 moves along the path indicated by arrow 5044. In FIG. 5Q, the contact 5040 has been lifted off from the touch screen 112. In some embodiments, as shown in FIGS. 5N-5Q, as the virtual chair 5020 is dragged across the floor surface 5038 by the contact 5040, the movement path of the virtual chair 5020 is Constrained by the floor surface 5038 within the camera's field of view 5034. In some embodiments, contact 5040 as described in connection with FIGS. 5N-5P is a continuation of contact 5026 as described in connection with FIGS. 5C-5F (e.g., There is no liftoff, the same contact that causes a portion of the messaging user interface 5008 to be replaced with the field of view 5034 of the camera(s) also drags the virtual chair 5020 within the field of view 5034 of the camera(s). ).

5Q-5U show inputs that cause movement of the virtual chair 5020 from the floor surface 5038 to a different plane (e.g., table surface 5046) detected within the field of view 5034 of the camera(s). Illustrate. In FIG. 5R, a contact 5050 with the touch screen 112 of the device 100 is detected at a position corresponding to the virtual chair 5020. 5R and 5S, as the contact 5048 moves along the path indicated by the arrow 5050, the virtual chair 5020 is dragged by the contact 5048. As the virtual chair 5020 is moved by the contact 5048, the size of the virtual chair 5020 is the virtual chair 5020 for the physical space 5002 as shown in the field of view 5034 of the camera(s). Is changed to keep the scale of. Additionally, as virtual chair 5020 is moved by contact 5040, table surface plane 5046 is emphasized (eg, as shown in FIG. 5S). 5S and 5T, virtual chair 5020 continues to be dragged by contact 5040 as contact 5048 moves along the path indicated by arrow 5052. In FIG. 5U, the contact 5048 has been lifted off from the touch screen 112 and the virtual chair 5020 is placed on the table surface plane 5046 in an upright orientation facing the same direction as before.

5u-5ad illustrate an input dragging the virtual chair 5020 to the edge of the touch screen display 112, causing the field of view 5034 of the camera(s) to stop being displayed. In FIG. 5V, a contact 5054 of the device 100 with the touch screen 112 is detected at a position corresponding to the virtual chair 5020. In FIGS. 5V and 5W, as the contact 5054 moves along the path indicated by the arrow 5056, the virtual chair 5020 is dragged by the contact 5054. 5W and 5X, as the contact 5054 moves along the path indicated by the arrow 5058, the virtual chair 5020 continues to be dragged by the contact 5054 to the position shown in FIG. 5X.

The input by contact 5054 illustrated in FIGS. 5U-5X is from displaying the field of view 5034 of the camera(s) within the platter 5030, as shown in FIGS. 5Y-5A. ) To stop displaying the field of view 5034 and return to fully displaying the messaging user interface 5008. In FIG. 5Y, the field of view 5034 of the camera(s) begins to fade out within the platter 5030. In Figures 5Y and 5Z, the platter 5030 is transitioned to the message balloon 5018. In Figure 5z, the field of view 5034 of the camera(s) is no longer displayed. In FIG. 5AA, the messaging user interface 5008 stops blurring and the size of the message balloon 5018 returns to the original size of the message balloon 5018 (eg, as shown in FIG. 5B).

5AA-5A shows the original position of the virtual chair 5020 in the messaging user interface 5008 (e.g., as shown in FIG. 5B) from the location where the virtual chair 5020 corresponds to the contact 5054 in FIG. 5AA. An animated transition of the virtual chair 5020 that occurs as it moves to is illustrated. In FIG. 5A, the contact 5054 has been lifted off from the touch screen 112. In Figs. 5A and 5A, the virtual chair 5020 gradually increases in size and rotates toward its final position in Fig. 5A.

5B-5A, the virtual chair 5020 has a substantially identical three-dimensional appearance within the messaging user interface 5008 and within the displayed field of view 5034 of the camera(s), and the virtual chair 5020 is Maintains the same three-dimensional appearance during the transition from displaying the messaging user interface 5008 to displaying the field of view 5034 of the camera(s) and during the reverse transition. In some embodiments, the representation of virtual chair 5020 has a different appearance in an application user interface (eg, a messaging user interface) than in an augmented reality environment (eg, the displayed field of view of the camera(s)). For example, the virtual chair 5020 optionally has a three-dimensional, more realistic and textured look in an augmented reality environment while optionally having a two-dimensional or more stylized look in the application user interface; During the transition between displaying the application user interface and displaying the augmented reality environment, the intermediate appearances of the virtual chair 5020 are a series of interpolated appearances between the two-dimensional and three-dimensional appearances of the virtual chair 5020.

5AE illustrates an Internet browser user interface 5060. The Internet browser user interface 5060 includes a URL/search input area 5062 configured to display a URL/search input for a web browser and browser controls 5064 (e.g., a navigation control unit including a back button and a forward button. Field, a sharing control unit for displaying a shared interface, a bookmark control unit for displaying a bookmark interface, and a tap control unit for displaying a tab interface). Internet browser user interface 5060 also includes web objects 5066, 5068, 5070, 5072, 5074, 5076. In some embodiments, each web object includes a link to cause the linked internet location corresponding to the web object to be displayed in the internet browser user interface 5060 in response to a tap input on each web object (e.g. , Replacing the display of each web object). Web objects 5066, 5068, 5072 include two-dimensional representations of three-dimensional virtual objects, as indicated by virtual object indicators 5078, 5080, and 5082, respectively. Web objects 5070, 5074, 5076 contain two-dimensional images (however, two-dimensional images of web objects 5070, 5074, 5076, as indicated by no virtual object indicators, are three-dimensional virtual Does not correspond to objects). The virtual object corresponding to the web object 5068 is a lamp object 5084.

5af-5ah illustrate an input that causes a portion of the internet browser user interface 5060 to be replaced with the field of view 5034 of the camera(s). In FIG. 5Af, a contact 5086 with the touch screen 112 of the device 100 is detected. The characteristic strength of the contact, as illustrated by the intensity level meter 5028, is above the contact detection intensity threshold IT ₀ and less than the hint press intensity threshold IT _H. 5AG, an increase in the characteristic intensity of the contact 5026 above the tapping intensity threshold IT _L as illustrated by the intensity level meter 5028 means that the field of view 5034 of the camera(s) (e.g., Displayed in web object 5068 (overlayed by virtual lamp 5084). 5Ah, the increase in the characteristic intensity of the contact 5086 that exceeds the deep pressing intensity threshold IT _D as exemplified by the intensity level meter 5028 indicates that the field of view 5034 of the camera(s) is To replace a larger portion of the interface 5060 (e.g., leaving only the URL/search input area 5062 and browser controls 5064), the tactile output generators 167 of the device 100 Outputs a tactile output (as illustrated at 5088) to indicate that the criteria for replacing a portion of the browser user interface 5060 with the field of view 5034 of the camera(s) have been met. In some embodiments, in response to input described in connection with FIGS. 5A-5A, the field of view 5034 of the camera(s) completely replaces the Internet browser user interface 506 on the touch screen display 112.

5Ai-5A illustrate the input that causes the movement of the virtual ramp 5084. 5Ai and 5Aj, as contact 5086 moves along the path indicated by arrow 5090, virtual ramp 5084 is dragged by contact 5086. As the virtual lamp 5084 is moved by the contact 5086, the size of the virtual lamp 5084 does not change, and the path of the virtual lamp 5084 is optionally captured within the field of view of the camera(s). It is not limited by the structure of the space. As the virtual lamp 5084 is moved by the contact 5086, the planes identified within the field of view 5034 of the camera(s) are highlighted. For example, the floor plane 5038 is highlighted in FIG. 5AJ as the imaginary ramp 5084 moves over the floor plane 5038. 5AJ and 5AK, as the contact 5086 moves along the path indicated by the arrow 5092, the virtual ramp 5084 continues to be dragged by the contact 5086. In Figures 5A and 5A, as the contact 5086 moves along the path indicated by the arrow 5094, the virtual ramp 5084 continues to be dragged by the contact 5086, and the floor plane 5038 is highlighted. When stopped, the table surface 5046 is highlighted as the virtual lamp 5084 moves over the table 5004. In FIG. 5A, contact 5086 has been lifted off from touch screen 112. When the contact 5086 is lifted off, the size of the virtual lamp 5086 is adjusted to have the correct scale for the table 5004 within the field of view 5034 of the camera(s), and the virtual lamp 5086 is It is placed in an upright orientation on the table surface 5046 in 5034.

Figures 5am-5aq show the virtual lamp 5084 to the edge of the touch screen display 112, causing the field of view 5034 of the camera(s) to stop displaying and the internet browser user interface 5060 to be restored. Illustrates the dragging input. In FIG. 5A, a contact 5096 with the touch screen 112 of the device 100 is detected at a location corresponding to the virtual lamp 5084. In FIGS. 5A and 5Ao, as contact 5096 moves along the path indicated by arrow 5098, virtual ramp 5084 is dragged by contact 5096. 5AO and 5AP, as contact 5054 moves along the path indicated by arrow 5100, virtual ramp 5084 continues to be dragged by contact 5096 to the location shown in FIG. 5AP. In Figure 5aq, the contact 5096 has been lifted off from the touch screen 112.

The input by contact 5096 illustrated in FIGS. 5A-5A is the field of view 5034 of the camera(s) from displaying the field of view 5034 of the camera(s), as shown in FIGS. ) To stop displaying and return to fully displaying the Internet browser user interface 5060. In Fig. 5A, the field of view 5034 of the camera(s) begins to fade out (as indicated by dashed lines). 5A-5A, the virtual ramp 5084 increases in size and moves toward its original position within the Internet browser user interface 5060. In FIG. 5A, the field of view 5034 of the camera(s) is no longer displayed, and the Internet browser user interface 5060 begins to fade in (as indicated by dashed lines). In FIG. 5A, the Internet browser user interface 5060 is fully displayed, and the virtual lamp 5084 has returned to its original size and position within the Internet browser user interface 5060.

6A-6AJ illustrate a first representation of a virtual object in a first user interface area, a second representation of a virtual object in a second user interface area, and a representation of a field of view of one or more cameras, according to some embodiments. Together illustrate example user interfaces for displaying a third representation of a virtual object. User interfaces in these figures are in FIGS. 8A to 8E, 9A to 9D, 10A to 10D, 16A to 16G, 17A to 17D, 18A to 18I, 19A to 19H, and FIG. It is used to illustrate the processes described below, including the processes of FIGS. 20A-20F. For ease of description, some of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is, optionally: an individual finger or stylus contact, or a representative point corresponding to the finger or stylus contact (e.g., the center of the individual contact, or a point associated with the individual contact), or It is the center of two or more contacts detected on the touch-sensitive display system 112. However, similar operations may be performed in response to detecting contacts on the touch-sensitive surface 451, optionally displaying the user interfaces shown in the figure with the focus selector on the display 450, and It is performed on a device with a separate touch-sensitive surface 451.

6A shows a message balloon 5010 including a received text message 5012, a message balloon 5014 including a transmitted text message 5016, and a virtual object (e.g., a virtual chair 5020) received in the message. ) And a virtual object indicator 5022 indicating that the virtual chair 5020 is an observable object in the augmented reality view (e.g., within the displayed field of view of one or more cameras of the device 100). Illustrative messaging user interface 5008 including. The messaging user interface 5008 is described in more detail with respect to FIG. 5B.

6B and 6C illustrate an input that causes rotation of the virtual chair 5020. In FIG. 6B, contact 6002 of device 100 with touch screen 112 is detected. The contact 6002 moves across the touch screen 112 along the path indicated by the arrow 6004. In FIG. 6C, in response to the movement of the contact, the messaging user interface 5008 is scrolled upward (which causes the message balloon 5010 to scroll off the display, the message balloon 5014, 5018 to scroll upward, and The additional message balloon 6005 is revealed), and the virtual chair 5020 is rotated (eg, tilted upwards). The magnitude and direction of rotation of the virtual chair 5020 corresponds to the movement of the contact 6002 along the path indicated by the arrow 6004. In FIG. 6D, the contact 6002 has been lifted off from the touch screen 112. In some embodiments, this rotational behavior of the virtual chair 5020 within the message balloon 5018 allows the virtual chair 5020 to be observable in an augmented reality environment that includes the field of view of the camera(s) of the device 100. It is used as an indication of a virtual object.

6E-6L illustrate inputs that cause messaging user interface 5008 to be replaced with staging user interface 6010 and subsequently change the orientation of virtual chair 5020. In FIG. 6E, a contact 6006 with the touch screen 112 of the device 100 is detected. The characteristic strength of the contact, as illustrated by the intensity level meter 5028, is above the contact detection intensity threshold IT ₀ and less than the hint press intensity threshold IT _H. In FIG. 6F, the increase in the characteristic intensity of the contact 6006 exceeding the hint pressing intensity threshold IT _H as exemplified by the intensity level meter 5028 causes the area of the message speech bubble 5018 to increase. The size of the chair 5020 is increased, and the messaging user interface 5008 begins to blur behind the message balloon 5018 (e.g., to provide the user with visual feedback of the effect of increasing the characteristic strength of the contact). I did. In FIG. 6G, an increase in the characteristic strength of the contact 6006 above the tapping intensity threshold IT _L as illustrated by the intensity level meter 5028 causes the message speech bubble 5018 to be replaced by the platter 6008. In addition, the size of the virtual chair 5020 was further increased, and the blurring of the messaging user interface 5008 behind the platter 6008 was increased. In FIG. 6H, an increase in the characteristic strength of the contact 6006 above the deep depress intensity threshold IT _D as illustrated by the intensity level meter 5028 causes the messaging user interface 5008 to stop displaying. Initiate a fade in (indicated by dotted lines) of the staging user interface 6010. Additionally, as shown in FIG. 6H, the increase in the characteristic strength of the contact 6006 that exceeds the deep pressing intensity threshold IT _D is the basis for replacing the messaging user interface 5008 with the staging user interface 6010. The tactile output generators 167 of the device 100 output the tactile output (as illustrated at 6012) to indicate that it has been met.

In some embodiments, the progress shown in FIGS. 6E-6G is reversible before the characteristic intensity of the contact 6006 reaches the deep pressing intensity threshold IT _D as shown in FIG. 6H. For example, decreasing the characteristic intensity of contact 6006 after the increases illustrated in FIGS. 6F and/or 6G will cause the interface state corresponding to the reduced intensity level of contact 6006 to be displayed (e.g. , The interface as shown in FIG. 6G is shown according to the determination that the reduced characteristic strength of the contact exceeds the light pressing intensity threshold (IT _L ), and the interface as shown in FIG. 6F hints at the reduced characteristic strength of the contact. The interface as shown in Fig. 6E is shown in accordance with the determination that the press strength threshold IT _H is exceeded, and the interface as shown in FIG. 6E is shown according to the determination that the reduced characteristic strength of the contact is less than the hint press strength threshold IT _H In some embodiments, reducing the characteristic strength of contact 6006 after the increases illustrated in FIGS. 6F and/or 6G will cause the interface as shown in FIG. 6E to be redisplayed.

In Fig. 6I, a staging user interface 6010 is displayed. The staging user interface 6010 includes a stage 6014 on which a virtual chair 5020 is displayed. 6H to 6I, the virtual chair 5020 is animated to show the transition from the position of the virtual chair 5020 in FIG. 6H to the position of the virtual chair 5020 in FIG. 6I. For example, the virtual chair 5020 is rotated to a predefined position, orientation, and/or distance relative to the stage 6014 (eg, such that the virtual chair appears to be supported by the stage 6014). The staging user interface 6010 also includes a back control 6016, which, when activated (e.g., by tap input at a position corresponding to the back control 6016), previously displayed Causes the user interface (eg, messaging user interface 5008) to be redisplayed. The staging user interface 6010 also indicates the current display mode (e.g., the current display mode is the staging user interface mode, as indicated by the highlighted “3D” indicator) and, when activated, the selected display mode It includes a toggle control 6018 that causes a transition. For example, while the staging user interface 6010 is displayed, a contact at a position corresponding to the toggle control 6018 (eg, a position corresponding to a portion of the toggle control 6018 including the text "World") The tap input by causes the staging user interface 6010 to be replaced with the field of view of the camera(s). The staging user interface 6010 also includes a sharing control 6020 (eg, for displaying a shared interface).

6J-6L illustrate the rotation of virtual chair 5020 relative to stage 6014 caused by movement of contact 6006. 6J and 6K, as the contact 6006 moves along the path indicated by the arrow 6022, the virtual chair 5020 rotates (e.g., about a first axis perpendicular to the movement of the contact 6066). do. 6K and 6L, as the contact 6006 moves along the path indicated by arrow 6024 and subsequently along the path indicated by arrow 6025, the virtual chair 5020 (e.g., contact 6066) Is rotated around a second axis perpendicular to the movement of. In FIG. 6M, contact 6006 has been lifted off from touch screen 112. In some embodiments, as shown in FIGS. 6J-6L, rotation of virtual chair 5020 is constrained by the surface of stage 6014. For example, at least one leg of the virtual chair 5020 remains in contact with the surface of the stage 6014 during rotation(s) of the virtual chair. In some embodiments, the surface of the stage 6014 serves as a frame of reference for free rotation and vertical translation of the virtual chair 5020 without imposing specific constraints on the movement of the virtual chair 5020.

6N and 6O illustrate an input for adjusting the displayed size of the virtual chair 5020. In FIG. 6N, a first contact 6026 and a second contact 6030 with the touch screen 112 are detected. The first contact 6026 moves along the path indicated by the arrow 6028, and simultaneously with the movement of the first contact 6026, the second contact 6030 moves along the path indicated by the arrow 6032. 6N and 6O, as the first contact 6026 and the second contact 6030 move along the paths indicated by arrows 6028 and 6032, respectively (e.g., in a de-pinch gesture), the virtual chair 5020 The displayed size of increases. 6P, the first contact 6030 and the second contact 6026 have been lifted off from the touch screen 112, and the virtual chair 5020 maintains the increased size after the lift-off of the contacts 6026, 6030. do.

6Q-6U illustrate an input that causes the staging user interface 6010 to be replaced with a field of view 6036 of one or more cameras of the device 100. In FIG. 6Q, a contact 6034 of the device 100 with the touch screen 112 is detected. The characteristic strength of the contact, as illustrated by the intensity level meter 5028, is above the contact detection intensity threshold IT ₀ and less than the hint press intensity threshold IT _H. In FIG. 6R, the increase in the characteristic strength of the contact 5026 that exceeds the hint click strength threshold IT _H as illustrated by the intensity level meter 5028 indicates that the staging user interface 6010 is (as indicated by dotted lines). Likewise) it started to blur behind the virtual chair 5020. In FIG. 6S, an increase in the characteristic strength of the contact 6034 above the tapping intensity threshold IT _L as illustrated by the intensity level meter 5028 caused the staging user interface 6010 to stop being displayed. Initiate a fade in (indicated by dashed lines) of the field of view 6036 of the camera(s). In FIG. 6T, an increase in the characteristic strength of the contact 6034 that exceeds the deep pressing intensity threshold IT _D as illustrated by the intensity level meter 5028 causes the field of view 6036 of the camera(s) to be displayed. . Additionally, as shown in FIG. 6T, an increase in the characteristic intensity of the contact 6034 that exceeds the deep pressing intensity threshold IT _D may cause the display of the staging user interface 6010 to be displayed in The tactile output generators 167 of the device 100 output a tactile output (as illustrated at 6038) to indicate that the criteria for replacing with a display have been met. In FIG. 6U, the contact 6034 has been lifted off from the touch screen 112. In some embodiments, the progression shown in FIGS. 6Q-6T is reversible before the characteristic strength of the contact 6034 reaches the deep press strength threshold IT _{D as} shown in FIG. 6T. For example, decreasing the characteristic strength of contact 6034 after the increases illustrated in FIGS. 6R and/or 6S will cause the interface state corresponding to the reduced intensity level of contact 6034 to be displayed.

6Q to 6U, the virtual chair 5020 is (e.g., as determined by the device 100 that the virtual chair 5020 is configured to be placed in an upright orientation on a detected horizontal surface, such as the floor surface 5038). ) Placed on the detected plane, and the size of the virtual chair 5020 is adjusted (e.g., the scale of the virtual chair 5020 relative to the physical space 5002 as shown in the field of view 6036 of the camera(s)) Is determined based on the defined “real world” size of the virtual chair 5020 within the field of view 6036 of the camera(s) and/or the detected size of objects (such as table 5004). The orientation of the virtual chair 5020 caused by rotation of the virtual chair 5020 while the staging interface 6010 is displayed (e.g., as described in connection with FIGS. 6J and 6K) is determined by the virtual chair 5020 It is maintained as it transitions from the staging user interface 6010 to the field of view 6036 of the camera(s). For example, the orientation of the virtual chair 5020 relative to the floor surface 5038 is the same as the final orientation of the virtual chair 5020 relative to the surface of the stage 5014. In some embodiments, the adjustment to the size of the virtual object 5020 in the staging user interface is considered when the size of the virtual chair 5020 is adjusted to the size of the physical space 5002 within the field of view 6036.

6V-6Y illustrate an input that causes the field of view 6036 of the camera(s) to be replaced with a staging user interface 6010. In FIG. 6V, an input (eg, tap input) by the contact 6040 is at a position corresponding to the toggle control unit 6018 (eg, at a position corresponding to a part of the toggle control unit 6018 including text “3D”) Is detected. 6W-6Y, in response to input by contact 6040, the field of view 6036 of the camera(s) fades out (as indicated by dashed lines in FIG. 6W), and the staging user interface 6010 It fades in (as indicated by dotted lines in Fig. 6x), and the staging user interface 6010 is fully displayed (as shown in Fig. 6y). 6V to 6Y, the size of the virtual chair 5020 is adjusted and the position of the virtual chair 5020 is adjusted (e.g., to return the virtual chair 5020 to a predefined position and size for the staging user interface). Changes.

6Z-6AC illustrate inputs that cause the staging user interface 6010 to be replaced with the messaging user interface 5008. In FIG. 6Z, an input (eg, a tap input) by a contact 6042 is detected at a position corresponding to the rewind control unit 6016. 6AA-6AC, in response to input by contact 6042, staging user interface 6010 fades out (as indicated by dashed lines in FIG. 6AA) and messaging user interface 5008 (FIG. Fade in (as indicated by the dotted lines in), and the messaging user interface 5008 is fully displayed (as shown in Figure 6ac). 6z to 6ab, the size, orientation, and position of the virtual chair 5020 (e.g., to return the virtual chair 5020 to the predefined location, size and orientation for the messaging user interface 5008). Adjusted continuously on the display.

6A-6AJ illustrate an input (eg, bypassing the display of staging user interface 6010) that causes the messaging user interface 5008 to be replaced with the field of view 6036 of the camera(s). In FIG. 6A, a contact 6044 is detected at a location corresponding to the virtual chair 5020. The input by the contact 6044 is a long touch gesture (during a long touch gesture, the contact 6044 moves below a threshold amount for at least a predefined threshold time in a location on the touch-sensitive surface corresponding to the representation of the virtual object 5020). Maintained) followed by an upward swipe gesture (dragging the virtual chair 5020 upward). 6A and 6A, the virtual chair 5020 is dragged upward as the contact 6044 moves along the path indicated by the arrow 6046. In Figure 6AE, the messaging user interface 5008 fades out behind the virtual chair 5020. As shown in FIGS. 6A and 6A, the virtual chair 5020 continues to be dragged upward as the contact 6044 moves along the path indicated by the arrow 6048. In FIG. 6Af, the field of view 5036 of the camera(s) fades in behind the virtual chair 5020. In FIG. 6AG, in response to input by contact 6044 including a long touch gesture followed by an upward swipe gesture, the field of view 5036 of the camera(s) is fully displayed. In FIG. 6A, the contact 6044 is lifted off from the touch screen 112. 6A-6AJ, in response to liftoff of contact 6044, virtual chair 5020 is released (e.g., because virtual chair 5020 is no longer constrained or dragged by the contact) and flat (e.g., , According to the determination that the horizontal (floor) surface corresponds to the virtual chair 5020, it falls to the floor surface 5038). Additionally, as illustrated in FIG. 6AJ, the tactile output generators 167 of the device 100 are (as illustrated at 6050) to indicate that the virtual chair 5020 has landed on the floor surface 5038. ) Outputs tactile output.

7A-7P illustrate example user interfaces for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object, according to some embodiments. User interfaces in these figures are in FIGS. 8A to 8E, 9A to 9D, 10A to 10D, 16A to 16G, 17A to 17D, 18A to 18I, 19A to 19H, and FIG. It is used to illustrate the processes described below, including the processes of FIGS. 20A-20F. For ease of description, some of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is, optionally: an individual finger or stylus contact, or a representative point corresponding to the finger or stylus contact (e.g., the center of the individual contact, or a point associated with the individual contact), or It is the center of two or more contacts detected on the touch-sensitive display system 112. However, similar operations may be performed in response to detecting contacts on the touch-sensitive surface 451, optionally displaying the user interfaces shown in the figure with the focus selector on the display 450, and It is performed on a device with a separate touch-sensitive surface 451.

7A illustrates input detected while the user interface 400 for a menu of applications is displayed. The input corresponds to a request to display a first user interface (eg, internet browser user interface 5060). In FIG. 7A, input by contact 7000 (eg, tap input) is detected at a location corresponding to icon 420 for browser module 147. In response to the input, an Internet browser user interface 5060 is displayed as shown in FIG. 7B.

7B illustrates an internet browser user interface 5060 (eg, as described in detail in connection with FIG. 5AE). The internet browser user interface 5060 includes web objects 5066, 5068, 5070, 5072, 5074, 5076. Web objects 5066, 5068, 5072 include two-dimensional representations of three-dimensional virtual objects, as indicated by virtual object indicators 5078, 5080, and 5082, respectively. Web objects 5070, 5074, 5076 contain two-dimensional images (however, two-dimensional images of web objects 5070, 5074, 5076, as indicated by no virtual object indicators, are three-dimensional virtual Does not correspond to objects).

7C and 7D illustrate inputs that cause translation (eg, scrolling) of Internet browser user interface 5060. In FIG. 7B, a contact 7002 with the touch screen 112 is detected. 7C and 7D, as the contact 7002 moves along the path indicated by the arrow 7004, the web objects 5066, 5068, 5070, 5072, 5074, 5076 scroll upward, resulting in an additional web object. Reveal field (7003, 7005). Additionally, as the contact 7002 moves along the path indicated by the arrow 7004, the virtual objects in the web objects 5066, 5068, 5072 including the virtual object indicators 5078, 5080, 5082 respectively It rotates according to the (vertical upward) direction of the input (eg, tilts upward). For example, the imaginary ramp 5084 is tilted upward from the first orientation of FIG. 7C to the second orientation of FIG. 7D. The two-dimensional images of web objects 5070, 5074, 5076 do not rotate as the contact scrolls the internet browser user interface 5060. In FIG. 7E, the contact 7002 has been lifted off from the touch screen 112. In some embodiments, the rotational behavior of objects shown in web objects 5066, 5068, 5072 is used as a visual indication that these web objects have corresponding three-dimensional virtual objects observable in an augmented reality environment, while The lack of such rotational behavior of objects shown in web objects 5070, 5074, 5076 is used as a visual indication that these web objects do not have corresponding three-dimensional virtual objects observable in an augmented reality environment.

7F and 7G illustrate a parallax effect in which virtual objects rotate on the display in response to a change in the orientation of the device 100 with respect to the physical world.

7F1 illustrates the device 100 held in the user's hand 5006 by the user 7006 such that the device 100 has a substantially vertical orientation. 7F2 illustrates an internet browser user interface 5060 as displayed by device 100 when device 100 is in the orientation illustrated in FIG. 7F1.

7G1 illustrates the device 100 held in the user's hand 5006 by the user 7006 such that the device 100 has a substantially horizontal orientation. 7G2 illustrates an internet browser user interface 5060 as displayed by device 100 when device 100 is in the orientation illustrated in FIG. 7G1. 7F2 to 7G2, the orientation of the virtual objects in the web objects 5066, 5068, 5072 including the virtual object indicators 5078, 5080, 5082, respectively, rotates according to the change of the orientation of the device (e.g. , Inclined upward). For example, the virtual lamp 5084 tilts upward from the first orientation of FIG. 7F2 to the second orientation of FIG. 7G2 according to the simultaneous change of the device orientation within the physical space. The two-dimensional images of the web objects 5070, 5074, 5076 do not rotate as the orientation of the device changes. In some embodiments, the rotational behavior of objects shown in web objects 5066, 5068, 5072 is used as a visual indication that these web objects have corresponding three-dimensional virtual objects observable in an augmented reality environment, while The lack of such rotational behavior of objects shown in web objects 5070, 5074, 5076 is used as a visual indication that these web objects do not have corresponding three-dimensional virtual objects observable in an augmented reality environment.

7H-7L illustrate input corresponding to a request to display a second user interface (eg, messaging user interface 5008). In FIG. 7H, a contact 7008 is detected at a location corresponding to the lower edge of the display 112. In FIGS. 7H and 7I, contact 7008 moves upwardly along the path indicated by arrow 7010. 7I and 7J, contact 7008 continues to move upwards along the path indicated by arrow 7012. 7H-7J, as the contact 7008 moves upward from the lower edge of the display 112 (e.g., in response to an upward edge swipe gesture by the contact 7008), as shown in FIG. 7I. As such, the size of the Internet browser user interface 5060 is reduced, and in Fig. 7J, the multitasking user interface 7012 is displayed. Multitasking user interface 7012 includes various applications and various applications with retained states (e.g., the retained state is the last state of each application when each application was a foreground application running on the device). It is configured to allow selection of an interface from among control interfaces (eg, control center user interface 7014, internet browser user interface 5060, and messaging user interface 5008, as illustrated in FIG. 7J). In FIG. 7K, the contact 7008 is lifted off from the touch screen 112. In FIG. 7L, input by contact 7016 (eg, tap input) is detected at a location corresponding to messaging user interface 5008. In response to input by contact 7016, a messaging user interface 5008 is displayed, as illustrated in FIG. 7M.

7M shows that the virtual object (e.g., virtual chair 5020) and virtual chair 5020 received in the message are a virtual three-dimensional object (e.g., an object observable in an augmented reality view and/or an object observable from different angles). Illustrative of a messaging user interface 5008 (eg, as described in more detail with respect to FIG. 5B) including a message balloon 5018 that includes a virtual object indicator 5022 indicating. The messaging user interface 5008 also includes a message balloon 6005 including a transmitted text message and a message balloon 7018 including a received text message including an emoji 7020. The emoji 7020 is a two-dimensional image that does not correspond to a virtual three-dimensional object. For this reason, the emoji 7020 is displayed without a virtual object indicator.

7N illustrates a map user interface 7022 including a map 7024, a point of interest information area 7026 for a first point of interest, and a point of interest information area 7032 for a second point of interest. For example, the first point of interest and the second point of interest are search results in or near the area shown by map 7024, corresponding to the search entry "Apple" in search input area 7025. In the first point of interest information area 7026, the first point of interest object 7028 is displayed together with a virtual object indicator 7030 indicating that the first point of interest object 7028 is a virtual three-dimensional object. In the second point of interest information area 7032, the second point of interest object 7034 is displayed without a virtual object indicator because the second point of interest object 7034 does not correspond to a virtual 3D object observable in the augmented reality view. do.

7O shows a file management control unit 7038, a file management search input area 7040, a file information area 7042 for a first file (eg, a portable document format (PDF) file), and a second file (eg, a photo file). ), a file information area 7046 for a third file (eg, a virtual chair object), and a file information area 7048 for a fourth file (eg, a PDF file). The file management user interface 7036 is illustrated. The third file information area 7046 is a virtual object indicator 7050 displayed adjacent to the file preview object 7045 of the file information area 7046 to indicate that the third file corresponds to a virtual 3D object. Includes. The first file information area 7042, the second file information area 7048, and the fourth file information area 7048 are corresponding virtual 3D objects in which files corresponding to these file information areas can be observed in an augmented reality environment. They are displayed without virtual object indicators because they do not have.

FIG. 7P shows an email user including email navigation controls 7054, an email information area 7056, and an email content area 7058 including a representation of a first attachment 7060 and a representation of a second attachment 7062 The interface 7052 is shown. The representation of first attachment 7060 includes a virtual object indicator 7054 indicating that the first attachment is a virtual three-dimensional object observable in an augmented reality environment. The second attachment 7062 is displayed without a virtual object indicator because the second attachment is not a virtual three-dimensional object observable in an augmented reality environment.

8A-8E are a flow diagram illustrating a method 800 of displaying a representation of a virtual object while switching from displaying a first user interface area to displaying a second user interface area, in accordance with some embodiments. admit. Method 800 includes an electronic device (e.g., the device of FIG. 3) having a display, a touch-sensitive surface, and one or more cameras (e. 300, or the portable multifunction device 100 of FIG. 1A). In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 800 are selectively combined and/or the order of some operations is selectively changed.

The method 800 is directed to detecting input by contact at a touch-sensitive surface of a device displaying a representation of a virtual object within a first user interface area. In response to the input, the device uses criteria to determine whether to continuously display the representation of the virtual object while replacing the display of at least a portion of the first user interface area with the field of view of one or more cameras of the device. Performing a number of different types of actions in response to input by using criteria for determining whether to continuously display a representation of the virtual object while replacing the display of at least a portion of the first user interface area with the field of view of one or more cameras. Makes it possible. (E.g., by replacing the display of at least a portion of the user interface with the field of view of one or more cameras, or without replacing the display of at least a portion of the first user interface with a representation of the field of view of the one or more cameras. Enabling the performance of a number of different types of actions in response to input (by maintaining the display) increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally: It reduces device power usage and improves battery life by enabling users to use the device more quickly and efficiently.

The device is within a first user interface area on the display 112 (e.g., a two-dimensional graphical user interface or a portion thereof (e.g., a browseable list of furniture images, an image comprising one or more selectable objects, etc.)). Representation of a virtual object (e.g., a graphic representation of a 3D object, such as a virtual chair 5020, a virtual lamp 5084, shoes, furniture, hand tools, decorations, people, emoji, game characters, virtual furniture, etc. ) Is displayed (802). For example, the first user interface area is a messaging user interface 5008 as shown in FIG. 5B or an Internet browser user interface 5060 as shown in FIG. 5A. In some embodiments, the first user interface area includes a background other than an image of the physical environment around the device (e.g., the background of the first user interface area is a preselected background color/pattern, or by one or more cameras. It is a background image distinct from the output image captured at the same time and distinct from the live content within the field of view of one or more cameras).

While displaying the first representation of the virtual object in the first user interface area on the display, the device detects a first input by contact at a location on the touch-sensitive surface 112 corresponding to the representation of the virtual object on the display ( For example, the contact is detected on a first representation of the virtual object on the touch screen display, or the contact is displayed simultaneously with the first representation of the virtual object in the first user interface area and when called by the contact, the display of the AR view of the virtual object Is detected on the affordance configured to trigger) (804). For example, the first input is input by contact 5020 as described in connection with FIGS. 5C-5F or input by contact 5086 as described in connection with FIGS. 5A-5A.

In response to detecting the first input by contact, according to a determination that the first input by contact meets the first (e.g., AR trigger) criteria (e.g., AR trigger criteria trigger activation of the camera(s). Swipe input, touch-hold input, press input, tap input, hard press with a strength exceeding a predefined strength threshold, or other types of predefined input gestures associated with doing , Display of an augmented reality (AR) view of the physical environment around the device, placement of a three-dimensional representation of a virtual object inside the augmented reality view of the physical environment, and/or criteria configured to identify a combination of two or more of the above actions) , The device displaying a second user interface area on the display, including replacing the display of at least a portion of the first user interface area with a representation of the field of view of the one or more cameras, and the device displaying the first user interface area The representation of the virtual object is continuously displayed while switching from doing to displaying the second user interface area (806). For example, the second user interface area on the display may be the field of view 5034 of the camera(s) in the platter 5030 as described in connection with FIG. 5H, or the camera(s) as described in connection with FIG. Of the field of view 5034. In FIGS. 5C to 5I, according to a determination that the input by the contact 5026 has a characteristic intensity that increases by exceeding the deep pressing intensity threshold IT _D , the virtual chair object 5020 is a first user interface area ( Switching from displaying the messaging user interface 5008 to displaying a second user interface area that replaces the display of a portion of the messaging user interface 5008 with the field of view 5034 of the camera(s) in the platter 5030. During the process, they are displayed continuously. 5A to 5A, according to the determination that the input by the contact 5086 has a characteristic intensity that increases by exceeding the deep pressing intensity threshold IT _D , the virtual lamp object 5084 is a first user interface area ( Continuously while switching from displaying the Internet browser user interface 5060 to displaying a second user interface region replacing the display of a portion of the Internet browser user interface 5060 with the field of view 5034 of the camera(s). Is displayed.

In some embodiments, continuously displaying the representation of the virtual object maintains a display of the representation of the virtual object, or a second representation of the virtual object (e.g., of a different size, from a different viewing angle, of a different rendering style). , Or a view of the virtual object at a different location on the display). In some embodiments, the field of view 5034 of one or more cameras is a physical environment around the device that is updated in real time when the device's location and orientation changes relative to the physical environment (e.g., as illustrated in FIGS. A live image of the environment 5002 is displayed. In some embodiments, the second user interface area completely replaces the first user interface area on the display.

In some embodiments, the second user interface area overlays a portion of the first user interface area (eg, a portion of the first user interface area is shown along an edge of the display or around its boundaries). In some embodiments, the second user interface area pops up after the first user interface area. In some embodiments, the background within the first user interface area is replaced with the content of the field of view 5034 of the camera(s). In some embodiments, the device pre-exists for the current orientation of the portion of the physical environment where the virtual object is captured from a first orientation as shown in the first user interface area to a second orientation (e.g., within the field of view of one or more cameras). (E.g., as illustrated in FIGS. 5E-5I) in a defined orientation) and an animated transition showing rotation. For example, animation includes a transition from displaying a two-dimensional representation of a virtual object while displaying a first user interface area to displaying a three-dimensional representation of a virtual object while displaying a second user interface area. . In some embodiments, the three-dimensional representation of the virtual object is a predefined anchor plane based on the shape and orientation of the virtual object as shown in the two-dimensional graphical user interface (e.g., first user interface area). Has. When transitioning to an augmented reality view (e.g., a second user interface area), the three-dimensional representation of the virtual object is transferred from the original position of the virtual object on the display to a new position on the display (e.g., the center of the augmented reality view, or the augmented reality view). Is moved, resized, reorientated, and during or at the end of the movement, the three-dimensional representation of the virtual object is identified within the field of view of one or more cameras). It is reoriented to be in a predefined position and/or orientation relative to a predefined plane (eg, a physical surface such as a vertical wall or horizontal floor surface, which can serve as a support plane for a three-dimensional representation of a virtual object).

In some embodiments, the first criteria are when the contact is maintained with less than a threshold amount of movement for at least a predefined time (e.g., a long press time threshold) at a location on the touch-sensitive surface corresponding to the representation of the virtual object ( Criteria that are satisfied (eg, depending on the determination to be maintained) are included (808). In some embodiments, upon determining that the contact satisfies the criteria for recognizing other types of gestures (e.g., taps), the device may pre-exist other than triggering the AR user interface while maintaining the display of the virtual object. It performs a defined function. View of the camera(s) at least a portion of the display of the first user interface area, depending on whether the contact is maintained with less than a threshold amount of movement for at least a predefined amount of time at a location on the touch-sensitive surface corresponding to the representation of the virtual object. Determining whether or not to continuously display the representation of the virtual object during substitution with enables the performance of a number of different types of operations in response to input. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, the first criteria are when the characteristic strength of the contact increases above a first intensity threshold (e.g., a light tap intensity threshold (IT _L ) or a deep tap intensity threshold (IT _D )) (e.g., 810). For example, as described in connection with FIGS. 5C-5F, the criteria are that the characteristic strength of the contact 5026 exceeds the deep pressing strength threshold IT _D , as indicated by the intensity level meter 5028. It is satisfied when it increases. In some embodiments, upon determining that the contact satisfies the criteria for recognizing other types of gestures (e.g., taps), the device may pre-exist other than triggering the AR user interface while maintaining the display of the virtual object. It performs a defined function. In some embodiments, the first criteria require that the first input is not a tap input (eg, that the input has a duration between a touchdown of a contact and a liftoff of a contact longer than a tap time threshold). Whether or not to continuously display the representation of the virtual object while replacing the display of at least a portion of the first user interface area with the field of view of the camera(s), depending on whether the characteristic intensity of the contact increases beyond the first intensity threshold. Determining enables the performance of a number of different types of actions in response to an input. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, the first criteria are when the movement of the contact meets predefined movement criteria (e.g., the contact is at a predefined threshold position across the touch-sensitive surface (e.g., the boundary of the first user interface area). When moving past a position corresponding to, a position that is a critical distance away from the original position of the contact, etc.), when the contact moves at a speed faster than a predefined threshold speed, when the movement of the contact ends with a pressing input, etc.) Criteria that are satisfied (eg, depending on the determination that so) are included (812). In some embodiments, the representation of the virtual object is dragged by the contact during an initial portion of the movement of the contact, and when the movement of the contact attempts to meet predefined defined movement criteria to indicate that the first criteria are to be met The object stops moving by contact; If the movement of the contact continues and the predefined movement criteria are met by the continued movement of the contact, a transition to display the second user interface area and display the virtual object in the augmented reality view is initiated. In some embodiments, when the virtual object is dragged during the initial portion of the first input, the object size and viewing point do not change, and once the augmented reality view is displayed and the virtual object falls into place in the augmented reality view, the virtual object Objects are displayed with a size and observation point dependent on a physical position represented by a drop-off position of a virtual object in the augmented reality view. Determining whether to continuously display a representation of the virtual object while replacing the display of at least a portion of the first user interface area with the field of view of the camera(s), depending on whether the movement of the contact meets predefined movement criteria. This enables the performance of a number of different types of operations in response to an input. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, in response to detecting the first input by the contact, in response to a determination that the first input by the contact has met the first criteria, the device determines satisfaction of the first criteria by the first input. To represent a tactile output (e.g., tactile output 5032 as described in connection with FIG. 5F or tactile output 5088 as described in connection with FIG. 5A), one or more tactile output generators By (167), it is output (814). In some embodiments, a haptic is generated before the field of view of one or more cameras is visible on the display. For example, haptic indicates satisfaction of first criteria that triggers activation of one or more camera(s) and subsequent plane detection of the field of view of one or more camera(s). Because it takes time for the cameras to activate and for the field of view to become available for display, the haptic costs the user that the device has detected the required input and that it will present the augmented reality user interface as soon as the device is ready. It serves as a visual cue.

Outputting a tactile output to indicate satisfaction of criteria (e.g., to replace the display of at least a portion of the user interface with the field of view of the camera(s)) provides feedback to the user indicating that the input provided meets the criteria. do. Providing improved tactile feedback improves the operability of the device (e.g., by helping the user to provide appropriate inputs when actuating/interacting with the device and reducing user mistakes), which additionally: It reduces the power usage of the device and improves battery life by enabling users to use the device more quickly and efficiently.

In some embodiments, detecting at least an initial portion of the first input (e.g., detecting a contact, or detecting an input by a contact that meets respective predefined criteria while not meeting the first criteria). Responsive to doing, or detecting an input that meets the first criteria), the device is capable of performing one or more planes (e.g., floor surface 5038, table surface 5046, Walls, etc.) to detect the field of view of one or more cameras (816). In some embodiments, one or more cameras are activated in response to detecting at least an initial portion of the first input, and plane detection is initiated simultaneously when the camera(s) are activated. In some embodiments, the display of the field of view of one or more cameras is delayed after activation of one or more cameras (e.g., from when one or more cameras are activated until at least one plane is detected within the field of view of the camera(s)). do. In some embodiments, display of the field of view of one or more cameras is initiated when one or more cameras are activated, and plane detection is completed after the field of view is already visible on the display (eg, in a second user interface area). In some embodiments, after detecting each plane within the field of view of one or more cameras, the device determines the size and/or position of the representation of the virtual object based on the relative position of each plane to the field of view of the one or more cameras. Decide. In some embodiments, as the electronic device is moved, the size and/or location of the representation of the virtual object is determined by the location of the field of view of one or more cameras (eg, as described in connection with FIGS. 5K and 5L ). It is updated as it changes with respect to the plane. Based on the location of each plane detected within the field of view of the camera(s) (e.g., without requiring additional user input to size and/or position the virtual object relative to the field of view of the camera(s)) Determining the size and/or location of the representation of the object improves the operability of the device, which in addition reduces the device's power usage and improves battery life by enabling users to use the device more quickly and efficiently. Let it.

In some embodiments, analyzing the field of view of the one or more cameras to detect one or more planes within the field of view of the one or more cameras results in detection of a contact at a location on the touch-sensitive surface corresponding to the representation of the virtual object on the display. Initiated 818 in response (eg, in response to detection of contact 5026 at a location on touch screen 112 corresponding to virtual chair 5020 ). For example, activation of the cameras and detection of planes within the field of view of the camera(s) may be performed prior to the first criteria being met by the first input (e.g., as described in connection with FIG. It starts before the intensity increases beyond the deep pressing intensity threshold IT _D ) and before the second user interface area is displayed. By starting plane detection upon detection of any interaction with the virtual object, plane detection can be completed before the AR trigger criteria are satisfied, and thus, into the augmented reality view when the AR trigger criteria are satisfied by the first input. There will be no visible delay to the user when viewing the virtual object transition of. One or more planes within the field of view of the camera(s) in response to detection of contact at the location of the representation of the virtual object (e.g., without requiring additional user input to initiate analysis of the field of view of the camera(s)). Initiating the analysis to detect improves the effectiveness of the device, which additionally reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, analyzing the field of view of the one or more cameras to detect one or more planes within the field of view of the one or more cameras in response to detecting that the first criteria are satisfied by the first input by contact ( For example, as described in connection with FIG. 5F, in response to detecting that the characteristic strength of the contact 5026 increases above the deep pressing strength threshold IT _D ), it is initiated (820). For example, activation of the cameras and detection of planes within the field of view of the camera(s) begins when the first criteria are met by a first input, and the field of view of the camera is displayed before the plane detection is complete. By starting camera activation and plane detection when the AR trigger criteria are satisfied, cameras and plane detection are not unnecessarily activated and do not continue, which conserves battery power and extends battery life and camera life.

In some embodiments, analyzing the field of view of the one or more cameras to detect one or more planes within the field of view of the one or more cameras does not satisfy the first criteria while the initial portion of the first input does not meet the first criteria. In response to detecting that it is satisfied, it is initiated (822). For example, activation of the cameras and detection of planes within the field of view of the camera(s) may occur when some criteria (e.g., criteria less stringent than AR trigger criteria) are met by the initial portion of the first input. It is started and the camera's field of view is selectively displayed before plane detection is complete. By starting camera activation and plane detection after satisfying certain criteria rather than upon detection of contact, cameras and plane detection are not unnecessarily activated and do not continue, which conserves battery power and prolongs battery life and camera life. . By starting the camera activation and plane detection before the AR trigger criteria are satisfied, the delay (due to camera activation and plane detection) to display the virtual object transition to the augmented reality view when the AR trigger criteria are satisfied by the first input is reduced. do.

In some embodiments, the device is such that the virtual object (e.g., virtual chair 5020) is oriented at a predefined angle relative to each plane detected within the field of view 5034 of one or more cameras (e.g., a floor surface ( The representation of the virtual object in the second user interface area is displayed in each manner such that there is no distance (or has a minimum distance) separating the lower surfaces of the four legs of the virtual chair 5020 from 5038) (824). . For example, the orientation and/or position of the virtual object relative to each plane is predefined based on the shape and orientation of the virtual object as shown in the two-dimensional graphical user interface (e.g., each plane is an augmented reality view). Corresponds to a horizontal physical surface (e.g., a horizontal table surface to support a vase) that can serve as a support surface for a three-dimensional representation of a virtual object within, or each plane is a three-dimensional representation of a virtual object in an augmented reality view It is a vertical physical surface (eg, a vertical wall for hanging a virtual picture frame) that can serve as a support surface for the. In some embodiments, the orientation and/or location of the virtual object is defined by each surface or boundary (eg, lower surface, lower boundary points, side surface, and/or side boundary points) of the virtual object. In some embodiments, the anchor plane corresponding to each plane is an attribute of one of a set of attributes of the virtual object, and is specified according to the characteristic of the physical object that the virtual object is supposed to represent. In some embodiments, the virtual object is placed in a predefined orientation and/or position with respect to multiple planes detected within the field of view of one or more cameras (e.g., multiple respective faces of the virtual object ) Is associated with each of the planes detected within the field of view). In some embodiments, if a predefined orientation and/or position for the virtual object is defined relative to the horizontal lower plane of the virtual object, the lower plane of the virtual object is on the floor plane detected within the field of view of the camera(s). Displayed (eg, the horizontal lower plane of the virtual object is parallel to the floor plane with a distance of zero from the floor plane). In some embodiments, if the predefined orientation and/or position for the virtual object is defined relative to the vertical rear plane of the virtual object, the rear surface of the virtual object is against the wall plane detected within the field of view of one or more cameras. Placed (eg, the vertical rear plane of the virtual object is parallel to the wall plane with a distance of zero from the wall plane). In some embodiments, the virtual object is placed at a fixed distance to each plane and/or at an angle other than 0 degrees or 90 degrees to each plane. Displaying a representation of the virtual object relative to the plane detected within the field of view of the camera(s) (e.g., without requiring additional user input to display the virtual object against the plane within the field of view of the camera(s)) Improves the operability of the device, which in addition, reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, in response to detecting each plane within the field of view of the one or more cameras, the device outputs a tactile output to indicate detection of each plane within the field of view of the one or more cameras, one or more tactile outputs. By generators 167, output 826. In some embodiments, a respective tactile output is generated for each plane (eg, floor surface 5038 and/or table surface 5046) detected within the field of view of the camera(s). In some embodiments, the tactile output is generated when plane detection is complete. In some embodiments, the tactile output is accompanied by a visual indication of the field of view (eg, a momentary highlight of the detected field of view) within the field of view shown in the second user interface portion. Outputting a tactile output to indicate detection of a plane in the field of view of the camera(s) provides feedback to the user indicating that the plane has been detected. Providing improved tactile feedback improves the operability of the device (e.g., by helping the user provide appropriate inputs and reducing unnecessary additional inputs to place the virtual object), which additionally allows the user to It reduces the power usage of the device and improves battery life by making it possible to use the device more quickly and efficiently.

In some embodiments, while switching from displaying the first user interface area to displaying the second user interface area, the device displays the representation of the virtual object (eg, as illustrated in FIGS. 5F-5I ). Display the animation as it transitions (e.g., move, rotate, resize, and/or re-render in a different style, etc.) to a predefined location for each plane into the second user interface area, each At a predefined angle relative to the plane of the virtual (e.g., in a predefined orientation and/or position for each plane, and in its size, rotation angle, and appearance to reach the final state to be seen in the augmented reality view). Along with displaying the representation of the object, the device generates a tactile output to indicate displaying the virtual object at a predefined angle for each plane within the second user interface area, one or more tactile output generators 167. ) To output (828). For example, as illustrated in FIG. 5I, the device outputs a tactile output 5036 along with displaying the virtual chair 5020 at a predefined angle relative to the floor surface 5038. In some embodiments, the generated tactile output is weight (e.g., heavy versus light), material (e.g., metal, cotton, wood, marble, liquid, rubber, glass), size (e.g., large versus small), shape (E.g., thin versus thick, long versus short, round versus pointed, etc.), elasticity (e.g., elastic versus rigid), nature (e.g., cheerful versus solemn, gentle versus coercion, etc.), and of or on virtual objects It is configured to have properties (eg, frequency, number of cycles, modulation, amplitude, accompanying audio waves, etc.) that reflect other properties of the physical object represented by. For example, tactile output uses one or more of the tactile output patterns illustrated in FIGS. 4F-4K. In some embodiments, a preset profile comprising one or more changes to one or more properties over time corresponds to a virtual object (eg, an emoji). For example, a "bouncing" tactile output profile is provided for a "smiley face" emoji virtual object. Outputting a tactile output to indicate the placement of the representation of the virtual object for each plane provides feedback to the user indicating that the representation of the virtual object has been automatically placed for each plane. Providing improved tactile feedback improves the operability of the device (e.g., by helping the user provide appropriate inputs and reducing unnecessary additional inputs to place the virtual object), which additionally allows the user to It reduces the power usage of the device and improves battery life by making it possible to use the device more quickly and efficiently.

In some embodiments, the tactile output has a tactile output profile 830 that corresponds to a property of the virtual object (eg, simulated physical property such as size, density, mass, and/or material). In some embodiments, the tactile output profile is variable based on one or more properties (e.g., weight, material, size, shape, and/or elasticity) of the virtual object (e.g., frequency, number of cycles, Modulation, amplitude, accompanying audio waves, etc.). For example, tactile output uses one or more of the tactile output patterns illustrated in FIGS. 4F-4K. In some embodiments, the amplitude and/or duration of the tactile output increases as the size, weight, and/or mass of the virtual object increases. In some embodiments, the tactile output pattern is selected based on the virtual material constituting the virtual object. Outputting a tactile output having a profile corresponding to the characteristics of the virtual object provides feedback to the user indicating information on the characteristics of the virtual object. Providing improved tactile feedback (e.g., helping the user to provide appropriate inputs, reducing unnecessary additional inputs for placing virtual objects, and not confusing the user interface with the displayed information about the properties) Improves the operability of the device by providing sensory information that allows the user to recognize the characteristics of the virtual object), which additionally reduces the power consumption of the device by enabling the user to use the device more quickly and efficiently. Improves battery life.

In some embodiments, while displaying a representation of the virtual object within the second user interface area, the device is configured to adjust the field of view 5034 of one or more cameras (eg, as illustrated in FIGS. 5K and 5L ). In response to detecting movement (e.g., lateral movement and/or rotation of the device), and in response to detecting the movement of the device, the device is configured to each plane within the field of view of one or more cameras (e.g., , Adjusts the representation of the virtual object (e.g., virtual chair 5020) within the second user interface area according to a fixed spatial relationship (e.g., orientation and/or position) between the floor surface 5038) and the virtual object. (E.g., a virtual object is displayed on the display with an orientation and position such that a fixed angle between the plane and the representation of the virtual object is maintained (e.g., the virtual object stays in a fixed position on the plane or rolls along the viewing plane). Seems to be)) (832). For example, in FIGS. 5K and 5L, the virtual chair 5020 in the second user interface area including the field of view 5034 of the camera(s) is relative to the floor surface 5038 as the device 100 is moved. Maintain a fixed orientation and position. In some embodiments, the virtual object appears to be stationary and unchanged relative to the surrounding physical environment 5002, i.e., the representation of the virtual object changes in size, position, and on the display as the device position and/or orientation changes. /Or the orientation changes because the field of view of one or more cameras changes as the device moves relative to the surrounding physical environment. Adjusting the representation of the virtual object according to the fixed relationship between the virtual object and each plane (e.g., without requiring additional user input to maintain the position of the virtual object with respect to each plane) is the operability of the device. , Which additionally reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, (e.g., at a time corresponding to replacing the display of at least a portion of the first user interface area with a representation of the field of view of one or more cameras), the device is a virtual object (e.g., virtual chair 5020). The animation (e.g., movement, movement, etc.) is displayed continuously while switching from displaying the first user interface area to displaying the second user interface area (e.g., as illustrated in FIGS. 5F-5I). Rotation about one or more axes, and/or scaling, is displayed (834). For example, animation includes a transition from displaying a two-dimensional representation of a virtual object while displaying a first user interface area to displaying a three-dimensional representation of a virtual object while displaying a second user interface area. . In some embodiments, the three-dimensional representation of the virtual object has a predefined orientation relative to the current orientation of the portion of the physical environment that is captured within the field of view of one or more cameras. In some embodiments, when transitioning to an augmented reality view, the representation of the virtual object is moved from an initial location on the display to a new location on the display (e.g., the center of the augmented reality view, or another predefined location within the augmented reality view) and , Resized, re-orientated, and, during or at the end of the movement, the virtual object is a plane detected within the field of view of the camera(s) (e.g., with a vertical wall or horizontal floor surface capable of supporting the representation of the virtual object). To be at a fixed angle to the same physical surface) In some embodiments, the contrast of the virtual object and/or the shadow cast by the virtual object is adjusted as an animated transition occurs (eg, to match the ambient lighting detected within the field of view of one or more cameras). Displaying the animation as a representation of the virtual object while switching from displaying the first user interface area to the second user interface area provides feedback to the user indicating that the first input meets the first criteria. Providing improved feedback improves the operability of the device (e.g., by helping the user to provide appropriate inputs when operating/interacting with the device and reducing user errors), which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use the device more quickly and efficiently.

In some embodiments, while displaying the second user interface area on the display, the device detects a second input by a second contact (e.g., contact 5040), where the second input is (e.g. 5n to 5p) (optionally, including pressing or touch input by a second contact to select a representation of the virtual object, and) movement of the second contact along a first path across the display. -, in response to detecting the second input by the second contact, the device is within the second user interface area along a second path corresponding to the first path (e.g., equal to or constrained by the first path). The representation of the virtual object (eg, virtual chair 5020) is moved (836). In some embodiments, the second contact is separate from the first contact and is detected after liftoff of the first contact (e.g., FIGS. 5N-5P detected after liftoff of contact 5026 in FIGS. 5C-5F. As illustrated by the contact 5040). In some embodiments, the second contact is (e.g., as illustrated by input by contact 5086 that meets AR trigger criteria and then moves across touch screen 112 to move virtual lamp 5084). Like) the same as the first contact continuously held on the touch-sensitive surface. In some embodiments, while a swipe input on the virtual object rotates the virtual object, the movement of the virtual object is optionally constrained by a plane within the field of view of the camera(s) (e.g., the swipe input is the camera(s). Rotate the representation of the chair on the floor plane within the field of view) Moving the representation of the virtual object in response to detecting the input provides feedback to the user indicating that the displayed location of the virtual object is movable in response to user input. Providing improved feedback improves the operability of the device (e.g., by helping the user to provide appropriate inputs when operating/interacting with the device and reducing user errors), which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use the device more quickly and efficiently.

In some embodiments, as the representation of the virtual object moves along the second path based on the movement of the contact and each plane corresponding to the virtual object (e.g., a virtual distance from the representation of the virtual object to the user) Based on this, the size of the representation of the virtual object is adjusted (838) to maintain an accurate viewpoint of the virtual object in the field of view. For example, in FIGS. 5N-5P, the size of the virtual chair 5020 decreases as the virtual chair moves deeper into the field of view 5034 of the camera(s) toward the table 5004 away from the device 100. do. (E.g., without requiring additional user input to resize the representation of the virtual object to maintain the representation of the virtual object at the actual size of the environment in the field of view of the camera(s)) Adjusting the size of the representation of the virtual object as it moves along the second path based on the movement and the plane corresponding to the virtual object improves the operability of the device, which additionally allows the user to use the device more quickly and efficiently. By making it possible to reduce the power usage of the device and improve battery life.

In some embodiments, the device provides a second representation of the virtual object (e.g., virtual ramp 5084) as the representation of the virtual object moves along the second path (e.g., as illustrated in FIGS. 5AI-5A). Maintaining the first size, the device detects the termination of the second input by the second contact (including detecting the liftoff of the second contact, e.g., as illustrated in FIGS. 5A and 5A ), and 2 In response to detecting the termination of the second input by contact, the device places the representation of the virtual object at a drop position (e.g., on the table surface 5046) within the second user interface area and distinguishes it from the first size. The representation of the virtual object is displayed at the falling position in the second user interface area with the second size (e.g., the size of the virtual lamp 5084 of FIG. 5A, which is after the input by the contact 5086 is terminated, is the contact 5086) It is distinguished from the size of the virtual ramp 5084 of FIG. 5A before the end of the input by (840). For example, an object does not change its size and viewpoint while being dragged by contact, and when the object falls to its final position in the augmented reality view, the object is the drop position of the virtual object visible in the field of view of the camera(s). It is displayed as a size determined based on a physical location in the physical environment corresponding to and an observation time point, so that the object has a second size according to the determination that the fall location is the first location in the field of view of the camera(s), and the fall location is Determination of a second position within the field of view of the camera(s) causes the object to have a third size different from the second size, where the second and third sizes are selected based on the distance of the fall position from one or more cameras. do. Termination of the second input to move the virtual object (e.g., without requiring additional user input to resize the virtual object to maintain the virtual object at the actual size of the environment in the field of view of the camera(s)). Displaying the representation of the virtual object in a changed size in response to detecting improves the operability of the device, which in addition reduces the power usage of the device and reduces battery power consumption by allowing the user to use the device more quickly and efficiently. Improves lifespan.

In some embodiments, the movement of the second contact along the first path across the display meets the second criteria (e.g., at the end of the first path, the contact is at the edge of the display (e.g., lower edge, upper edge). , And/or a side edge) or within or outside a threshold distance of the edge of the second user interface area), the device displays a second user interface area comprising a representation of the field of view of one or more cameras. Stop and redisplay the (all) first user interface area with the representation of the virtual object (e.g., when a part of the first user interface area is previously displayed simultaneously with the second user interface area, the device 2 Display the entire first user interface area after the user interface area is no longer displayed) (842). For example, in response to the movement of the contact 5054 dragging the virtual chair 5054 to the edge of the touch screen 112, as illustrated in FIGS. 5V to 5X, as illustrated in FIGS. 5Y to 5A Likewise, the field of view 5034 of the camera(s) stops being displayed and the entire messaging user interface 5008 is redisplayed. In some embodiments, as the contact approaches the edge of the display or the edge of the second user interface area, the second user interface area fades out and/or fades out (e.g., as illustrated in FIGS. 5X and 5Y). One user interface area (undisplayed or blocked portion of) fades in (eg, as illustrated in FIGS. 5Z and 5AA). In some embodiments, a gesture for transitioning from a non-AR view (eg, a first user interface area) to an AR view (eg, a second user interface area) and a gesture for transitioning from an AR view to a non-AR view are the same. Do. For example, a drag gesture on a virtual object that crosses a threshold position in the currently displayed user interface (e.g., within a threshold distance of the border of the currently displayed user interface area, or crosses the border of the currently displayed user interface area) Is from the currently displayed user interface area to the counterpart user interface area (e.g., from displaying the first user interface area to displaying the second user interface area, or alternatively, the second user interface. From displaying the area to displaying the first user interface area). In some embodiments, the visual indication (e.g., fading out the currently displayed user interface area and fading in the relative user interface) is shown before the first/second criteria are met, and the input continues and the first/ It is reversible if the second criteria are not met before the end of the input (eg, liftoff of the contact) is detected. Re-displaying the first user interface in response to detecting an input that satisfies the input criteria includes the displayed additional controls (e.g., controls for displaying the first user interface from the second user interface) to the second user interface. It provides additional control options without confusing the system. Providing additional control options without cluttering the second user interface with the displayed additional controls improves the operability of the device, which additionally enables the user to use the device more quickly and efficiently, thereby It reduces the power usage of the battery and improves battery life.

In some embodiments, at a time corresponding to redisplaying the first user interface area, the device is configured with a second user interface (e.g., as illustrated by animation of the virtual chair 5020 in FIGS. An animated transition (e.g., movement, rotation around one or more axes, and/or scaling) from displaying a representation of a virtual object within a region to displaying a representation of a virtual object within a first user interface region. Display (844). Displaying the representation of the virtual object in the first user interface from displaying the representation of the virtual object in the second user interface (e.g., without requiring additional user input to reposition the virtual object in the first user interface) Displaying an animated transition to one improves the operability of the device, which additionally reduces the device's power usage and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, as the second contact moves along the first path, the device changes the visual appearance of one or more respective planes identified within the field of view of one or more cameras corresponding to the current location of the contact (e.g. , Highlighting, marking, contouring, and/or otherwise visually altering the appearance of one or more planes) (846). For example, as contact 5042 drags virtual chair 5020 along a path as illustrated by arrows 5042 and 5044 in FIGS. 5O and 5P, floor surface 5038 (e.g., (Compared to FIG. 5M before movement of contact 5042) is highlighted. In some embodiments, upon determination that the contact is in a position corresponding to the detected first plane within the field of view of the camera(s), the first plane is highlighted. According to the determination that the contact has moved to a position corresponding to the second plane detected within the field of view of the camera(s) (e.g., as shown in FIGS. 5S-5U), the first plane (e.g. 5038)) stops being emphasized and the second plane (e.g., table surface 5046) is emphasized. In some embodiments, multiple planes are emphasized simultaneously. In some embodiments, the first of the plurality of visually altered planes is visually altered in a manner separate from the way the other planes are visually altered to indicate that the contact is in a position corresponding to the first plane. . Changing the visual appearance of one or more respective planes identified within the field of view of the camera(s) provides feedback to the user indicating that the plane (eg, a virtual object may be positioned relative to the plane) has been identified. Providing improved visual feedback improves the operability of the device (e.g., by helping the user to provide appropriate inputs when operating/interacting with the device and reducing user mistakes), which additionally It reduces device power usage and improves battery life by making it possible to use the device faster and more efficiently.

In some embodiments, in response to detecting the first input by the contact, in response to a determination that the first input by the contact meets third (eg, staging user interface display) criteria (eg, staging user interface display). Criteria are criteria configured to identify a swipe input, a keep touching input, a tap input, a tap input, or a strong press having an intensity exceeding a predefined intensity threshold), the device may be configured to: Displaying (848) a third user interface area on the display, including replacing the display of at least a portion of the first user interface area (comprising a 3D model of the virtual object that has been replaced by ). In some embodiments, while displaying a staging user interface (e.g., staging user interface 6010 as described in connection with FIG. 6I), the device is described in more detail below (e.g., with reference to method 900). As is) the appearance of the representation of the virtual object is updated based on the detected inputs corresponding to the staging user interface. In some embodiments, if another input is detected while the virtual object is displayed in the staging user interface and the input meets the criteria for transitioning to displaying the second user interface area, the device continuously displays the virtual object The display of the staging user interface is replaced with the second user interface area. Further details are described in connection with method 900. Displaying the third user interface in accordance with the determination that the first input satisfies the third criteria includes the displayed additional controls (eg, controls for displaying a third user interface from the first user interface) to the first user. It provides additional control options without cluttering the interface. Providing additional control options without cluttering the second user interface with the displayed additional controls improves the operability of the device, which additionally enables the user to use the device more quickly and efficiently, thereby It reduces the power usage of the battery and improves battery life.

In some embodiments, responding to a first input by contact (e.g., a swipe input corresponding to scrolling a first user interface area or a request to display an email or web page corresponding to content within the first user interface area) Depending on the determination that the tap input) does not meet the first (e.g., AR trigger) criteria, the device displays at least a portion of the first user interface area (e.g., as described in connection with FIGS. 6B and 6C). The display of the first user interface area is maintained (850) without substituting a representation of the field of view of one or more cameras. Input by using first criteria to determine whether to maintain the display of the first user interface area or to continuously display a representation of the virtual object while replacing the display of at least a portion of the first user interface area with the field of view of one or more cameras. In response to a number of different types of operations. (E.g., by replacing the display of at least a portion of the user interface with the field of view of one or more cameras, or without replacing the display of at least a portion of the first user interface with a representation of the field of view of the one or more cameras. Enabling the performance of a number of different types of actions in response to input (by maintaining the display) increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally: It reduces device power usage and improves battery life by enabling users to use the device more quickly and efficiently.

It should be understood that the specific order in which the operations in FIGS. 8A-8E are described is merely an example and is not intended to indicate that the described order is the only order in which the operations may be performed. One of skill in the art will recognize various ways of reordering the operations described herein. In addition, details of the other processes described herein with respect to other methods described herein (e.g., methods 900, 1000) are described with respect to the method 800 described above with respect to FIGS. 8A-8E. It should be noted that it is also applicable in a similar way. For example, contacts, inputs, virtual objects, user interface regions, intensity thresholds, tactile outputs, fields of view, movements, and/or animations described above with respect to method 800 may optionally be , Contacts, inputs, virtual objects, user interface described herein in connection with other methods described herein (e.g., methods 900, 1000, 16000, 17000, 18000, 19000, 20000) It has one or more of the characteristics of areas, intensity thresholds, tactile outputs, fields of view, movements, and/or animations. For brevity, these details are not repeated here.

9A-9D illustrate a first representation of a virtual object in a first user interface area, a second representation of a virtual object in a second user interface area, and a representation of a field of view of one or more cameras, according to some embodiments. Together are flow charts illustrating a method 900 of displaying a third representation of a virtual object. Method 900 includes an electronic device (e.g., the device of FIG. 3) having a display, a touch-sensitive surface, and one or more cameras (e.g., one or more rear cameras on the side of the device opposite the display and the touch-sensitive surface). 300, or the portable multifunction device 100 of FIG. 1A). In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 900 are selectively combined and/or the order of some operations is selectively changed.

As described below, method 900 relates to detecting input by contact at a touch-sensitive surface of a device displaying a representation of a virtual object within a first user interface (e.g., a two-dimensional graphical user interface). . In response to the first input, the device is configured to display a second representation of the virtual object within a second user interface (e.g., a staging user interface in which the three-dimensional representation of the virtual object can be moved, resized, and/or reoriented) Use criteria to determine whether or not. While displaying the second representation of the virtual object in the second user interface, in response to the second input, the device changes the display property of the second representation of the virtual object based on the second input or Display a third representation of the virtual object within a third user interface that includes the field of view. Enabling the performance of a number of different types of actions in response to an input (e.g., by changing the display properties of the virtual object or by displaying the virtual object in a third user interface) is the efficiency with which the user can perform these actions. And thereby improving the operability of the device, which additionally reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

The device displays a virtual object within a first user interface area (e.g., a two-dimensional graphical user interface or a portion thereof (e.g., a viewable list of furniture images, an image including one or more selectable objects, etc.)) on the display 112. 1 Expression (e.g., a graphic representation of a 3D object, for example, a virtual chair 5020, a virtual lamp 5084, shoes, furniture, hand tools, decorations, people, emoji, game characters, virtual furniture, etc.) Display (902). For example, the first user interface area is the messaging user interface 5008 as shown in FIG. 6A. In some embodiments, the first user interface area includes a background other than an image of the physical environment around the device (e.g., the background of the first user interface area is a preselected background color/pattern, or by one or more cameras. It is a background image distinct from the output image captured at the same time and distinct from the live content within the field of view of one or more cameras).

While displaying the first representation of the virtual object in the first user interface area on the display, the device detects a first input by the first contact at a location on the touch-sensitive surface corresponding to the first representation of the virtual object on the display. (E.g., the first contact is detected on the first representation of the virtual object on the touch screen display, or the first contact is displayed simultaneously in the first user interface area with the first representation of the virtual object and is called by the first contact. When an affordance configured to trigger the display of the staging user interface 6010 including a representation of the AR view (e.g., field of view 6036 of the camera(s)) and/or a virtual object (e.g., virtual chair 5020) For example, it is detected on the toggle control unit 6018) (904). For example, the first input is an input by contact 6006 as described in connection with FIGS. 6E-6I.

In response to detecting a first input by the first contact and according to a determination that the first input by the first contact meets first (e.g., staging trigger) criteria (e.g., the staging trigger criteria are swipe input, Continue to touch input, press input, tap input, touchdown of contact, initial movement of contact, or other types of other types associated with triggering activation of the camera(s) and/or detection of planes of view within the field of view of the camera(s). Criteria configured to identify a predefined input gesture), the device displays a second representation of the virtual object in a second user interface area different from the first user interface area (e.g., the second user interface area is the camera(s)) Is a staging user interface 6010 that does not include a field of view of and includes a simulated three-dimensional space in which a three-dimensional representation of the virtual object can be manipulated (eg, rotated and moved) in response to user input) (906). For example, in FIGS. 6E to 6H, according to the determination that the input by the contact 6006 has a characteristic intensity that increases beyond the deep pressing intensity threshold IT _D , the virtual chair object 5020 is (for example, , Displayed in a staging user interface 6010 (eg, as shown in FIG. 6I) that is distinct from the messaging user interface 5008 (as shown in FIG. 6E).

In some embodiments, in response to detecting the first input and upon determining that the first input satisfies the staging trigger criteria, the device provides the three-dimensional representation of the virtual object as shown in the first user interface area. From one orientation (e.g., the first orientation of the virtual chair 5020 as shown in the messaging user interface 5008 in FIG. 6E) to the virtual on the display oriented independently of the device's current orientation relative to the physical environment around the device. A first showing movement and reorientation to a second orientation determined based on the plane (e.g., a second orientation of the virtual chair 5020 determined based on the stage plane 6014, as shown in FIG. 6I) Display an animated transition. For example, a three-dimensional representation of a virtual object has a predefined orientation (e.g., based on the shape and orientation of the virtual object as shown in a two-dimensional graphical user interface) and/or a distance from the plane, and the staging view ( For example, when transitioning to the staging user interface 6010), the three-dimensional representation is moved, resized, and re-orientated from the original position of the virtual object on the display to a new position on the display (e.g., the center of the virtual stage 6014). And, during or at the end of the movement, the three-dimensional representation is reoriented so that the virtual object is at a fixed angle with respect to the predefined staging virtual plane 6014 defined independently of the physical environment around the device.

While displaying the second representation of the virtual object within the second user interface area, the device detects (908) a second input (eg, input by contact 6034 as illustrated in FIGS. 6Q-6T). In some embodiments, detecting the second input comprises detecting one or more second contacts at a location on the touch screen corresponding to the second representation of the virtual object, the physical environment around the device when invoked by the second contact. Detecting a second contact on the affordance configured to trigger the display of an augmented reality view of the second contact(s), detecting movement of the second contact(s), and/or detecting liftoff of the second contact(s) Includes that. In some embodiments, the second input is a continuation of the first input by the same contact (e.g., the second input is a first input by contact 6006 as illustrated in FIGS. 6E-6I, followed by FIGS. It is an input by contact 6034 as illustrated in FIG. 6T (no lift off of the contact), or a separate input by completely different contact (e.g., the second input is as illustrated in FIGS. 6E-6I). The first input by the same contact 6006 followed by the input by the contact 6034 as illustrated in FIGS. 6Q to 6T (there is a lift-off of the contact), or a continuation of the input by additional contact (e.g. , The second input is the input by the contact 6006 as illustrated in FIGS. 6J to 6L after the first input by the contact 6006 as illustrated in FIGS. 6E to 6I ). For example, the second input may be a continuous swipe input, a second tap input, a second press input, a press input following the first input, a second continuous touch input, a sustained touch continued from the first input, etc. .

In response to detecting the second input, in response to a determination that the second input corresponds to a request to manipulate a virtual object within the second user interface area (e.g., without transitioning to an augmented reality view), the device responds to the second input. Based on the determination that the display property of the second representation of the virtual object in the second user interface area is changed, and the second input corresponds to a request to display the virtual object in the augmented reality environment, the device Display a third representation of the virtual object along with the representation (e.g., the device displays a third user interface that includes a field of view 6036 of one or more cameras, and a physical plane (e.g., a) within the physical environment 5002 around the device. A three-dimensional representation of a virtual object (e.g., virtual chair 5020) is placed on a virtual plane (e.g., floor surface 5038) detected within the field of view of the camera(s) corresponding to the floor) (910). .

In some embodiments, the second input corresponding to the request to manipulate the virtual object in the second user interface area is a second contact at a location on the touch-sensitive surface corresponding to the second representation of the virtual object in the second user interface area. It is a pinch or swipe by (s). For example, the second input is an input by contact 6006 as illustrated in FIGS. 6J-6L or by contacts 6026 and 6030 as illustrated in FIGS. 6N and 6O.

In some embodiments, the second input corresponding to the request to display the virtual object in the augmented reality environment is a tap at or from a location on the touch-sensitive surface corresponding to the representation of the virtual object in the second user interface area. It is a drag input following an input, a tap input, or a continuous touch or tap input. For example, the second input is a deep pressing input by contact 6034 as illustrated in FIGS. 6Q-6T.

In some embodiments, changing the display property of the second representation of the virtual object in the second user interface area based on the second input is about one or more axes (e.g., via a vertical and/or horizontal swipe). Rotating, resizing (e.g., a pinch to resize), tilting about one or more axes (e.g., by tilting the device), (e.g., by moving the device horizontally-this is in some embodiments. Used for analysis of the field of view of one or more cameras to detect one or more planes of view in) changing the viewpoint, and/or changing the color of the representation of the virtual object. For example, changing the display property of the second representation of the virtual object is to rotate the virtual chair 5020 in response to a horizontal swipe gesture by contact 6006 as illustrated in FIGS. 6J and 6K, Rotating virtual chair 5020 in response to a diagonal swipe gesture by contact 6006 as illustrated in FIGS. 6K and 6L, or contacts 6026 and 6030 as illustrated in FIGS. 6N and 6O. And increasing the size of the virtual chair 5020 in response to the de-pinch gesture by ). In some embodiments, the amount by which the display attribute of the second representation of the virtual object is changed is the amount by which the attribute of the second input changes (e.g., the distance or speed of movement by the contact(s), the intensity of the contact, the duration of the contact). Etc.).

In some embodiments, upon determining that the second input corresponds to a request to display the virtual object in the augmented reality environment (e.g., within the field of view 6036 of one or more cameras, as described in connection with FIG. 6T), The device provides the current portion of the physical environment in which the three-dimensional representation of the virtual object is captured within the field of view of one or more cameras from each orientation relative to the virtual plane on the display (e.g., the orientation of the virtual chair 5020 shown in FIG. 6R). Display a second animated transition showing reorientation to a third orientation determined based on the orientation (eg, orientation of virtual chair 5020 shown in FIG. 6T). For example, the three-dimensional representation of the virtual object may be a predefined plane (e.g., floor surface 5038) identified in a live image of the physical environment 5002 where the three-dimensional representation of the virtual object is captured within the field of view of the camera(s). )) (e.g., a physical surface such as a vertical wall or a horizontal floor surface that can support a three-dimensional representation of a virtual object) to be at a fixed angle. In some embodiments, the orientation of the virtual object in the augmented reality view is constrained, in at least one aspect, by the orientation of the virtual object in the staging user interface. For example, the rotation angle of the virtual object about at least one axis of the three-dimensional coordinate system is maintained when transitioning the virtual object from the staging user interface to the augmented reality view (e.g., as described in connection with FIGS. 6Q to 6U). Likewise, rotation of the virtual chair 5020 as described in connection with FIGS. 6J and 6K is maintained). In some embodiments, the source of light cast on the representation of the virtual object in the second user interface area is a virtual light source. In some embodiments, the third representation of the virtual object within the third user interface area is illuminated by a real-world light source (eg, as detected and/or determined from within the field of view of one or more cameras).

In some embodiments, the first criteria correspond to a virtual object indicator 5022 in which the first input (e.g., an indicator, e.g., an icon, displayed adjacent to and/or overlays a representation of the virtual object on the display). Criteria that are satisfied (eg, according to a determination to include) when including a tap input by a first contact at a location on the touch-sensitive surface to be included (912). For example, the virtual object indicator 5022 indicates that the virtual object to which it corresponds is a staging view (e.g., staging user interface 6010) (e.g., as described in more detail below with reference to method 1000) and Provides an indication that it is observable in an augmented reality view (eg, field of view 6036 of the camera(s)). Determining whether to display a second representation of the virtual object in the second user interface area, depending on whether the first input includes a tap input, enables performing a number of different types of operations in response to the first input. Let's do it. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, the first criteria include a movement of less than a threshold amount for at least a predefined threshold time (e.g., a long press time threshold) at a location on the touch-sensitive surface in which the first contact corresponds to the first representation of the virtual object. Contains criteria that are satisfied (eg, according to a determination to be maintained) when maintained at 914. For example, the first criteria are satisfied by a keep touching input. In some embodiments, the first criterion is, after the first contact is maintained with less than a threshold amount of movement for at least a predefined threshold time at a location on the touch-sensitive surface corresponding to the representation of the virtual object in order for the criterion to be met. Includes criteria requiring movement of the first contact. For example, the first criteria are satisfied by a continually touching input followed by a drag input. Whether to display a second representation of the virtual object within the second user interface area, depending on whether the contact is maintained with a movement less than a threshold amount for at least a predefined time at a location on the touch-sensitive surface corresponding to the representation of the virtual object. Determining enables the performance of a number of different types of actions in response to a first input. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, the first criteria are satisfied (e.g., according to a determination to increase) when the characteristic strength of the first contact increases beyond a first intensity threshold (e.g., deep press intensity threshold (IT _D )). It includes the criteria to be (916). For example, as described in connection with FIGS. 6Q-6T, the criteria are that the characteristic strength of the contact 6034 exceeds the deep pressing strength threshold IT _D , as indicated by the intensity level meter 5028. It is satisfied when it increases. In some embodiments, upon determining that the contact satisfies the criteria for recognizing another type of gesture (e.g., tap), the device triggers a second (e.g., staging) user interface while maintaining the display of the virtual object. It performs a predefined function other than what it does. In some embodiments, the first criteria are a strong tap input having an intensity that reaches above the threshold intensity before the first input is detected within the tap time threshold of the initial touchdown of the contact (e.g., a lift-off of the contact ) Not. In some embodiments, the first criteria include a criterion that requires movement of the first contact after the strength of the first contact exceeds a first strength threshold in order for the criterion to be met. For example, the first criteria are satisfied by a click input followed by a drag input. Depending on whether the characteristic intensity of the contact increases beyond the first intensity threshold, determining whether to display the virtual object in the second user interface area can result in the performance of a number of different types of operations in response to the first input. Make it possible. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, in response to detecting a first input by the first contact and according to a determination that the first input by the first contact meets second criteria (eg, interface scroll criteria) The 2 criteria require that the first input include movement of the first contact while exceeding a threshold distance in a direction across the touch-sensitive surface (e.g., the second criteria are by a swipe gesture such as a vertical swipe or a horizontal gesture. Met) -, the device scrolls the first user interface area (and the representation of the virtual object) in a direction corresponding to the direction of movement of the first contact (e.g., the first criteria are not satisfied and within the second user interface area) Displaying the representation of the virtual object is pending) (918). For example, as described in connection with FIGS. 6B and 6C, an upward vertical swipe gesture by contact 6002 causes messaging user interface 5008 and virtual chair 5020 to scroll upward. In some embodiments, the first criteria also require that the first input include movement of the first contact while exceeding a threshold distance in order for the first criteria to be met, and the device selects an object in which the initial portion of the first input is Based on whether or not the first input meets the first criteria (e.g., staging trigger criteria) or second criteria (e.g., based on whether it meets the criteria (e.g., continuous touching or pressing on the representation of the virtual object)) , Interface scroll criteria) are met. In some embodiments, the second criteria are satisfied by a swipe input initiated at the AR icon of the virtual object and a touch location outside the location of the virtual object. Depending on whether the first input satisfies the second criteria, determining whether to scroll the first user interface area in response to the first input enables the performance of a number of different types of operations in response to the first input. Let's do it. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, in response to detecting the first input by the first contact and upon determining that the first input by the first contact meets third (e.g., AR trigger) criteria, the device The third representation of the virtual object is displayed together with the representation of the field of view of the cameras (920). For example, as described in connection with FIGS. 6A to 6AG, a long touch input by contact 6044 followed by an upward drag input by contact 6044 dragging the virtual chair 5020 is a virtual chair 5020 ) Is displayed along with the field of view 6036 of the camera(s).

In some embodiments, the third criteria are, for example, that one or more cameras are active and/or the device orientation is within a defined range (e.g., from a defined origin orientation, a defined rotation angle about one or more axes). And/or the input by contact comprises a selection input (e.g., a long touch) followed by a drag input (movement of the contact that moves the virtual object on the display (e.g., within a predetermined distance from the edge of the display)) and/or Or, the characteristic strength of the contact increases above the AR trigger intensity threshold (e.g., light press threshold (IT _L ) or deep press threshold (IT _D )) and/or the duration of the contact increases by the AR trigger duration threshold (e.g. , And/or the distance traveled by the contact increases beyond an AR trigger distance threshold (eg, a long swipe threshold) and/or is satisfied according to the determination that the distance traveled increases above the AR trigger distance threshold (eg, long swipe threshold). In some embodiments, a control unit (eg, toggle control unit 6018) for displaying a representation of a virtual object in a second user interface area (eg, staging user interface 6010) is Displayed in a user interface (eg, a third user interface area that replaces at least a portion of the second user interface area) including the field of view 6036.

In some embodiments, when transitioning directly from a first user interface area (e.g., non-AR, non-staging, touch screen UI view) to a third user interface area (e.g., augmented reality view), the device A three-dimensional representation of a predefined relative to the current orientation of the portion of the physical environment captured within the field of view of one or more cameras from each orientation represented within the touch screen UI (e.g., non-AR, non-staging view) on the display. Displays an animated transition showing reorientation to orientation. For example, as shown in FIGS. 6A to 6AJ, from a first user interface area (eg, a messaging user interface 5008 as shown in FIG. 6A) to a third user interface area (eg, shown in FIG. 6AJ). When transferred directly to the augmented reality user interface including the field of view 6036 of the camera(s) as shown), the virtual chair 5020 is from a first orientation as shown in FIGS. 6A-6A (e.g., FIG. It is changed to a predefined orientation relative to the floor surface 5038 in the physical environment 5002 as captured within the field of view 6036 of the camera(s) as shown in 6aj). For example, the three-dimensional representation of the virtual object is a predefined plane (e.g., a vertical wall or a vertical wall capable of supporting the three-dimensional representation of the virtual object) in which the three-dimensional representation of the virtual object is identified in the live image of the physical environment 5002. It is reoriented to be at a fixed angle relative to a horizontal floor surface (eg, a physical surface such as floor surface 5038). Depending on whether the first input satisfies the third criteria, determining whether to display a third representation of the virtual object with the field of view of the camera(s) in response to the first input includes a plurality of Allows the performance of different types of operations. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, in response to detecting the first input by the first contact, the device, by one or more device orientation sensors, the current device orientation of the device (eg, orientation relative to the physical environment around the device). And the third criteria (e.g., AR trigger criteria) require that the current device orientation be within the first range of orientations in order for the third criteria to be met (e.g., the second criteria are between the device and the ground. Is satisfied when the angle of is less than the critical angle, indicating that the device is sufficiently parallel to the ground (to bypass the interstitial state) (922). In some embodiments, the first criteria (e.g., staging trigger criteria) require that the current device orientation be within a second range of orientations in order for the first criteria to be met (e.g., the first criteria It is satisfied when the angle between the ground is within the threshold to 90 degrees, indicating that the device is sufficiently upright with respect to the ground and first goes to an intermediate state). Depending on whether the device orientation is within a range of orientations, determining whether to display a third representation of the virtual object along with the field of view of the camera(s) in response to the first input is a number of Allows the performance of different types of operations. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, at least one display attribute of the second representation of the virtual object (e.g., size, shape, yaw axis, pitch axis, and respective angles about the roll axis, etc.) of the virtual object 3 Applies to expression (924). For example, the rotation of the second representation of the virtual chair 5020 applied to the staging user interface 6010 as described in connection with FIGS. 6J and 6K, as described in connection with FIGS. 6Q-6U, is , Maintained when the third representation of the virtual chair 5020 is displayed in the augmented reality view including the field of view 6036 of the camera(s) (eg, as shown in FIG. 6U). In some embodiments, the orientation of the virtual object in the augmented reality view is constrained, in at least one aspect, by the orientation of the virtual object in the staging user interface. For example, the rotation angle of the virtual object about at least one axis (e.g., yaw axis, pitch axis, or roll axis) of a predefined three-dimensional coordinate system is when transitioning the virtual object from the staging view to the augmented reality view. maintain. In some embodiments, at least one display attribute of the second representation of the virtual object is if the second representation of the virtual object has been manipulated in any way (e.g., changing size, shape, texture, orientation, etc.) by user input. Applies only to the third representation of the virtual object. In other words, changes made in the staging view are maintained when the object is viewed in the augmented reality view or used to constrain the appearance of the object in the augmented reality view in one or more ways. (E.g., without requiring an additional user input to apply the same display property to the second representation of the virtual object and the third representation of the virtual object) at least one display property of the second representation of the virtual object 3 Applying to the representation (e.g., allows the user to apply a rotation to the second virtual object while a large form of the virtual object is displayed in the second user interface, and allows the user to apply a rotation to the second virtual object, By applying rotation to the third representation) it improves the operability of the device, which additionally reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, detecting at least an initial portion of the first input by the first contact (e.g., detecting the first contact, or meeting each predefined criteria without meeting the first criteria). Responsive to detecting an input by a first contact, or detecting an input that meets the first criteria), the device activates one or more cameras (e.g., the field of view of the camera(s) on the display). Activating the camera(s) without displaying immediately), the device analyzes 926 the field of view of one or more cameras to detect one or more planes within the field of view of the one or more cameras. In some embodiments, displaying the field of view 6036 of one or more cameras after activating one or more cameras (e.g., until a second input corresponding to a request to display a virtual object in an augmented reality environment is detected, It is delayed until at least one viewing plane is detected, or until a viewing plane corresponding to an anchor plane defined for the virtual object is detected). In some embodiments, the field of view 6036 of one or more cameras is displayed (eg, concurrently with the activation) at a time corresponding to the activation of the one or more cameras. In some embodiments, the field of view 6036 of one or more cameras is displayed before a plane is detected within the field of view of the one or more cameras (e.g., the field of view of one or more cameras is in response to detecting the first input by contact. And displayed according to the decision). To activate the camera(s) and detect one or more planes of view in response to detecting an initial portion of the first input (e.g., prior to displaying a third representation of the virtual object with a representation of the field of view of one or more cameras) Analyzing the field of view of the camera(s) is the efficiency of the device (e.g., by reducing the time required to determine the position and/or orientation of the third representation of the virtual object relative to each plane within the field of view of the camera(s)). And, in addition, it reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, in response to detecting each plane (e.g., floor surface 5038) within the field of view of the one or more cameras, the device tactiles to indicate detection of each plane within the field of view of the one or more cameras. The enemy output is output (928), by one or more tactile output generators 167. In some embodiments, the field of view 6036 may be viewed before the plane of view is identified. In some embodiments, after at least one viewing plane is detected or after all of the viewing planes have been identified, additional user interface controls and/or icons are overlaid on the real world image within the field of view. Outputting a tactile output to indicate detection of a plane in the field of view of the camera(s) provides feedback to the user indicating that the plane has been detected. Providing improved tactile feedback improves the operability of the device (e.g., by helping the user provide appropriate inputs and reducing unnecessary additional inputs to place the virtual object), which additionally allows the user to It reduces the power usage of the device and improves battery life by making it possible to use the device more quickly and efficiently.

In some embodiments, the size of the third representation of the virtual object on the display is a location within the field of view 6036 of one or more cameras having a fixed spatial relationship with the third representation of the virtual object (e.g., virtual chair 5020). For example, it is determined based on the distance between the one or more cameras and the simulated real-world size of the virtual object (930) between the plane of the floor surface 5038 to which the virtual object is attached). In some embodiments, the size of the third representation of the virtual object is constrained such that a scale of the size of the third representation of the virtual object with respect to the field of view of one or more cameras is maintained. In some embodiments, one or more physical dimension parameters (eg, length, width, depth, and/or radius) are defined for the virtual object. In some embodiments, in the second user interface (e.g., staging user interface), the virtual object is not constrained by its limited physical dimension parameters (e.g., the size of the virtual object is changeable in response to user input). . In some embodiments, the third representation of the virtual object is constrained by its finite dimensional parameters. When a user input to change the position of a virtual object in an augmented reality view of the physical environment represented within the field of view is detected, or when a user input to change the zoom level of the field of view is detected, or around the device. When a user input moving relative to the physical environment is detected, the appearance (e.g., size, viewpoint) of the virtual object is (e.g., as represented by a fixed spatial relationship between the anchor plane of the virtual object and the augmented reality environment). ) A fixed spatial relationship between the virtual object and the physical environment, and a fixed scale based on the predefined dimensional parameters of the virtual object and the actual dimensions of the physical environment. (E.g., without requiring additional user input to resize a third representation of the virtual object to simulate the real world size of the virtual object) and the distance between one or more cameras and a location within the field of view of the camera(s) and the virtual Determining the size of the third representation of the virtual object based on the simulated real-world size of the object improves the operability of the device, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the power of the device. Reduces usage and improves battery life.

In some embodiments, the second input corresponding to the request to display the virtual object in the augmented reality environment selects (selects) a second representation of the virtual object (e.g., exceeds a distance threshold, crosses a defined boundary, and And/or an input of dragging an increasing distance to a location within a threshold distance of the display or edge of the second user interface area (eg, lower edge, upper edge, and/or side edge) (932). In response to detecting a second input corresponding to a request to display the virtual object in the augmented reality environment, displaying the third representation of the virtual object along with the representation of the field of view of the camera(s) may include displayed additional controls (e.g., Controls for displaying the augmented reality environment from the second user interface) to provide additional control options without confusing the second user interface. Providing additional control options without cluttering the second user interface with the displayed additional controls improves the operability of the device, which additionally enables the user to use the device more quickly and efficiently, thereby It reduces the power usage of the battery and improves battery life.

In some embodiments, while displaying the second representation of the virtual object in the second user interface area (e.g., staging user interface 6010 as shown in FIG. 6Z), the device redisplays the first user interface area. A fourth input that satisfies each of the criteria for making (e.g., a location on the touch-sensitive surface corresponding to the second representation of the virtual object or another location on the touch-sensitive surface (e.g., the bottom or edge of the second user interface area) Tap, strong press, or continuous touch and drag input, and/or input at a location on the touch-sensitive surface corresponding to the control to return to the first user interface area), and detect a fourth input In response to doing so, the device stops displaying the second representation of the virtual object in the second user interface area, and the device redisplays (934) the first representation of the virtual object in the first user interface area. For example, in response to an input by contact 6042 at a location corresponding to the back control 6016 displayed on the staging user interface 6010, as shown in FIGS. 6Z to 6AC, the device 2 Stop displaying the second representation of the virtual chair 5020 in the user interface area (e.g., staging user interface 6010), and the device stops displaying the virtual chair in the first user interface area (e.g., messaging user interface 5008). The first representation of the chair 5020 is redisplayed. In some embodiments, the first representation of the virtual object is displayed within the first user interface area with the same appearance, location, and/or orientation as those seen before transitioning to the staging view and/or augmented reality view. For example, in FIG. 6Ac, the virtual chair 5020 is displayed in the messaging user interface 5008 in the same orientation as the virtual chair 5020 displayed in the messaging user interface 5008 in FIG. 6A. In some embodiments, the device continuously displays the virtual object on the screen when transitioning back to displaying the virtual object within the first user interface area. For example, in FIGS. 6Y-6C, virtual chair 5020 is continuously displayed during the transition from displaying staging user interface 6010 to displaying messaging user interface 5008. According to whether the fourth input detected while displaying the second representation of the virtual object in the second user interface satisfies the criteria for redisplaying the first user interface, the first representation of the virtual object is displayed in the first user interface. Determining whether to redisplay or not enables the performance of a number of different types of operations in response to a fourth input. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, while displaying a third representation of the virtual object with a representation of the field of view 5036 of one or more cameras (e.g., as shown in FIG. 6U), the device redisplays the second user interface area. A fifth input that satisfies each of the criteria for making (e.g., a tap, a strong press, or a touch and drag input at a location on the touch-sensitive surface corresponding to the third representation of the virtual object or another location on the touch-sensitive surface, And/or an input at a location on the touch-sensitive surface corresponding to the control for returning to displaying the second user interface area), and in response to detecting the fifth input, the device Stop displaying the representation and the representation of the field of view of the one or more cameras, and redisplay (936) a second representation of the virtual object within the second user interface area. For example, as shown in Figs. 6V to 6Y, the contact 6040 at a position corresponding to the toggle control 6018 displayed on the third user interface including the field of view 6036 of the camera(s) In response to the input by, the device stops displaying the field of view 6036 of the camera(s) and redisplays the staging user interface 6010. In some embodiments, the second representation of the virtual object is displayed within the second user interface area in the same orientation as shown in the augmented reality view. In some embodiments, the device continuously displays the virtual object on the screen when transitioning back to displaying the virtual object within the second user interface area. For example, in FIGS. 6V-6Y, virtual chair 5020 is continuously displayed during the transition from displaying field of view 6036 of the camera(s) to displaying staging user interface 6010. Depending on whether the fifth input detected while displaying the third representation of the virtual object with the field of view of the camera(s) meets the criteria for redisplaying the second user interface, the second user interface 2 Determining whether to redisplay the representation enables the performance of a number of different types of operations in response to the fifth input. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, while displaying the third representation of the virtual object along with the representation 6036 of the field of view of one or more cameras, the device redisplays the first user interface area (e.g., messaging user interface 5008). In response to detecting a sixth input that meets respective criteria for, and in response to detecting the sixth input, the device (e.g., as shown in FIG. 6U) of the virtual object (e.g., virtual chair 5020). 3 stops displaying the representation and the representation of the field of view 6036 of one or more cameras, and the device redisplays the first representation of the virtual object in the first user interface area (e.g., as shown in FIG. 938). In some embodiments, the sixth input is, for example, a tap, strong press, or touch and drag input at a location on the touch-sensitive surface or other location on the touch-sensitive surface corresponding to the third representation of the virtual object, and /Or input at a location on the touch-sensitive surface corresponding to the control for returning to displaying the first user interface area. In some embodiments, the first representation of the virtual object is displayed within the first user interface area with the same appearance and location as those seen before transitioning to the staging view and/or augmented reality view. In some embodiments, the device continuously displays the virtual object on the screen when transitioning back to displaying the virtual object within the first user interface area. Depending on whether the sixth input detected while displaying the third representation of the virtual object with the field of view of the camera(s) satisfies the criteria for redisplaying the first user interface, the second input of the virtual object within the first user interface. Determining whether to redisplay the 1 representation enables the performance of a number of different types of operations in response to the sixth input. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, in response to detecting the first input by the first contact and upon determining that the input by the first contact meets the first criteria, the device A first user comprising displaying an animation (e.g., movement, rotation around one or more axes, and/or scaling) in which the first representation is converted to a second representation of the virtual object within the second user interface area. When transitioning from displaying an interface area (eg, messaging user interface 5008) to displaying a second user interface area (eg, staging user interface 6010), the virtual object is continuously displayed (940). For example, in Figures 6E-6I, the virtual chair 5020 is continuously displayed and animated during the transition from displaying the messaging user interface 5008 to displaying the staging user interface 6010 (e.g., The orientation of the virtual chair 5020 changes). In some embodiments, the virtual object has a defined orientation relative to a plane within the field of view of the camera(s) (e.g., defined based on the shape and orientation of the first representation of the virtual object as seen in the first user interface area). , Position and/or distance, and when transitioning to the second user interface area, the first representation of the virtual object is the first representation of the virtual object at a new location on the display (e.g., the center of the virtual staging plane in the second user interface area). 2 When moving, resizing, and/or reorienting to the representation, during the movement or at the end of the movement, the virtual object is at a predetermined angle to a predefined virtual staging plane in which the virtual object is defined independently of the physical environment around the device. To be reorientated Displaying the animation as the first representation of the virtual object in the first user interface is converted to a second representation of the virtual object in the second user interface provides feedback to the user indicating that the first input meets the first criteria. . Providing improved feedback improves the operability of the device (e.g., by helping the user to provide appropriate inputs when operating/interacting with the device and reducing user errors), which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use the device more quickly and efficiently.

In some embodiments, in response to detecting a second input by the second contact and upon determining that the second input by the second contact corresponds to a request to display the virtual object in the augmented reality environment, the device: Animation in which the second representation of the virtual object in the second user interface area is converted into a third representation of the virtual object in the third user interface area including the field of view of one or more cameras (e.g., movement, rotation around one or more axes) A third user interface region comprising a field of view 6036 of one or more cameras from displaying a second user interface region (e.g., staging user interface 6010), including displaying, and/or scaling). The virtual object is continuously displayed when it transitions to displaying (942). For example, in Figures 6Q-6U, the virtual chair 5020 is continuously displayed and animated during the transition from displaying the staging user interface 6010 to displaying the field of view 6036 of the camera(s). (For example, the position and size of the virtual chair 5020 are changed). In some embodiments, the virtual object is located on a viewing plane (e.g., a physical surface such as a vertical wall or horizontal floor surface that can support a three-dimensional representation of a user interface object) where the virtual object is detected within the field of view of one or more cameras It is reoriented to be in a predefined orientation, location, and/or distance to. Displaying the animation as the second representation of the virtual object in the second user interface is converted to a third representation of the virtual object in the third user interface indicates that the second input corresponds to a request to display the virtual object in an augmented reality environment. Provide the user with the feedback that indicates. Providing improved visual feedback to the user improves the operability of the device (e.g., by helping the user to provide appropriate inputs when operating/interacting with the device and reducing user mistakes), which additionally , Reducing the power usage of the device and improving battery life by enabling users to use the device more quickly and efficiently.

It should be understood that the specific order in which the operations in FIGS. 9A-9D are described is merely an example and is not intended to indicate that the described order is the only order in which the operations may be performed. One of skill in the art will recognize various ways of reordering the operations described herein. Additionally, details of other processes described herein in connection with other methods described herein (e.g., methods 800, 900, 16000, 17000, 18000, 19000, 20000) are provided in FIGS. It should be noted that it is also applicable in a manner similar to method 900 described above with respect to FIG. 9D. For example, contacts, inputs, virtual objects, user interface regions, intensity thresholds, fields of view, tactile outputs, movements, and/or animations described above with respect to method 900 can optionally be , Contacts, inputs, virtual objects, user interface described herein in connection with other methods described herein (e.g., methods 800, 900, 16000, 17000, 18000, 19000, 20000) It has one or more of the characteristics of areas, intensity thresholds, fields of view, tactile outputs, movements, and/or animations. For brevity, these details are not repeated here.

10A-10D are flow charts illustrating a method 1000 of displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object, according to some embodiments. Method 1000 includes an electronic device (e.g., device 300 of FIG. 3, or portable device 300 of FIG. It is performed on the multifunctional device 100). In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 1000 are selectively combined and/or the order of some operations is selectively changed.

As described below, method 1000 is directed to displaying items in first and second user interfaces. Each item is displayed with or without a visual indication indicating that the item corresponds to a virtual three-dimensional object, depending on whether the item corresponds to a respective virtual three-dimensional object. Providing the user with an indication of whether the item is a virtual three-dimensional object (e.g., by helping the user provide appropriate inputs depending on whether the item is a virtual three-dimensional object or not) allows the user to perform actions on the first item. It increases the efficiency it can perform, thereby improving the operability of the device, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently. .

The device receives a request to display a first user interface including a first item (eg, an icon, a thumbnail image, an image, an emoji, an attachment, a sticker, an app icon, an avatar, etc.) (1002). For example, in some embodiments, the request is a user interface for displaying a representation of the first item in a predefined environment associated with the first item (e.g., Internet browser user interface 5060 as shown in FIG. 7B). ) To open (eg, as described in connection with FIG. 7A) The predefined environment is optionally, a user interface of an application (e.g., an email application, a messaging application, a browser application, a word processing application, an e-book application, etc.) or a system user interface (e.g., a lock screen, a notification interface, a suggestion interface, a control panel). User interface, home screen user interface, etc.).

In response to the request to display the first user interface, the device displays 1004 a first user interface (eg, an internet browser user interface 5060 as shown in FIG. 7B) with a representation of the first item. Upon determining that the first item corresponds to each virtual three-dimensional object, the device determines that the first item is an icon and/or displayed at a position corresponding to the first respective virtual three-dimensional object (e.g. Or display a representation of the first item with a visual indication indicating that it corresponds to an image such as a background panel; an outline; and/or text). Upon determining that the first item does not correspond to each virtual three-dimensional object, the device displays a representation of the first item without a visual indication. For example, as illustrated in FIG. 7B, in the Internet browser user interface 5060, the web object 5068 (including the representation of the virtual three-dimensional lamp object 5084) is the virtual lamp 8084 It is displayed with a visual indication indicating that it is a dimensional object (virtual object indicator 5080) and the web object 5074 does not contain a visual object indicator because the web object 5074 does not contain an item corresponding to the virtual three-dimensional object Is displayed.

After displaying the expression of the first item, the device displays a second user interface (e.g., FIG. 7M) including a second item (e.g., icon, thumbnail image, image, emoji, attachment, sticker, app icon, avatar, etc.) A request to display a messaging user interface 5008 as illustrated in (eg, input as described in connection with FIGS. 7H-7L) is received (1006). The second item is distinguished from the first item, and the second user interface is distinguished from the first user interface. For example, in some embodiments, the request is another input to open a user interface for displaying a representation of the second item in a predefined environment associated with the second item. The predefined environment optionally includes a user interface of an application other than the application used to display the first item (e.g., email application, messaging application, browser application, word processing application, e-book application, etc.), or the first item. It is a system user interface other than the system user interface (eg, lock screen, notification interface, suggestion interface, control panel user interface, home screen user interface, etc.) used to display.

In response to the request to display the second user interface, the device displays 1008 a second user interface (eg, messaging user interface 5008 as shown in FIG. 7M) with a representation of the second item. Upon determination that the second item corresponds to each virtual three-dimensional object, the device displays a visual indication that the second item corresponds to each second virtual three-dimensional object (e.g., the first item corresponds to the virtual three-dimensional object). The representation of the second item together with the same visual indication indicating that they correspond. Upon determining that the second item does not correspond to each virtual three-dimensional object, the device displays a representation of the second item without a visual indication. For example, as illustrated in FIG. 7M, in the messaging user interface 5008, the virtual three-dimensional chair object 5020 is a visual display indicating that the virtual chair 5020 is a virtual three-dimensional object (virtual object indicator ( 5022)), and the emoji 7020 is displayed without a visual object indicator because the emoji 7020 does not include an item corresponding to a virtual three-dimensional object.

In some embodiments, a first item (e.g., virtual lamp 5084) with a visual indication (e.g., virtual object indicator 5080) indicating that the first item corresponds to a first respective virtual three-dimensional object. Displaying a representation of the device causing a change from the first device orientation to the second device orientation (e.g., as detected by an orientation sensor (e.g., one or more accelerometers 168 of device 100)) In response to detecting the movement of the first item corresponding to the change from the first device orientation to the second device orientation (e.g., tilting of the first item and/or the first item relative to the first user interface) (1010). For example, the first device orientation is the orientation of the device 100 as illustrated in FIG. 7F1, and the second device orientation is the orientation of the device 100 as illustrated in FIG. 7G1. In response to the movement illustrated by FIG. 7G1 in FIG. 7F1, the first item (eg, virtual lamp 5084) is tilted (eg, as illustrated by FIG. 7G2 in FIG. 7F2 ). In some embodiments, if the second object corresponds to a virtual three-dimensional object, the second object is also used in the manner described above (e.g., to indicate that the second object also corresponds to a virtual three-dimensional object). Respond to detecting movement.

Displaying the movement of the first item corresponding to the change from the first device orientation to the second devices orientation provides the user with visual feedback indicative of the behavior of the virtual three-dimensional object. Providing improved visual feedback to the user improves the operability of the device (e.g., by allowing the user to observe a virtual three-dimensional object from certain orientations without having to provide additional input), which additionally allows the user It reduces device power usage and improves battery life by enabling faster and more efficient use.

In some embodiments, displaying the representation of the first item with a visual indication indicating that the first item corresponds to each of the first virtual three-dimensional objects means that the representation of the first item is displayed in the first user interface. Detecting a first input (e.g., a swipe input in a first direction on the first user interface, or a continuous touch input for a scroll button on the end of the scroll bar) by a first contact scrolling the first user interface while In response to doing so, the device translating the representation of the first item on the display according to the scrolling of the first user interface (e.g., by a distance based on the amount of scroll made for the first user interface and in the direction opposite to the first user interface). Moving the anchor position of the item (e.g., when the first user interface is dragged upwards by a moving contact across the touch-sensitive surface, the representation of the first item is upwards on the display with the first user interface). Moving)) and rotating the representation of the first item with respect to a plane defined by the first user interface (or display) according to the direction in which the first user interface is scrolled (1012). For example, as illustrated in FIGS. 7C and 7D, by contact 7002 scrolling the internet browser user interface 5060 while the representation of the virtual lamp 5084 is displayed in the internet browser user interface 5060. In response to detecting the input, the virtual lamp 5084 is translated according to the scrolling of the internet browser user interface 5060, and the virtual lamp 5084 is directed to the display 112 according to the direction of the travel path of the contact 7002. Is rotated about. In some embodiments, according to the determination that the first user interface is dragged upward, the representation of the first item moves upward with the first user interface, and the viewing point of the first item as shown on the first user interface. Changes as if the user is viewing the first item from a different viewing angle (eg, a lower angle). In some embodiments, upon the determination that the second user interface is dragged upward, the representation of the second item moves upward along with the second user interface, and the viewing point of the second item as shown on the second user interface. Changes as if the user is viewing the second item from a different viewing angle (eg, a lower angle).

Displaying the movement of the item in which the movement corresponds to a change from the first device orientation to the second device orientation provides visual feedback to the user indicative of the change in device orientation. Providing improved visual feedback to the user improves the operability of the device (e.g., by allowing the user to observe a virtual three-dimensional object from certain orientations without having to provide additional input), which additionally allows the user It reduces device power usage and improves battery life by enabling faster and more efficient use.

In some embodiments, a first item (e.g., a first item (e.g., a visual object indicator 5080)) within a first user interface (e.g., internet browser user interface 5060 as illustrated in FIG. 7B) While displaying the representation of the lamp object 5084), the device displays the representation of the third item, the representation of the third item being displayed without a visual indication to indicate that the third item does not correspond to a virtual three-dimensional object. (E.g., the third item does not correspond to any three-dimensional object that can be rendered in an augmented reality environment) (1014). For example, as illustrated in FIG. 7B, in the internet browser user interface 5060, the web objects 5074, 5070, 5076 correspond to the virtual 3D objects 5074, 5070, 5076. Because it does not, it is displayed without visual object indicators.

Displaying, in the first user interface, a first item with a visual indication indicating that the first item is a virtual three-dimensional object and a third item displayed without a visual indication (e.g., the item the user is interacting with is By helping the user to provide appropriate inputs depending on whether it is a virtual 3D object or not) increases the efficiency with which the user can perform operations using the first user interface, thereby improving the operability of the device, which In addition, it reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, a second item (e.g., virtual object indicator 5022) with a visual indication (e.g., virtual object indicator 5022) within a second user interface (e.g., messaging user interface 5008 as illustrated in FIG. While displaying the representation of the chair 5020), the device displays the representation of a fourth item (e.g., emoji 7020), and the representation of the fourth item corresponds to each virtual 3D object. It is displayed 1016 without a visual indication to indicate that it does not.

Displaying, in the second user interface, a second item with a visual indication indicating that the second item is a virtual three-dimensional object and a fourth item displayed without a visual indication (e.g., the item the user is interacting with is By helping the user provide appropriate inputs depending on whether it is a virtual 3D object or not), it increases the efficiency with which the user can perform operations using the second user interface, thereby improving the operability of the device, which In addition, it reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, a first user interface (e.g., Internet browser user interface 5060 as illustrated in FIG. 7B) corresponds to a first application (e.g., Internet browser application), and a second user interface (e.g., The messaging user interface 5008 as illustrated in FIG. 7M corresponds to a second application (e.g., a messaging application) that is distinct from the first application, and is displayed with a visual indication (e.g., virtual object indicator 5080). The expression of the first item (eg, the lamp object 5084) and the expression of the second item (eg, the virtual chair 5020) displayed together with the visual display (eg, the virtual object indicator 5022) are predefined. Share a set of visual and/or behavioral characteristics (e.g., using the same indicator icon, or having the same texture or rendering style, and/or behavior when invoked by predefined types of inputs) (1018). For example, the icons for the virtual object indicator 5080 and the virtual object indicator 5022 include the same symbol.

Displaying the first item with a visual indication in the first user interface of the first application and displaying the second item together with the visual indication in the second user interface of the second application to display the visual indications of the first item and the second item in advance. Allowing the user to share a defined set of visual characteristics and/or behavioral characteristics (e.g., by helping the user provide appropriate inputs depending on whether the item the user is interacting with is a virtual three-dimensional object) allows the user to second Increases the efficiency of performing operations using the user interface, thereby improving the operability of the device, which additionally reduces the power usage of the device by enabling the user to use the device more quickly and efficiently. Reduce and improve battery life.

In some embodiments, the first user interface is an Internet browser application user interface (e.g., Internet browser user interface 5060 as illustrated in FIG. 7B), and the first item is an element of a web page (e.g., a first Items are represented in web pages as embedded images, hyperlinks, applets, emojis, embedded media objects, etc.) (1020). For example, the first item is the virtual lamp object 5084 of the web object 5068.

Displaying a web page element with a visual indication that the web page element is a virtual three-dimensional object (e.g., allowing the user to provide appropriate inputs depending on whether or not the web page element the user is interacting with is a virtual three-dimensional object). As an aid) increases the efficiency with which the user can perform operations using the Internet browser application, thereby improving the operability of the device, which additionally enables the user to use the device more quickly and efficiently. This reduces device power usage and improves battery life.

In some embodiments, the first user interface is an email application user interface (eg, email user interface 7052 as illustrated in FIG. 7P), and the first item is an attachment to an email (eg, attachment 7060). (1022).

Displaying an email attachment with a visual indication that the email attachment is a virtual 3D object (e.g., by helping the user provide appropriate inputs depending on whether the email attachment the user is interacting with is a virtual 3D object or not) It increases the efficiency with which the user can perform actions using the email application user interface, thereby improving the operability of the device, which additionally enables the user to use the device more quickly and efficiently, thereby It reduces the power usage of the battery and improves battery life.

In some embodiments, the first user interface is a messaging application user interface (e.g., messaging user interface 5008 as illustrated in FIG. 7M), and the first item is an attachment or element in the message (e.g., virtual chair 5020). )) (e.g., the first item is an image, hyperlink, mini program, emoji, media object, etc.) 1024.

Displaying a message attachment or element with a visual indication that the message attachment or element is a virtual three-dimensional object (e.g., depending on whether the message attachment or element the user is interacting with is a virtual three-dimensional object (By helping to provide users) increases the efficiency with which the user can perform operations using the messaging user interface, thereby improving the operability of the device, which in addition allows the user to use the device more quickly and efficiently. By making it possible to reduce the power usage of the device and improve battery life.

In some embodiments, the first user interface is a file management application user interface (e.g., file management user interface 7036 as illustrated in FIG. 7O), and the first item is a file preview object (e.g., file information area). The file preview object 7045 in 7046) is 1026.

Displaying the file preview object with a visual indication that the file preview object is a virtual 3D object (e.g., depending on whether the file preview object the user is interacting with is a virtual 3D object, the user can input By helping to provide the user with the file management application user interface), thereby increasing the efficiency of the user's ability to perform operations, thereby improving the operability of the device, which additionally allows the user to access the device more quickly and efficiently. By making it possible to use, it reduces the device's power usage and improves battery life.

In some embodiments, the first user interface is a map application user interface (e.g., map application user interface 7024), and the first item is a representation of a point of interest (e.g., point of interest object 7028) in the map (e.g. , Resulting in the display of a three-dimensional representation of a feature corresponding to a certain location on the map (e.g., including a three-dimensional representation of a terrain and/or structure corresponding to a certain location on the map) or a three-dimensional representation of the map when activated. It is a three-dimensional representation of the control unit) (1028).

Displaying the representation of the point of interest with a visual indication indicating that the representation of the point of interest in the map is a virtual 3D object (e.g., depending on whether the representation of the point of interest the user is interacting with is a virtual 3D object By helping to provide appropriate inputs) the user can perform actions using the Maps application user interface, thereby increasing the efficiency of the device's operability, which in addition allows the user to make the device faster and more efficient. By making it possible to use it, it reduces the power usage of the device and improves battery life.

In some embodiments, the visual indication that the first item corresponds to each virtual three-dimensional object is an animation of the first item (e.g., over time) that occurs without requiring input for the representation of each three-dimensional object Accordingly, a continuous moving or changing visual effect (eg, sparkling, shimmering, etc.) applied to the first item is included (1030).

Displaying an animation of the first item that occurs without input to the representation of each three-dimensional object (e.g., by reducing the number of inputs required for the user to view three-dimensional aspects of the first item) enables the operability of the device. Improves, and in addition, reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, representation of the second item (e.g., virtual chair 5020) with a visual indication (e.g., virtual object indicator 5022) indicating that the second item corresponds to each virtual three-dimensional object. During displaying, the device detects a second input by the second contact (e.g., an input as described in connection with FIGS. 5C-5F) at a location on the touch-sensitive surface corresponding to the representation of the second item, and , In response to detecting the second input by the second contact and upon determining that the second input by the second contact meets the first (eg, AR trigger) criteria, the device is configured with a second user interface (eg, , Replacing the display of at least a portion of the messaging user interface 5008 with a representation of the field of view 5036 of one or more cameras (e.g., described in connection with FIGS. 5F-5I) and displaying a second user interface. Displaying the third user interface area on the display, including continuously displaying the second virtual three-dimensional object while switching from one to displaying the third user interface area (1032). (Eg, as described in more detail herein with reference to method 800). In some embodiments, the device switches from displaying a portion of the second user interface with a representation of the field of view of one or more cameras (e.g., as described in more detail herein with reference to operation 834). While the representation of the virtual object is continuously displayed, the animation is displayed.

Using the first criteria to determine whether to display the third user interface area enables the performance of a number of different types of operations in response to a second input. Enabling the performance of multiple different types of actions in response to input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use it quickly and efficiently.

In some embodiments, a visual indication (e.g., a virtual object indicator) indicating that the second item corresponds to each virtual three-dimensional object (e.g., as described in more detail herein with reference to method 900). (5022)), while displaying a second item (e.g., virtual chair 5020), the device is configured with a third input (e.g., a third touch) at a location on the touch-sensitive surface corresponding to the representation of the second item. , In response to detecting an input as described in connection with FIGS. 6E-6I ), and in response to detecting a third input by the third contact and by the third contact is a first (e.g., staging trigger) ) Upon determining that the criteria are met, the device is placed in a second virtual three-dimensional in a fourth user interface different from the second user interface (e.g., staging user interface 6010 as described in more detail with reference to method 900). The object is displayed (1034). In some embodiments, while displaying a second virtual three-dimensional object in a fourth user interface (e.g., staging user interface 6010 as illustrated in FIG. 6I), the device detects a fourth input and In response to detecting the input, upon determining that the fourth input corresponds to a request to manipulate the second virtual three-dimensional object in the fourth user interface, the device (e.g., as described in connection with FIGS. Likewise, and/or as described in connection with FIGS. 6N to 6P) based on the fourth input) the display property of the second virtual 3D object in the fourth user interface is changed, and the fourth input is applied to the augmented reality environment. A request to display a second virtual object (e.g., a tap input at or from a location on a touch-sensitive surface corresponding to the representation of the virtual object in the second user interface area, a press input, or a continuous touch or press input followed by Drag input), the device displays a second virtual three-dimensional object with a representation of the field of view of one or more cameras (eg, as described in connection with FIGS. 6Q-6U ).

While displaying the second 3D object in the fourth user interface (eg, staging user interface 6010), in response to the fourth input, the device changes display properties of the second 3D object based on the fourth input. Or display a second three-dimensional object with a representation of the field of view of one or more cameras of the device. (E.g., by changing the display characteristic of the second 3D object or by displaying the second 3D object with a representation of the field of view of one or more cameras of the device) enabling the performance of a number of different types of actions in response to an input It increases the efficiency with which the user can perform these operations, thereby improving the operability of the device, which additionally reduces the power usage of the device by allowing the user to use the device more quickly and efficiently. And improve battery life.

It should be understood that the specific order in which the operations in FIGS. 10A-10D are described is merely an example and is not intended to indicate that the described order is the only order in which the operations may be performed. One of skill in the art will recognize various ways of reordering the operations described herein. Additionally, details of other processes described herein in connection with other methods described herein (e.g., methods 800, 900, 16000, 17000, 18000, 19000, 20000) are provided in FIGS. It should be noted that it is also applicable in a manner similar to method 1000 described above with respect to FIG. 10D. For example, contacts, inputs, virtual objects, user interfaces, user interface regions, fields of view, movements, and/or animations described above with respect to method 1000 are optionally described herein. Contacts, inputs, virtual objects, user interfaces, user interface areas described herein in connection with other methods (e.g., methods 800, 900, 16000, 17000, 18000, 19000, 20000) Have one or more of the characteristics of fields, fields of view, movements, and/or animations. For brevity, these details are not repeated here.

11A-11V illustrate example user interfaces for displaying a virtual object having different visual properties depending on whether object placement criteria are met. User interfaces in these figures are in FIGS. 8A to 8E, 9A to 9D, 10A to 10D, 16A to 16G, 17A to 17D, 18A to 18I, 19A to 19H, and FIG. It is used to illustrate the processes described below, including the processes of FIGS. 20A-20F. For ease of description, some of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is, optionally: an individual finger or stylus contact, or a representative point corresponding to the finger or stylus contact (e.g., the center of the individual contact, or a point associated with the individual contact), or It is the center of two or more contacts detected on the touch-sensitive display system 112. However, similar operations may be performed in response to detecting contacts on the touch-sensitive surface 451, optionally displaying the user interfaces shown in the figure with the focus selector on the display 450, and It is performed on a device with a separate touch-sensitive surface 451.

11A to 11E illustrate inputs for displaying a virtual object in a staging view. For example, the input is a two-dimensional representation of a three-dimensional object (e.g., a thumbnail) is a user interface (e.g., email user interface 7052, file management user interface 7036), map user interface 7022, messaging user interface ( 5008), Internet browser user interface 5060, or third party application user interface).

In Fig. 11A, the Internet browser user interface 5060 includes a two-dimensional representation of a three-dimensional virtual object 11002 (chair). An input (eg, tap input) by the contact 11004 is detected at a position corresponding to the virtual object 11002. In response to the tap input, the display of the Internet browser user interface 5060 is replaced with the display of the staging user interface 6010.

11B-11E illustrate the transition that occurs as the Internet browser user interface 5060 is replaced with the display of the staging user interface 6010. In some embodiments, during transition, the virtual object 10002 gradually fades in to the view and/or the controls of the staging user interface 6010 (e.g., back control 6016, toggle control 6018, and/or Alternatively, the shared control unit 6020 gradually fades in to the view. For example, the staging user interface 6010 after the virtual object 11002 fades in to the view (e.g., to delay the display of the controllers for a period necessary for the three-dimensional representation of the virtual object 11002 to be rendered on the display). )'S controls fade into the view. In some embodiments, the “fade in” of the virtual object 11002 represents a final three-dimensional representation of the virtual object 11002 following displaying the low-resolution, two-dimensional, and/or holographic shape of the virtual object 11002. Includes displaying. 11B-11D illustrate the gradual fade-in of the virtual object 11002. In FIG. 11D, the shadow 11006 of the virtual object 11002 is displayed. 11D and 11E illustrate a gradual fade-in of the controllers 6016, 6018, and 6020.

11F and 11G illustrate an input that causes a three-dimensional representation of a virtual object 11002 to be displayed in a user interface including a field of view 6036 of one or more cameras of the device 100. In FIG. 11F, an input by contact 11008 is detected at a position corresponding to toggle control 6018. In response to the input, the display of the staging user interface 6010 is replaced with the display of the user interface including the field of view 6036 of the camera(s), as shown in FIG. 11G.

As illustrated in Figs. 6036) can be displayed.

11G and 11H illustrate a translucent representation of a virtual object 11002 displayed in a user interface including the field of view 6036 of the camera(s). The translucent representation of the virtual object 11002 is displayed in a fixed position relative to the display 112. For example, from Figure 11G to Figure 11H, the device 100 is moved relative to the physical environment 5002 (e.g., as indicated by the altered position of the table 5004 within the field of view 6036 of the camera(s)). As such, the virtual object 11002 is held in a fixed position relative to the display 112.

In some embodiments, upon determining that a plane corresponding to the virtual object has been detected within the field of view 6036 of the camera(s), the virtual object is placed on the detected plane.

In FIG. 11I, a plane corresponding to the virtual object 11002 has been detected within the field of view 6036 of the camera(s), and the virtual object 11002 is placed on the detected plane. The device generated a tactile output as illustrated at 11010 (eg, to indicate that at least one plane (eg, floor surface 5038) was detected within the field of view 6036 of the camera(s)). When the virtual object 11002 is placed in a certain position with respect to the plane detected within the field of view 6036 of the camera(s), the virtual object 11002 is used for the physical environment 5002 captured by one or more cameras. It remains in a fixed position. 11I to 11J, as the device 100 is moved relative to the physical environment 5002 (e.g., as indicated by the altered position of the table 5004 within the displayed field of view 6036 of the camera(s)). , The virtual object 11002 is maintained in a fixed position with respect to the physical environment 5002.

In some embodiments, while the field of view 6036 of the camera(s) is displayed, the controls (e.g., back control 6016, toggle control 6018, and/or shared control 6020) are (e.g. , According to the determination that a certain period in which no input has been received has elapsed) the display is stopped. In FIGS. 11J to 11L, the controls 6016, 6018, 6020 are gradually faded out (e.g., as shown in FIG. 11K), so that the camera(s) The field of view 6036 increases the portion of the display 112 that is displayed.

11M-11S illustrate inputs for manipulating a virtual object 11002 when it is displayed on a user interface that includes the field of view 6036 of the camera(s).

In FIGS. 11M and 11N, an input (eg, a de-pinch gesture) by contacts 11012 and 11014 to change the simulated physical size of the virtual object 11002 is detected. In response to detection of the input, the controls 6016, 6018, 6020 are redisplayed. As the contact 11012 moves along the path indicated by the arrow 11016 and the contact 11014 moves along the path indicated by the arrow 11018, the size of the virtual object 11002 increases.

In FIGS. 11N-11P, an input (eg, a pinch gesture) by contacts 11012 and 1104 to change the simulated physical size of the virtual object 11002 is detected. As the contact 11012 moves along the path indicated by the arrow 11020 and the contact 11014 moves along the path indicated by the arrow 11022, the size of the virtual object 11002 (FIGS. 11N and 11O, and 11o and 11p). As illustrated in FIG. 11O, the size of the virtual object 11002 is initially on the detected plane within the physical environment 5002 (e.g., as shown in FIG. When scaled to the size of the virtual object 11002 when placed), the tactile output (as illustrated at 11024) is to provide feedback indicating that the virtual object 11002 has returned to its original size (e.g., ) Occurs. In FIG. 11Q, contacts 11012 and 11014 have been lifted from touch screen display 112.

In FIG. 11R, an input (eg, double tap input) to return the virtual object 11002 to its original size for the physical environment 5002 is detected. The input is detected at a location corresponding to the virtual object 11002, as indicated by the contact 11026. In response to the input, the virtual object 11002 is scaled from the reduced size illustrated in FIG. 11R to the original size of the virtual object 11002 as shown in FIG. 11S. As illustrated in FIG. Occurs to provide feedback indicating that is returned to its original size).

In FIG. 11T, an input by a contact 11030 is detected at a position corresponding to the toggle control 6018. In response to the input, the display of the user interface including the field of view 6036 of the camera(s) is replaced with a staging user interface 6010, as shown in FIG. 11U.

In Fig. 11U, an input by contact 11032 is detected at a position corresponding to the rewind control unit 6016. In response to the input, the display of the staging user interface 6010 is replaced with an Internet browser user interface 5060, as shown in FIG. 11V.

12A-12L illustrate exemplary user interfaces for displaying a calibration user interface object that is dynamically animated according to movement of one or more cameras of the device. User interfaces in these figures are in FIGS. 8A to 8E, 9A to 9D, 10A to 10D, 16A to 16G, 17A to 17D, 18A to 18I, 19A to 19H, and FIG. It is used to illustrate the processes described below, including the processes of FIGS. 20A-20F. For ease of description, some of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is, optionally: an individual finger or stylus contact, or a representative point corresponding to the finger or stylus contact (e.g., the center of the individual contact, or a point associated with the individual contact), or It is the center of two or more contacts detected on the touch-sensitive display system 112. However, similar operations may be performed in response to detecting contacts on the touch-sensitive surface 451, optionally displaying the user interfaces shown in the figure with the focus selector on the display 450, and It is performed on a device with a separate touch-sensitive surface 451.

According to some embodiments, if a request is received to display a virtual object on a user interface that includes a field of view of one or more cameras, but additional data is required for calibration of the device, the calibration user interface object is displayed.

12A illustrates an input requesting to display a virtual object 11002 within a user interface that includes a field of view 6036 of one or more cameras. An input by the contact 12002 is detected at a position corresponding to the toggle control unit 6018. In response to the input, the display of the staging user interface 6010 is replaced with the display of the user interface including the field of view 6036 of the camera(s), as shown in FIG. 12B. A translucent representation of the virtual object 11002 is displayed in the user interface including the field of view 6036 of the camera(s). While calibration is required (e.g., because the plane corresponding to the virtual object 11002 was not detected within the field of view 6036 of the camera(s)), the field of view 6036 of the camera(s) is (e.g., as described below). Likewise, the prompts and/or the behavior of the calibration object are blurred).

12B-12D illustrate animated images and text (eg, displayed in response to a determination that calibration is necessary) prompting the user to move the device. The animated image is a representation 12004 of the device 100, device 100 (e.g., to indicate that the device 100 should move relative to the plane to detect the plane corresponding to the virtual object 11002). It includes arrows (12006, 12008) indicating the need to move left and right of a plane, and a representation of a plane (12010). The text prompt 12012 provides information on movement of the device 100 required for calibration. In FIGS. 12B and 12C, and in FIGS. 12C and 12D, representation 12004 and arrow 12006 of device 100 are representations 12010 of a plane to provide an indication of movement of device 100 required for calibration. Is adjusted for. 12C to 12D, the device 100 is moved relative to the physical environment 5002 (eg, as indicated by the altered position of the table 5004 in the field of view 6036 of the camera(s)). As a result of detection of the movement of the device 100, a calibration user interface object 12014 (contour of the cube) is displayed, as shown in Fig. 12E-1.

12E-1 to 12I-1 are calibration user interface objects 12014 corresponding to movement of the device 100 relative to the physical environment 5002, as illustrated in FIGS. 12E-2 to 12I-2, respectively. Illustrate the behavior of The calibration user interface object 12014 is animated in response to the movement of the device 100 (e.g., lateral movement) (e.g., to provide feedback to the user about movement conducive to calibration) (e.g., cubical The contour rotates). In FIG. 12E-1, the calibration user interface object 12014 is shown to have a first rotation angle in the user interface including the field of view 6036 of the camera(s) of the device 100. In FIG. 12E-2, the device 100 is shown held by the user's hand 5006 in a first location relative to the physical environment 5002. 12E-2 to 12F-2, the device 100 has moved laterally (to the right) relative to the physical environment 5002. As a result of the movement, the field of view 6036 of the camera(s) as displayed by the device 100 is updated and the calibration user interface object 12014 is (Fig. 12E-1), as shown in Fig.12f-1. Rotated relative to his position at. 12F-2 to 12G-2, the device 100 continues its rightward movement relative to the physical environment 5002. As a result of the movement, the field of view 6036 of the camera(s) as displayed by the device 100 is updated again and the calibration user interface object 12014 is further rotated, as shown in FIG. 12G-1. . From FIGS. 12G-2 to 12H-2, the device 100 has moved upward with respect to the physical environment 5002. As a result of the movement, the field of view 6036 of the camera(s) as displayed by the device 100 is updated. 12G-1 to 12H-1, the calibration user interface object 12014 (e.g., to provide an indication to the user that the vertical movement of the device is not contributing to calibration) in FIG. It does not rotate in response to the upward movement of the device illustrated in FIG. 12H-2. 12H-2 to 12I-2, the device 100 has moved further to the right with respect to the physical environment 5002. As a result of the movement, the field of view 6036 of the camera(s) as displayed by the device 100 is updated again and the calibration user interface object 12014 is rotated, as shown in FIG. 12I-1.

In FIG. 12J, the movement of the device 100 (e.g., as illustrated in FIGS. 12E to 12I) satisfies the required calibration (e.g., and the plane corresponding to the virtual object 11002 is the camera(s). Was detected within the field of view 6036). The virtual object 11002 is placed on the detected plane, and the field of view 6036 of the camera(s) stops being blurred. The tactile output generators output a tactile output (as illustrated at 12016) to indicate that a plane (eg, floor surface 5038) has been detected within the field of view 6036 of the camera(s). The bottom surface 5038 is highlighted to provide an indication of the plane that was detected.

When the virtual object 11002 is placed at a position with respect to the plane detected within the field of view 6036 of the camera(s), the virtual object 11002 is It remains in a fixed position. As the device 100 is moved relative to the physical environment 5002 (as shown in FIGS. 12K-2 to 12L-2), the virtual object 11002 (shown in FIGS. 12K-1 to 12L-1) As previously described), it is held in a fixed position relative to the physical environment 5002.

13A-13M illustrate exemplary user interfaces for restricting rotation of a virtual object about an axis. User interfaces in these figures are in FIGS. 8A to 8E, 9A to 9D, 10A to 10D, 16A to 16G, 17A to 17D, 18A to 18I, 19A to 19H, and FIG. It is used to illustrate the processes described below, including the processes of FIGS. 20A-20F. For ease of description, some of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is, optionally: an individual finger or stylus contact, or a representative point corresponding to the finger or stylus contact (e.g., the center of the individual contact, or a point associated with the individual contact), or It is the center of two or more contacts detected on the touch-sensitive display system 112. However, similar operations may be performed in response to detecting contacts on the touch-sensitive surface 451, optionally displaying the user interfaces shown in the figure with the focus selector on the display 450, and It is performed on a device with a separate touch-sensitive surface 451.

In FIG. 13A, a virtual object 11002 is shown within a staging user interface 6010. The x-axis, y-axis, and z-axis are shown for virtual object 11002.

13B and 13C illustrate an input for rotating the virtual object 11002 about the y-axis shown in FIG. 13A. In FIG. 13B, an input by a contact 13002 is detected at a location corresponding to the virtual object 11002. The input moves by a distance d ₁ along the path indicated by the arrow 13004. As the input moves along the path, the virtual object 11002 rotates (eg, by 35 degrees) around the y-axis to the position shown in FIG. 13B. In the staging user interface 6010, a shadow 13006 corresponding to the virtual object 11002 is displayed. 13B to 13C, the shadow 13006 changes according to the changed position of the virtual object 11002.

After the contact 13002 is lifted off from the touch screen 112, the virtual object 11002 is moved (e.g., to provide the impression that the virtual object 11002 behaves similar to a physical object, the movement of the contact 1302 13C and 13D (depending on the "momentum" imparted by), it continues to rotate.

13E and 13F illustrate an input for rotating the virtual object 11002 about the x-axis shown in FIG. 13A. In FIG. 13E, an input by contact 13008 is detected at a location corresponding to virtual object 11002. The input moves by a distance d ₁ along the path indicated by the arrow 13010. As the input moves along the path, the virtual object 11002 rotates (eg, by 5 degrees) around the x-axis to the position shown in FIG. 13F. Even if the contact 13008 moves along the x-axis in FIGS. 13E and 13F by the same distance d _{1 that the} contact 13002 has moved from FIGS. 13B and 13C, The rotation angle of the virtual object 11002 is smaller than the rotation angle of the virtual object 11002 about the y-axis in FIGS. 13B and 13C.

13F and 13G illustrate additional inputs for rotating the virtual object 11002 about the x-axis shown in FIG. 13A. In FIG. 13F, contact 13008 continues its movement, moving a distance d ₂ (greater than distance d ₁ ) along the path indicated by arrow 13012. As the input moves along the path, the virtual object 11002 rotates (by 25 degrees) around the x-axis to the position shown in Fig. 13G. 13E to 13G, the movement of the contact 13008 by the distance (d ₁ + d ₂ ) causes the virtual object 11002 to rotate 30 degrees about the x-axis, while FIGS. 13B and 13B In 13c, movement of the contact 13004 by the distance d ₁ causes the virtual object 11002 to rotate 35 degrees around the y-axis.

After the contact 13008 is lifted off from the touch screen 112, the virtual object 11002 is, as shown in Figs. 13G and 13H (e.g., when the movement of the contact In order to indicate that the amount of rotation of the virtual object 11002 is caused), it rotates in a direction opposite to the direction of rotation caused by the movement of the contact 13008.

13G-13I, the shadow 13006 is not shown (eg, because the virtual object 11002 does not cast a shadow when the object is viewed from below).

In Fig. 13I, an input (e.g., double tap input) for returning the virtual object 11002 to the time at which it was originally displayed (e.g., as shown in Fig. 13A) is detected. The input occurs at a location corresponding to virtual object 11002, as indicated by contact 13014. In response to the input, the virtual object 11002 is centered on the y-axis (to reverse the rotation occurring from Fig. 13E to 13H) and around the x-axis (to reverse the rotation occurring from Fig. 13B to 13D). Rotated. In Fig. 13J, the input by the contact 13016 caused the virtual object 11002 to return to the original displayed time point.

In some embodiments, an input is received to resize virtual object 11002 while staging user interface 6010 is displayed. For example, the input for resizing the virtual object 11002 may be a de-pinch gesture or virtual object (e.g., as described in connection with FIGS. 6N and 6O) to increase the size of the virtual object 11002 It is a pinch gesture to reduce the size of (11002).

In FIG. 13J, input is received to replace the display of the staging user interface 6010 with the display of the user interface including the field of view 6036 of the camera(s). An input by the contact 13016 is detected at a position corresponding to the toggle control unit 6018. In response to the input, the display of the staging user interface 6010 is replaced with a user interface that includes the field of view 6036 of the camera(s), as shown in FIG. 13K.

In FIG. 13K, a virtual object 11002 is displayed in a user interface that includes a field of view 6036 of the camera(s). A tactile output (as illustrated at 13018) is generated to indicate that a plane corresponding to the virtual object 11002 has been detected within the field of view 6036 of the camera(s). The rotation angle of the virtual object 11002 in the user interface including the field of view 6036 of the camera(s) corresponds to the rotation angle of the virtual object 11002 in the staging user interface 6010.

When the user interface comprising the field of view 6036 of the camera(s) is displayed, the input comprising lateral movement comprises the field of view 6036 of the camera(s), as illustrated in FIGS. 13L and 13M. Causing a lateral movement of the virtual object 11002 within the user interface. In FIG. 13L, a contact 13020 is detected at a position corresponding to the virtual object 11002, and the contact moves along a path indicated by an arrow 13022. As the contact moves, the virtual object 11002 takes a path corresponding to the movement of the contact 1302 from a first location (as shown in FIG. 13L) to a second location (as shown in FIG. 13M). So it moves.

In some embodiments, the input provided when the user interface including the field of view 6036 of the camera(s) is displayed is a first plane (e.g., a floor plane), as described in connection with FIGS. (5038)) to a second plane (e.g., table surface plane 5046).

14A-14Z illustrate exemplary user interfaces for increasing the second threshold size of movement required for the second object manipulation behavior, upon determining that the first threshold size of movement is satisfied for the first object manipulation behavior. . User interfaces in these drawings are in FIGS. 8A to 8E, 9A to 9D, 10A to 10D, 14A to 14A, 16A to 16G, 17A to 17D, 18A to 18I, and 19A. 19H, and 20A-20F are used to illustrate the processes described below. For ease of description, some of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is, optionally: an individual finger or stylus contact, or a representative point corresponding to the finger or stylus contact (e.g., the center of the individual contact, or a point associated with the individual contact), or It is the center of two or more contacts detected on the touch-sensitive display system 112. However, similar operations may be performed in response to detecting contacts on the touch-sensitive surface 451, optionally displaying the user interfaces shown in the figure with the focus selector on the display 450, and It is performed on a device with a separate touch-sensitive surface 451.

In FIG. 14A, a virtual object 11002 is displayed in a user interface that includes a field of view 6036 of the camera(s). As further described with respect to FIGS. 14B to 14Z, the translational movement measuring device 14002, the scaling movement measuring device 14004, and the rotational movement measuring device 14006 may include object manipulation behaviors (e.g., translational movement, scaling operation , And/or rotational motion). The translation measurer 14002 indicates the magnitude of the lateral (eg, left or right) movement of a set of contacts on the touch screen display 112. The scaled movement meter 14004 indicates the amount of increasing or decreasing distance between each of the contacts in a set of contacts on the touch screen display 112 (eg, the size of a pinch or de-pinch gesture). The rotational movement meter 14006 indicates the magnitude of rotational movement of a set of contacts on the touch screen display 112.

14B-14E illustrate an input for rotating a virtual object 11002 in a user interface that includes a field of view 6036 of one or more cameras. The input for rotating the virtual object 11002 is that the first contact 14008 rotates clockwise along the path indicated by the arrow 14010, and the second contact 14012 moves along the path indicated by the arrow 14014. It includes a gesture of rotating clockwise. In FIG. 14B, contacts 14008 and 14012 with the touch screen 112 are detected. In FIG. 14C, contact 14008 has moved along the path indicated by arrow 14010, and contact 14012 has moved along the path indicated by arrow 14012. In FIG. 14C, since the magnitude of the rotational movement of the contact 14008 and the contact 14012 has not reached the threshold RT, the virtual object 11002 has not yet rotated in response to the input. In Fig. 14D, the magnitude of the rotational movement of the contact 14008 and the contact 14012 has increased beyond the threshold RT, and the virtual object 11002 is in response to an input (virtual object 11002 shown in Fig. 14B). For the position of). When the size of the rotational movement increases beyond the threshold value RT, the required size of the movement for scaling the virtual object 11002 is increased (e.g., as shown in the scaling movement measurer 14004, The adjustment threshold ST has increased from ST to ST'), and the required size of the movement for translating the virtual object 11002 is increased (e.g., as shown in the translational movement meter 14002, the translational threshold TT ) Increased from TT to TT'). In FIG. 14E, contact 14008 and contact 14012 continued to move along rotation paths indicated by arrows 14010 and 14014, respectively, and virtual object 11002 continued to rotate in response to input. In FIG. 14F, contacts 14008 and 14012 have been lifted off from touch screen 112.

14G-14I illustrate an input to scale (eg, increase its size) a virtual object 11002 within a user interface that includes a field of view 6036 of one or more cameras. The input for increasing the size of the virtual object 11002 is to move the first contact 1416 along the path indicated by the arrow 1401 (for example, to increase the distance between the contact 1416 and the contact 1402). And a gesture in which the second contact 14020 moves along the path indicated by the arrow 14022. In FIG. 14G, contacts 1416 and 14020 with the touch screen 112 are detected. In FIG. 14H, contact 1416 has moved along the path indicated by arrow 1401, and contact 14020 has moved along the path indicated by arrow 1402. In FIG. 14H, since the amount of movement of the contact 1416 away from the contact 1402 has not reached the threshold ST, the size of the virtual object 11002 has not yet been adjusted in response to the input. In FIG. 14I, the magnitude of the scaling movement of the contact 1416 and the contact 1402 increases beyond the threshold ST, and the size of the virtual object 11002 is in response to an input (the virtual object shown in FIG. Increased (compared to the size of 11002). When the magnitude of the scaling movement increases beyond the threshold ST, the required magnitude of the movement for rotating the virtual object 11002 increases (e.g., as shown in the rotation movement meter 14006, the rotation threshold (RT) increased from RT to RT'), and the required amount of movement for translating the virtual object 11002 is increased (e.g., as shown in the translational movement meter 14002, the translation threshold TT is Increased from TT to TT'). In FIG. 14J, contacts 14016 and 14020 have been lifted off from touch screen 112.

14K-14M illustrate an input (eg, to move the virtual object 11002 to the left) to translate the virtual object 11002 in a user interface that includes a field of view 6036 of one or more cameras. The input for moving the virtual object 11002 is that the first contact 14024 moves along the path indicated by the arrow 14026 (e.g., so that both contact 14024 and contact 14028 move leftward) and The second contact 14028 includes a gesture of moving along the path indicated by the arrow 1430. In FIG. 14K, contacts 14024 and 14028 with touch screen 112 are detected. In FIG. 14L, contact 14024 has moved along the path indicated by arrow 14026, and contact 14028 has moved along the path indicated by arrow 1430. In FIG. 14L, since the magnitude of the leftward movement of the contacts 14024 and 14028 has not reached the threshold TT, the virtual object 11002 has not yet moved in response to the input. In FIG. 14M, the magnitude of the contact 14024 and the leftward movement of the contact 14028 has increased beyond the threshold TT, and the virtual object 11002 has been moved in the movement direction of the contacts 14024 and 14028. When the size of the translational movement increases beyond the threshold value TT, the required size of the movement for scaling the virtual object 11002 is increased (e.g., as shown in the scaling movement measurer 14004, The adjustment threshold ST has increased from ST to ST'), and the required amount of movement for rotating the virtual object 11002 is increased (e.g., as shown in the rotation movement meter 14006, the rotation threshold RT ) Increased from RT to RT'). In FIG. 14N, contacts 14024 and 14028 have been lifted off from touch screen 112.

14O to 14Z illustrate translating the virtual object 11002 (e.g., moving the virtual object 11002 to the right) and scaling the virtual object 11002 (e.g., increasing the size of the virtual object 11002). To rotate), and gestures for rotating the virtual object 11002. In FIG. 14O, contacts 14032 and 14036 with the touch screen 112 are detected. In FIGS. 14O and 14P, contact 14032 moves along the path indicated by arrow 14034, and contact 14036 moves along the path indicated by arrow 14038. The magnitude of the rightward movement of the contacts 14032 and 14036 has increased beyond the threshold TT, and the virtual object 11002 has been moved in the movement direction of the contacts 14032 and 14036. As a result of satisfaction of the threshold TT by the movement of the contacts 14032 and 14036, the required amount of movement for scaling the virtual object 11002 is increased to ST', and the virtual object 11002 is rotated. The required amount of movement to make is increased to RT'. After the threshold TT is satisfied (as indicated by the high water mark 14043 shown in the translational motion meter 14002 in FIG. 14Q), any lateral movement of the contacts 14032, 14036 is It will cause the lateral movement of the virtual object 11002.

14Q and 14R, contact 14032 moves along the path indicated by arrow 14040, and contact 14036 moves along the path indicated by arrow 14042. In FIG. 14R, the magnitude of the movement of the contact 14032 away from the contact 14036 exceeded the original scaling threshold ST, but did not reach the increased scaling threshold ST'. When the increased scaling movement threshold ST' is applied, the scaling is performed until the amount of movement of the contact 14032 away from the contact 14036 increases by exceeding the increased scaling movement threshold ST'. Since it did not occur, the size of the virtual object 11002 did not change from FIG. 14Q to FIG. 14R. In Figures 14R and 14S, the distance between contact 14032 and contact 14046 is as follows: contact 14032 moves along the path indicated by arrow 14044 and contact 14036 follows the path indicated by arrow 14046. It keeps increasing as you move. In FIG. 14S, the amount of movement of the contact 14032 away from the contact 14036 exceeds the increased scaling threshold ST', and the size of the virtual object 11002 increases. After the threshold ST' is satisfied (as indicated by the high watermark 14047 shown in the scaling movement measurer 14004 of FIG. 14T), any scaling movement of the contacts 14032 and 14036 is a virtual object. (11002) will cause scaling.

In FIGS. 14T and 14U, contact 14032 moves along the path indicated by arrow 14048, and contact 14036 moves along the path indicated by arrow 14050. Since the threshold TT was satisfied (as indicated by the high watermark 14043 shown in the translation measurement device 14002), the virtual object 11002 is in the direction of the lateral movement of the contacts 14032, 14036. Move freely

14V and 14W, contact 14032 moves along the path indicated by arrow 14052, and contact 14036 moves along the path indicated by arrow 14054. The movement of the contacts 14032 and 14036 is a translational movement (a movement to the left of the contacts 14032 and 14036) and a scaling movement (a movement that reduces the distance between the contact 14032 and the contact 14036 (e.g., pinch Gesture)). Since the translation threshold TT was satisfied (as indicated by the high watermark 14043 shown in the translation measurement device 14002), the virtual object 11002 is the direction of the lateral movement of the contacts 14032, 14036. And the increased scaling threshold (ST') is satisfied (as indicated by the high watermark 14047 shown in the scaling movement meter 14004), the virtual object 11002 It is freely scaled in response to the movement of the contact 14032 toward ). 14V to 14W, the size of the virtual object 11002 has been reduced, and the virtual object 11002 moves the contact 14032 along the path indicated by the arrow 14052 and the contact along the path indicated by the arrow 14054. In response to the movement of (14036), it moved to the left.

In FIGS. 14X to 14Z, the contact 14032 rotates counterclockwise along the path indicated by the arrow 14056, and the contact 14036 rotates counterclockwise along the path indicated by the arrow 14058. . In Fig. 14Y, the magnitude of the rotational movement of the contact 14032 and the contact 14036 exceeded the original scaling threshold RT, but did not reach the increased scaling threshold RT'. When the increased scaling movement threshold RT' is applied, the amount of rotational movement of the contacts 14032 and 14036 increases beyond the increased rotational movement threshold RT'. Since rotation did not occur, the virtual object 11002 did not rotate from FIG. 14X to FIG. 14Y and 14Z, contacts 14032, 14046 are counterclockwise as contact 14032 moves along the path indicated by arrow 14060 and contact 14036 moves along the path indicated by arrow 14062. Continue to rotate and move. In Fig. 14Z, the magnitude of the rotational movement of the contact 14032 and the contact 14036 exceeded the increased scaling threshold RT', and the virtual object 11102 rotated in response to an input.

14aa to 14ad are flowcharts illustrating operations for increasing the second threshold size of movement required for the second object manipulation behavior, according to a determination that the first threshold size of movement is satisfied for the first object manipulation behavior. The operations described in connection with FIGS. 14AA-14A may include a display generating component (e.g., display, projector, head-up display, etc.) and a touch-sensitive surface (e.g., a touch-sensitive surface, or both a display-generating component and a touch-sensitive surface). It is performed on an electronic device (eg, device 300 of FIG. 3, or portable multifunction device 100 of FIG. 1A) having a touch screen display that serves as all. Some of the operations described in connection with FIGS. 14A to 14A are selectively combined and/or the order of some of the operations is selectively changed.

In operation 14066, a first portion of the user input including movement of one or more contacts is detected. In operation 14068, the movement of one or more contacts (e.g., at a location corresponding to the virtual object 11002) determines an object rotation threshold (e.g., a rotation threshold (RT) indicated by the rotational movement meter 14006). Whether it increases in excess is determined. Upon determination that the movement of one or more contacts increases above the object rotation threshold (eg, as described in connection with FIGS. 14B-14D ), the flow proceeds to operation 14070. Upon determination that the movement of one or more contacts does not increase beyond the object rotation threshold, flow proceeds to operation 14074.

In operation 14070, the object (eg, virtual object 11002) is rotated based on a first portion of user input (eg, as described in connection with FIGS. 14B-14D ). In operation 14072, the object translation threshold is increased (e.g., from TT to TT', as described in connection with Figure 14D), and the object scaling threshold is increased (e.g., as described in connection with Figure 14D, From ST to ST'). Flow proceeds from operation 14072 to operation 14086 in FIG. 14A, as indicated by A.

In operation 14074, the movement of one or more contacts (e.g., at a location corresponding to the virtual object 11002) determines an object translation threshold (e.g., a translation threshold TT indicated by the translational movement meter 14002) Whether it increases in excess is determined. Upon determination that the movement of one or more contacts increases above the object translation threshold (eg, as described in connection with FIGS. 14K-14M ), the flow proceeds to operation 14076. Upon determining that the movement of one or more contacts does not increase beyond the object translation threshold, flow proceeds to operation 14080.

In operation 14076, the object (eg, virtual object 11002) is translated based on a first portion of user input (eg, as described in connection with FIGS. 14K-14M). In operation 14078, the object rotation threshold is increased (e.g., from RT to RT', as described in connection with FIG. 14M) and the object scaling threshold is increased (e.g., as described in connection with FIG. 14M, From ST to ST'). Flow proceeds from operation 14078 to operation 14100 of FIG. 14A, as indicated at B.

In operation 14080, the movement of one or more contacts (e.g., at a location corresponding to the virtual object 11002) is an object scaling threshold (e.g., a scaling threshold ST indicated by the scaling movement measurer 1404 It is determined whether it increases beyond )). Upon determination that the movement of one or more contacts increases above the object scaling threshold (eg, as described in connection with FIGS. 14G-14I ), flow proceeds to operation 14082. Upon determination that the movement of one or more contacts does not increase beyond the object scaling threshold, flow proceeds to operation 14085.

In operation 14082, the object (eg, virtual object 11002) is scaled based on the first portion of the user input (eg, as described in connection with FIGS. 14G-14I ). In operation 14084, the object rotation threshold is increased (e.g., from RT to RT', as described in connection with FIG. 14I) and the object translation threshold is increased (e.g., as described in connection with FIG. From TT'). The flow proceeds from operation 14084 to operation 14114 in FIG. 14A, as indicated at C.

In operation 14085, an additional portion of user input is detected that includes movement of one or more contacts. Flow proceeds from operation 14086 to operation 14066.

In FIG. 14A, in operation 14086, an additional portion of user input is detected that includes movement of one or more contacts. Flow proceeds from operation 14086 to operation 14088.

In operation 14088, it is determined whether the movement of the one or more contacts is a rotational movement. Upon determination that the movement of one or more contacts is a rotational movement, flow proceeds to operation 14090. Upon determination that the movement of one or more contacts is not a rotational movement, flow proceeds to operation 14092.

In operation 14090, the object (eg, virtual object 11002) is rotated based on the additional portion of the user input (eg, as described in connection with FIGS. 14D and 14E). Since the rotation threshold was previously met, the object rotates freely according to the additional rotation input.

In operation 14092, it is determined whether the movement of the one or more contacts increases above an increased object translation threshold (eg, a translation threshold TT′ indicated by the translational movement meter 14002 in FIG. 14D). Upon determination that the movement of one or more contacts increases beyond the increased object translation threshold, flow proceeds to operation 14094. Upon determination that the movement of one or more contacts does not increase beyond the increased object translation threshold, flow proceeds to operation 14096.

In operation 14094, the object (eg, virtual object 11002) is translated based on the additional portion of the user input.

In operation 14096, it is determined whether the movement of the one or more contacts increases beyond an increased object scaling threshold (e.g., a scaling threshold ST' indicated by the scaling movement measurer 14004 of FIG. 14D). do. Upon determination that the movement of one or more contacts increases beyond the increased object scaling threshold, flow proceeds to operation 14098. Upon determination that the movement of one or more contacts does not increase beyond the increased object scaling threshold, flow returns to operation 14086.

In operation 14098, the object (eg, virtual object 11002) is scaled based on the additional portion of the user input.

In Figure 14ac, in operation 14100, an additional portion of user input is detected that includes movement of one or more contacts. Flow proceeds from operation 14100 to operation 14102.

In operation 14102, it is determined whether the movement of the one or more contacts is a translational movement. Upon determination that the movement of one or more contacts is a translational movement, flow proceeds to operation 140104. Upon determination that the movement of one or more contacts is not a translational movement, flow proceeds to operation 14106.

In operation 14104, the object (eg, virtual object 11002) is translated based on the additional portion of the user input. Since the translation threshold has previously been met, the object is freely translated according to further translation inputs.

In operation 14106, it is determined whether the movement of the one or more contacts increases above an increased object rotation threshold (eg, rotation threshold RT′ indicated by rotation movement meter 14006 in FIG. 14M). Upon determination that the movement of the one or more contacts increases beyond the increased object rotation threshold, flow proceeds to operation 14108. Upon determining that the movement of one or more contacts does not increase beyond the increased object rotation threshold, flow proceeds to operation 14110.

In operation 14108, the object (eg, virtual object 11002) is rotated based on the additional portion of the user input.

In operation 14110, it is determined whether the movement of the one or more contacts increases beyond the increased object scaling threshold (e.g., the scaling threshold ST' indicated by the scaling movement measurer 14004 of FIG. 14M). do. Upon determination that the movement of one or more contacts increases beyond the increased object scaling threshold, flow proceeds to operation 14112. Upon determination that the movement of one or more contacts does not increase beyond the increased object scaling threshold, flow returns to operation 14100.

In operation 14112, the object (eg, virtual object 11002) is scaled based on the additional portion of the user input.

In FIG. 14A, in operation 14114, an additional portion of user input is detected that includes movement of one or more contacts. Flow proceeds from operation 14114 to operation 14116.

In operation 14116, it is determined whether the movement of the one or more contacts is a scaling movement. Upon determination that the movement of one or more contacts is a scaling movement, flow proceeds to operation 140118. Upon determination that the movement of the one or more contacts is not a scaling movement, flow proceeds to operation 14120.

In operation 14118, the object (eg, virtual object 11002) is scaled based on the additional portion of the user input. Since the scaling threshold has previously been met, the object is freely scaled according to the additional scaling input.

In operation 14120, it is determined whether the movement of the one or more contacts increases above an increased object rotation threshold (eg, rotation threshold RT′ indicated by rotation movement meter 14006 in FIG. 14I). Upon determination that the movement of the one or more contacts increases beyond the increased object rotation threshold, flow proceeds to operation 14122. Upon determining that the movement of one or more contacts does not increase beyond the increased object rotation threshold, flow proceeds to operation 14124.

In operation 14122, the object (eg, virtual object 11002) is rotated based on the additional portion of the user input.

In operation 14124, it is determined whether the movement of the one or more contacts increases above an increased object translation threshold (eg, a translation threshold TT' indicated by the translational movement meter 14002 of FIG. 14I). Upon determination that the movement of one or more contacts increases beyond the increased object translation threshold, flow proceeds to operation 14126. Upon determining that the movement of one or more contacts does not increase beyond the increased object translation threshold, flow proceeds to operation 14114.

15A-15Ai illustrate example user interfaces for generating an audio alert upon determining that movement of the device causes the virtual object to move outside the displayed field of view of one or more device cameras. User interfaces in these figures are in FIGS. 8A to 8E, 9A to 9D, 10A to 10D, 16A to 16G, 17A to 17D, 18A to 18I, 19A to 19H, and FIG. It is used to illustrate the processes described below, including the processes of FIGS. 20A-20F. For ease of description, some of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is, optionally: an individual finger or stylus contact, or a representative point corresponding to the finger or stylus contact (e.g., the center of the individual contact, or a point associated with the individual contact), or It is the center of two or more contacts detected on the touch-sensitive display system 112. However, similar operations may be performed in response to detecting contacts on the touch-sensitive surface 451, optionally displaying the user interfaces shown in the figure with the focus selector on the display 450, and It is performed on a device with a separate touch-sensitive surface 451.

15A-15Ai illustrate user interfaces and device operations that occur when the accessibility feature is active. In some embodiments, the accessibility feature is an alternative or reduced number of inputs (eg, to increase the ease of accessing device features for users with limited ability to provide the aforementioned input gestures). Includes a mode that enables inputs to be used to access device features. For example, the accessibility mode allows a first input gesture (e.g., swipe input) to be used to advance or reverse through available device actions and a selection input (e.g., double tap input) to perform the currently displayed action. This is the switch control mode used. As the user interacts with the device, (e.g., providing feedback to the user indicating that an action has been performed, indicating the field of view of one or more cameras of the device or the current display state of the virtual object 11002 with respect to the staging user interface) Etc.) audio alerts are generated.

In FIG. 15A, a messaging user interface 5008 includes a two-dimensional representation of a three-dimensional virtual object 11002. A selection cursor 15001 is shown surrounding a three-dimensional virtual object 11002 (eg, to indicate that the currently selected operation is an operation to be performed on the virtual object 11002). An input by contact 15002 (eg, double tap input) is detected to perform the currently displayed operation (eg, displaying a three-dimensional representation of virtual object 11002 in staging user interface 6010). In response to the input, the display of the messaging user interface 5060 is replaced with the display of the staging user interface 6010, as shown in FIG. 15B.

In FIG. 15B, a virtual object 11002 is displayed within the staging user interface 6010. To indicate the state of the device, an audio alert is generated (eg, by the device speaker 111 ), as indicated by 15008. For example, audio alert 15008 includes a notification, as indicated by 15010, "chair is now shown in the staging view".

In FIG. 15B, a selection cursor 15001 is shown surrounding the sharing control 6020 (eg, to indicate that the currently selected operation is a sharing operation). An input by contact 15004 (e.g., a swipe to the right along the path indicated by arrow 15006) is detected. In response to the input, the selected action advances to the next action.

In FIG. 15C, a tilt up control unit 15012 is displayed (eg, to indicate that the currently selected operation is an operation for tilting the displayed virtual object 11002 upward). An audio alert is generated, as indicated by 15014, to indicate the state of the device. For example, the audio alert includes a notification, as indicated by 15016, "selected: tilt up button". An input by contact 15018 (e.g., a swipe right along the path indicated by arrow 15020) is detected. In response to the input, the selected action advances to the next action.

In FIG. 15D, a tilt down control unit 15022 is displayed (eg, to indicate that the currently selected operation is an operation for tilting the displayed virtual object 11002 downward). An audio alert is generated, as indicated by 15024, to indicate the state of the device. For example, the audio alert includes a notification, as indicated by 15026, "selected: tilt down button". An input (eg, double tap input) by contact 15028 is detected. In response to the input, the selected action is performed (eg, the virtual object 11002 is tilted downward within the staging view).

In Fig. 15E, the virtual object 11002 is tilted downward in the staging view. An audio alert is generated, as indicated by 15030, to indicate the state of the device. For example, an audio alarm is a notification as indicated by 15032, "Chair tilted five degrees down. Chair is now tilted 10 degrees toward the screen. Chair is now tilted 10 degrees toward the screen. It includes".

In FIG. 15F, input by contact 15034 (eg, a swipe to the right along the path indicated by arrow 15036) is detected. In response to the input, the selected action advances to the next action.

In Fig. 15G, the clockwise rotation control unit 15038 is displayed (eg, to indicate that the currently selected operation is an operation for rotating the displayed virtual object 11002 clockwise). The audio alert 15040 includes a notification, as indicated by 15042, "selected: rotate clockwise button". An input by contact 15044 (e.g., a swipe to the right along the path indicated by arrow 15046) is detected. In response to the input, the selected action advances to the next action.

In Fig. 15H, the counterclockwise rotation control unit 15048 is displayed (eg, to indicate that the currently selected operation is an operation for rotating the displayed virtual object 11002 counterclockwise). The audio alert 15050 includes a notification, as indicated by 15052, "selected: rotate counterclockwise button". An input (eg, a double tap input) by contact 15054 is detected. In response to the input, the selected operation is performed (eg, the virtual object 11002 is rotated counterclockwise within the staging view, as shown in Fig. 15I).

In Fig. 15I, the audio alarm 15056 is a notification, as indicated by 15058, "Chair rotated by five degrees counterclockwise. Chair is now rotated five degrees away the screen. The chair is now screened. It is rotated 5 degrees away from)".

In FIG. 15J, input by contact 15060 (eg, a swipe to the right along the path indicated by arrow 15062) is detected. In response to the input, the selected action advances to the next action.

In FIG. 15K, the zoom control unit 15064 is displayed (eg, to indicate that the currently selected operation is an operation for zooming the displayed virtual object 11002 ). The audio alert 15066 includes a notification, "scale: adjustable", as indicated by 15068. The keyword “adjustable” along with the control unit name in the notification indicates that a swipe input (eg, vertical swipe input) is available to activate the control unit. For example, an upward swipe input is provided by contact 5070 as it moves upwardly along the path indicated by arrow 5072. In response to the input, a zoom operation is performed (e.g., as shown in Figs. 15K and 15L, the size of the virtual object 11002 is increased).

In Fig. 15L, audio alert 15074 includes a notification, as indicated by 15076, "Chair is now adjusted to 150 percent of original size". An input for reducing the size of the virtual object 11002 (eg, a downward swipe input) is provided by contact 5078 moving downward along the path indicated by arrow 5078. In response to the input, a zoom operation is performed (e.g., as shown in Figs. 15L and 15M, the size of the virtual object 11002 is reduced).

In FIG. 15M, the audio alert 15082 includes a notification, as indicated by 15084, "Chair is now adjusted to 100 percent of original size". Because the size of the virtual object 11002 is scaled to its original displayed size within the staging view 6010, the tactile output (as illustrated by 15086) is To provide feedback indicating that it has been returned).

In FIG. 15N, input by contact 15088 (eg, a swipe to the right along the path indicated by arrow 15090) is detected. In response to the input, the selected action advances to the next action.

In FIG. 15O, a selection cursor 15001 is shown surrounding the back control 6016 (eg, to indicate that the currently selected operation is an operation to return to the previous user interface). The audio alert 15092 includes a notification, "selected: return button", as indicated by 15094. Input by contact 15096 (e.g., a swipe to the right along the path indicated by arrow 15098) is detected. In response to the input, the selected action advances to the next action.

In Figure 15P, the selection cursor 15001 is (e.g., that the currently selected action is an action to toggle between the display of the staging user interface 6010 and the display of the user interface including the field of view 6036 of the camera(s). To indicate that) is shown surrounding the toggle control 6018. The audio alert 15098 includes a notification, as indicated by 50100, "selected: world view/staging view toggle". An input by contact 15102 (eg, a double tap input) is detected. In response to the input, the display of the staging user interface 6010 is replaced with the display of the user interface including the field of view 6036 of the camera(s) (as shown in FIG.

15Q-15T show when the field of view 6036 of the camera(s) is displayed (e.g., because the plane corresponding to the virtual object 11002 has not yet been detected within the field of view 6036 of the camera(s)). The calibration sequence that occurs is illustrated. During the calibration sequence, a translucent representation of the virtual object 11002 is displayed, the field of view 6036 of the camera(s) is blurred, (including a representation 12004 of the device 100 and a representation 12010 of a plane). ) A prompt containing an animated image is displayed to prompt the user to move the device. In FIG. 15Q, the audio alert 15102 includes a notification, "move the device to detect a plane", as indicated by 50104. 15Q to 15R, the device 100 is moved relative to the physical environment 5002 (eg, as indicated by the altered position of the table 5004 within the field of view 6036 of the camera(s)). As a result of detection of the movement of the device 100, a calibration user interface object 12014 is displayed, as shown in Fig. 15S.

In FIG. 15S, the audio alert 15106 includes a notification, "move the device to detect a plane", as indicated by 50108. In Figures 15S and 15T, the calibration user interface object 12014 is the physical environment 5002 of the device 100 (e.g., as indicated by the altered position of the table 5004 within the field of view 6036 of the camera(s)). ) Rotates as it moves about. In FIG. 15T, enough motion has occurred with respect to the plane corresponding to the virtual object 11002 to be detected within the field of view 6036 of the camera(s), and the audio alert 15110 is a notification, as indicated by 50112, "plane detected. (Plane is detected)". 15U and 15V, the translucency of the virtual object 11002 is reduced, and the virtual object 11002 is disposed on the detected plane.

In FIG. 15V, the audio alarm 15114 is a notification, as indicated by 50116, "chair is now projected in the world, 100 percent visible, occupying 10 percent of the screen (the chair is now projected in the world, 100 percent visible, Occupies 10 percent of the screen)". The tactile output generators output a tactile output (as illustrated at 15118) to indicate that the virtual object 11002 has been placed on a plane. The virtual object 11002 is displayed in a fixed position relative to the physical environment 5002.

In FIGS. 15V and 15W, the device 100 has a table 5004 in the field of view 6036 of the camera(s) such that the virtual object 11002 is no longer visible within the field of view 6036 of the camera(s). ) Is moved relative to the physical environment 5002). As a result of the movement of the virtual object 11002 out of the field of view 6036 of the camera(s), the audio alert 15122 gives a notification, as indicated by 50124, "chair is not on the screen. )".

In FIGS. 15W and 15X, the device 100 has moved relative to the physical environment 5002 such that the virtual object 11002 is again visible within the field of view 6036 of the camera(s) of FIG. 15X. As a result of the movement of the virtual object 11002 into the field of view 6036 of the camera(s), a notification as indicated by 50120, "chair is now projected in the world, 100 percent visible, occupying 10 percent of the screen" A containing audio alert 15118 is generated.

In FIGS. 15X and 15Y, the device 100 is “closer” to the virtual object 11002 (e.g., as the device 100 is projected into the field of view 6036 of the camera(s), and the virtual object ( 11002 has been moved relative to the physical environment 5002) so that it is partially visible within the field of view 6036 of the camera(s) of FIG. 15Y. Partly as a result of the movement of the virtual object 11002 out of the field of view 6036 of the camera(s), the audio alert 15126 is a notification, as indicated by 50128, "chair is 90 percent visible, occupying 20 percent of the screen. (The chair is 90 percent visible and takes up 20 percent of the screen)".

In some embodiments, input provided at a location corresponding to the virtual object 11002 causes an audio message to be provided that includes verbal information about the virtual object 11002. In contrast, when the input is provided at a location remote from the virtual object 11002 and the controls, an audio message containing verbal information about the virtual object 11002 is not provided. In FIG. 15Z, an audio output 15130 (eg, “click” or “buzz”) is used to indicate that the contact 15132 is detected at a location that does not correspond to the location of the control or virtual object 11002 in the user interface. Occurs. In FIG. 15AA, an input by contact 15134 is detected at a location corresponding to the location of the virtual object 11002. In response to the input, corresponding to the virtual object 11002 (e.g., indicating the state of the virtual object 11002), including a notification as indicated by 50138, "chair is 90 percent visible, occupying 20 percent of the screen" An audio alert 15136 is generated.

15A through 15AI illustrate inputs for selection and execution of operations in a switch control mode while a user interface including a field of view 6036 of the camera(s) is displayed.

In FIG. 15Ab, input by contact 15140 (eg, a swipe to the right along the path indicated by arrow 15142) is detected. In response to the input, an action is selected, as shown in Figure 15ac.

In Fig. 15ac, a rightward lateral movement control unit 15144 is displayed (eg, to indicate that the currently selected operation is an operation for moving the virtual object 11002 to the right). The audio alert 15146 includes a notification, "selected: move right button", as indicated by 15148. An input (eg, a double tap input) by the contact 15150 is detected. In response to the input, the selected action is performed (eg, the virtual object 11002 is moved to the right within the field of view 6036 of the camera(s), as shown in FIG. 15AD).

In Fig. 15A, the movement of the virtual object 11002 results in a notification, "chair is 100 percent visible, occupying 30 percent of the screen", as indicated by 15154 (the chair is 100 percent visible, occupying 30 percent of the screen). Reported by the containing audio alert 15152.

In FIG. 15AE, input by contact 15156 (eg, swiping to the right along the path indicated by arrow 15158) is detected. In response to the input, the selected action advances to the next action.

In FIG. 15Af, the lateral movement control unit 15160 facing left is displayed (eg, to indicate that the currently selected operation is an operation for moving the virtual object 11002 to the left). The audio alert 15162 includes a notification, "selected: move left", as indicated by 15164. An input by contact 15166 (e.g., a swipe to the right along the path indicated by arrow 15168) is detected. In response to the input, the selected action advances to the next action.

In FIG. 15AG, the clockwise rotation control unit 15170 is displayed (eg, to indicate that the currently selected operation is an operation for rotating the virtual object 11002 clockwise). The audio alert 15172 includes a notification, as indicated by 15174, "selected: rotate clockwise". An input by contact 15176 (e.g., a swipe to the right along the path indicated by arrow 15178) is detected. In response to the input, the selected action advances to the next action.

In Fig. 15Ah, the counterclockwise rotation control unit 15180 is displayed (eg, to indicate that the currently selected operation is an operation for rotating the virtual object 11002 clockwise). The audio alert 15182 includes a notification, as indicated by 15184, "selected: rotate counterclockwise". An input (eg, a double tap input) by the contact 15185 is detected. In response to the input, the selected operation is performed (eg, the virtual object 11002 is rotated counterclockwise, as shown in Fig. 15Ai).

In Fig. 15ai, the audio alarm 15190 is a notification, as indicated by 15164, "Chair rotated by five degrees counterclockwise. Chair is now rotated by zero degrees relative to the screen. Chair is now rotated by five degrees relative to the screen. Now it is rotated 0 degrees relative to the screen)".

In some embodiments, a reflection is created on at least one surface (eg, lower surface) of an object (eg, virtual object 11002). The reflective image is created using image data captured by one or more cameras of device 100. For example, the reflective image is captured image data (e.g., an image, a set of images, and/or a horizontal plane (e.g., floor plane 5038)) that is detected within the field of view 6036 of one or more cameras. Video) at least in part. In some embodiments, generating the reflective image includes generating a spherical model comprising the captured image data (eg, by mapping the captured image data onto a model of a virtual sphere).

In some embodiments, the reflection image created on the surface of the object includes a reflection gradient (eg, such that a portion of the surface closer to the plane has a higher magnitude of reflectivity than a portion of the surface further away from the plane). In some embodiments, the magnitude of the reflectivity of the reflection generated on the surface of the object is based on the reflectivity value of the texture corresponding to the surface. For example, no reflective image is generated in the non-reflective portion of the surface.

In some embodiments, the reflective image adjusts over time. For example, the reflective image is adjusted as input to move and/or scale the object is received (e.g., as the object moves, the reflective image of the object reflects a portion of the plane at the location corresponding to the object. Is adjusted). In some embodiments, the reflective image is not adjusted when the object is rotated (eg, about the z-axis).

In some embodiments, prior to displaying the object at a determined location (eg, on a plane detected within the field of view 6036 of the camera(s) corresponding to the object), a reflective image is not created on the surface of the object. For example, when a translucent representation of the object is displayed (e.g., as described with respect to FIGS. 11G and 11H) and/or a calibration is performed (e.g., as described with respect to FIGS. 12B-12I). When doing so, no reflection is created on the surface of the object.

In some embodiments, the reflective image of the object is created on one or more planes that are detected within the field of view 6036 of the camera(s). In some embodiments, the reflective image of the object is not created within the field of view 6036 of the camera(s).

16A-16G are flow charts illustrating a method 16000 of displaying a virtual object within a user interface that includes a field of view of one or more cameras using different visual properties depending on whether object placement criteria are met. The method 16000 includes a display generating component (e.g., a display, a projector, a head-up display, etc.), one or more input devices (e.g., a touch-sensitive surface, or a touch serving as both a display-generating component and a touch-sensitive surface). Screen display), and an electronic device with one or more cameras (e.g., one or more rear cameras on the side of the device opposite the display and the touch-sensitive surface) (e.g., device 300 of FIG. 3, or It is carried out in a portable multifunction device 100). In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 16000 are selectively combined and/or the order of some operations is selectively changed.

The device comprises a first user interface area (e.g., while a staging user interface comprising a movable representation of the virtual object is displayed, and before the field of view of the cameras is displayed) comprising at least a portion of the field of view of one or more cameras (e.g., A request to display a virtual object (e.g., a representation of a 3D model) in the augmented reality observer interface) is received (16002) (e.g., the request is input by a touch detected on the representation of the virtual object on the touch screen display, or Is detected on an affordance that is displayed concurrently with the representation of the virtual object and configured to trigger the display of the AR view when invoked by the first contact (eg, a tap on the “AR view” or “world view” button)). For example, the request is an input to display the virtual object 11002 within the field of view 6036 of one or more cameras, as described in connection with FIG. 11F.

In response to a request to display a virtual object in the first user interface area (e.g., a request to display a virtual object in a view of the physical environment around the device), the device is included in the first user interface area, through a display generating component Display a representation of the virtual object over at least a portion of the field of view of one or more cameras (e.g., the field of view of one or more cameras is displayed in response to a request to display the virtual object within the first user interface area), the field of view of one or more cameras Is a view of the physical environment in which one or more cameras are located (16004 ). For example, as described in connection with FIG. 11G, the virtual object 11002 is displayed within the field of view 6036 of one or more cameras, which is a view of the physical environment 5002 in which the one or more cameras are located. Displaying the representation of the virtual object is based on the determination that the object placement criteria are not met-the object placement criteria are determined that the placement position (e.g., plane) for the virtual object is within the field of view of one or more cameras so that the object placement criteria are satisfied. Required to be identified (e.g., object placement criteria are not met when the device has not identified a location or plane for placing the virtual object relative to the field of view of one or more cameras within the first user interface area (e.g., plane identification is Still in progress, or there is not enough image data to identify the plane))-with a first set of visual attributes (e.g., with a first translucency level, or a first brightness level, or a first saturation level, etc.) ) And displaying the representation of the virtual object in a first orientation independent of which portion of the physical environment is displayed within the field of view of one or more cameras (e.g., the virtual object is relative to a predefined plane independent of the physical environment). It floats above the camera's field of view in an orientation (eg, an orientation set within the staging view) and in an orientation independent of the change that occurs in the camera's field of view (eg, due to movement of the device relative to the physical environment). For example, in FIGS. 11G and 11H, because the placement position for the virtual object 11002 has not been identified within the field of view 6036 of the cameras, the translucent form of the virtual object 11002 is displayed. As the device moves (as shown from FIG. 11G to FIG. 11H), the orientation of the virtual object 11002 does not change. In some embodiments, the object placement criteria include the requirement that the field of view is stable and provides a still view of the physical environment (e.g., the camera moves below a threshold amount for at least a threshold time and/or at least since the request is received. A predetermined time has elapsed and/or the camera has been calibrated for plane detection with sufficient previous movement of the device). Depending on the determination that the object placement criteria are satisfied (e.g., the object placement criteria are satisfied when the device has not identified a location or plane for placing the virtual object with respect to the field of view of one or more cameras within the first user interface area), The device has a second set of visual attributes that are distinct from the first set of visual attributes (e.g., with a second translucency level, or a second brightness level, or a second saturation level, etc.) and within the field of view of one or more cameras. Display the representation of the virtual object in a second orientation corresponding to the plane in the physical environment to be detected. For example, in FIG. 11I, since the placement location for the virtual object 11002 (e.g., the plane corresponding to the floor surface 5038 in the physical environment 5002) has been identified within the field of view 6036 of the cameras, the virtual The non-translucent shape of the object 11002 is displayed. The orientation of the virtual object 11002 (eg, a position on the touch screen display 112) has been changed from the first orientation shown in FIG. 11H to the second orientation shown in FIG. 11I. As the device moves (as shown in Figure 11I to Figure 11J), the orientation of the virtual object 11002 changes (since the virtual object 11002 is now displayed in a fixed orientation relative to the physical environment 5002). do. Depending on whether the object placement criteria are met, displaying a virtual object with a first set of visual attributes or a second set of visual attributes (e.g., a request to display a virtual object has been received, but one or more Visual feedback is provided to the user (indicating that additional time and/or calibration information is needed to place within the camera's field of view). Providing improved visual feedback to the user (e.g., prior to placement of the object in a second orientation corresponding to the plane helps the user to provide appropriate inputs and to avoid attempts to provide input to manipulate the virtual object). Thereby) improving the operability of the device and making the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device faster and more efficiently.

In some embodiments, the device detects that object placement criteria are met while the representation of the virtual object is displayed in a first set of visual properties and a first orientation (e.g., a plane for placing the virtual object). Is identified while the virtual object is suspended in a translucent state over a view of the physical environment around the device). Detecting that object placement criteria are met while the virtual object is displayed (e.g., in a translucent state) with a first set of visual properties, without requiring additional user input to initiate detection of the object placement criteria Reduces the number of inputs required for deployment. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, Reduces power usage and improves battery life.

In some embodiments, in response to detecting that the object placement criteria are met, the device moves (eg, rotates and/or scales and/or from the first orientation to the second orientation, via the display generation component). Or, translate and/or move to a combination of the above) and display an animated transition showing the representation of a virtual object that changes from having a first set of visual properties to having a second set of visual properties (16008). For example, once a plane for placing a virtual object is identified within the camera's field of view, the virtual object is placed on that plane with visual adjustments of its orientation, size, and translucency (etc.). Displaying an animated transition from a first orientation to a second orientation (eg, without requiring additional user input to reorient the virtual object in the first user interface) reduces the number of inputs required for object placement. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, Reduces power usage and improves battery life.

In some embodiments, detecting that object placement criteria are met includes detecting that a plane has been identified within the field of view of the one or more cameras; Detecting less than a threshold amount of movement between the device and the physical environment for at least a critical time period (eg, leading to a substantially stationary view of the physical environment within the field of view of the camera); And detecting that at least a predetermined time has elapsed since receiving the request to display the virtual object in the first user interface area (16010). Detecting that object placement criteria are met (eg, by detecting a plane within the field of view of one or more cameras without requiring user input to detect the plane) reduces the number of inputs required for object placement. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, Reduces power usage and improves battery life.

In some embodiments, the device provides a representation of the virtual object over a first portion of the physical environment that is captured within the field of view of one or more cameras (e.g., the first portion of the physical environment is visible to the user through a translucent virtual object). The first movement of one or more cameras (e.g., physical environment around the device) while displayed in a first orientation (e.g., while a virtual object is suspended translucent over a view of the physical environment around the device) Rotation and/or translation of the device relative to (16012). For example, in Figures 11G and 11H, one or more cameras are displayed (e.g., indicated by the altered position of the table 5004 within the field of view 6036 of the cameras) while the translucent representation of the virtual object 11002 is displayed. As follows). The walls and tables of the physical environment as captured within the cameras' field of view 6036 and displayed in the user interface are visible through the translucent virtual object 11002. In response to detecting the first movement of one or more cameras, the device displays a representation of the virtual object in a first orientation and a first set of visual properties over a second portion of the physical environment that is captured within the field of view of the one or more cameras. The second portion of the physical environment is distinguished from the first portion of the physical environment (16014). For example, while the translucent form of the virtual object is displayed as hovering over the physical environment visible in the camera's field of view, the view of the physical environment within the camera's field of view is displayed as the device moves relative to the physical environment (e.g. ) It is shifted and scaled. Thus, during the movement of the device, the translucent shape of the virtual object is overlaid on top of different parts of the physical environment represented within the field of view, as a result of the translation and scaling of the view of the physical environment within the field of view of the camera. For example, in FIG. 11H, the field of view 6036 of the cameras displays a second portion of the physical environment 5002 that is distinct from the first portion of the physical environment 5002 displayed in FIG. 11G. The orientation of the translucent representation of the virtual object 11002 does not change when movement of one or more cameras occurs in FIGS. 11G and 11H. Displaying the virtual object in a first orientation in response to detecting the movement of one or more cameras (e.g., the virtual object has not yet been placed in a fixed position relative to the physical environment, and thus is captured within the field of view of one or more cameras. Provides visual feedback to the user indicating that a portion of the physical environment does not move as it changes with the movement of one or more cameras. Providing improved visual feedback to the user can be achieved by helping the user avoid attempts to provide input to manipulate the virtual object (e.g., prior to placement of the object in a second orientation corresponding to the plane). And makes the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, the device provides a direct view of a third portion of the physical environment (e.g., a third portion of the physical environment (e.g., a portion of the detected plane supporting the virtual object) that is captured within the field of view of one or more cameras). Blocked by the virtual object) while the representation of the virtual object is displayed in a second set of visual properties and a second orientation (e.g., object placement criteria have been met and the virtual object is detected in the physical environment within the field of view of the cameras). After being placed in) a second movement of one or more cameras (eg, rotation and/or translation of the device relative to the physical environment around the device) is detected (16016). For example, in FIGS. 11I and 11J, one or more cameras may be placed in a changed position of the table 5004 within the field of view 6036 of the cameras (e.g., while a non-translucent representation of the virtual object 11002 is displayed. Move as indicated by). In response to detecting the second movement of the device, the device moves (e.g., shifts and scales) the physical environment as captured within the field of view of the one or more cameras according to the second movement of the device, and the second orientation A virtual object in a second orientation and a second set of visual properties over a third portion of the physical environment captured within the field of view of one or more cameras, while continuing to correspond to a plane within the physical environment detected within the field of view of these one or more cameras. Maintain the display of the representation of (16018). For example, after the non-translucent form of the virtual object falls from a stationary position on the plane detected in the physical environment shown in the camera's field of view, the position and orientation of the virtual object is fixed with respect to the physical environment within the camera's field of view, and The object will be shifted and scaled with the physical environment within the field of view of the cameras as the device moves relative to the physical environment (e.g., the non-translucent representation of the virtual object 11002 is that the movement of one or more cameras is As it occurs, it remains fixed in an orientation relative to the floor plane within the physical environment 5002). Maintaining displaying the virtual object in a second orientation in response to detecting the movement of one or more cameras (e.g., the virtual object has been placed in a fixed position relative to the physical environment, thus capturing within the field of view of one or more cameras). Visual feedback to the user indicating that a portion of the physical environment being moved as it changes as one or more cameras move. Providing improved visual feedback to the user improves the operability of the device and improves the user-device interface (e.g., by helping the user provide appropriate inputs for a virtual object placed in a second orientation corresponding to the plane). It makes it more efficient, which in addition reduces the device's power usage and improves battery life by enabling users to use the device more quickly and efficiently.

In some embodiments, object placement criteria are met (e.g., when the device has not identified a location or plane to place the virtual object with respect to the field of view of one or more cameras within the first user interface area. Depending on the determination, the device may have a second set of visual properties (e.g., with a reduced translucency level, or a higher brightness level, or a higher saturation level, etc.) and the physical environment detected within the field of view of one or more cameras. Generate a tactile output (e.g., with one or more tactile output generators of the device) along with displaying the representation of the virtual object in a second orientation corresponding to the plane within the Synchronized with the completion of the transition to the non-translucent appearance and the completion of the rotation and translation of the virtual object to settle in the falling position on the plane detected in the physical environment) (16020). For example, as shown in Fig. 11I, with displaying a non-translucent representation of the virtual object 11002 attached to the plane (e.g., floor surface 5038) corresponding to the virtual object 11002 A tactile output as indicated is produced. Generating the tactile output in accordance with the determination that the object placement criteria are met provides improved tactile feedback to the user (eg, indicating that the operation of placing the virtual object has been successfully executed). Providing improved feedback to the user improves the operability of the device and improves the operability of the user (e.g., by providing sensory information that makes the user aware that object placement criteria have been met without cluttering the user interface with the displayed information). -Make the device interface more efficient, which in addition, reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, while displaying a representation of the virtual object with a second set of visual properties and in a second orientation corresponding to a plane in the physical environment detected within the field of view of the one or more cameras, the device Receive an update regarding at least the position or orientation of the plane in the physical environment that is detected within the field of view (e.g., the updated plane position and orientation is based on additional data accumulated after the initial plane detection result is used to place the virtual object. It is the result of a more accurate calculation, or more time consuming calculation methods (eg, fewer approximations, etc.)) 16022. In response to receiving an update regarding at least the position or orientation of the plane in the physical environment detected within the field of view of the one or more cameras, the device adjusts at least the position and/or orientation of the representation of the virtual object according to the update (e.g., The virtual object is gradually moved (eg, translated and rotated) closer to the updated plane (16024). Adjusting the position and/or orientation of the virtual object in response to receiving an update regarding the plane within the physical environment (e.g., without requiring user input to place the virtual object relative to the plane) can be used to adjust the virtual object. Reduce the number of inputs required Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, Reduces power usage and improves battery life.

In some embodiments, the first set of visual properties includes a first size and a first translucency level (e.g., before falling into an AR view, the object has a fixed size and a fixed high translucency level for the display). (16026), the second set of visual properties are displayed with a second size that is distinct from the first size (e.g., once dropped in the AR view, the object is displayed with a simulated physical size in relation to the drop location and size in the physical environment) ) And a second translucency level (eg, the object is no longer translucent in the AR view) that is lower (eg, more opaque) than the first translucency level (16028). For example, in FIG. 11H, a translucent representation of the virtual object 11002 is shown in a first size, and in FIG. 11I, a non-translucent representation of the virtual object 11004 is shown in a second (smaller) size. have. Depending on whether the object placement criteria are met, displaying a virtual object at a first size and a first translucency level or at a second size and a second translucency level (e.g., a request to display the virtual object has been received, but the virtual object Visual feedback is provided to the user indicating that additional time and/or calibration information is needed to place it within the field of view of one or more cameras. Providing improved visual feedback to the user (e.g., prior to placement of the object in a second orientation corresponding to the plane helps the user to provide appropriate inputs and to avoid attempts to provide input to manipulate the virtual object). Thereby) improving the operability of the device and making the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device faster and more efficiently.

In some embodiments, while the virtual object is displayed within each user interface (e.g., a staging user interface) that does not include at least a portion of the field of view of one or more cameras (e.g., the virtual object has an orientation independent of the physical environment of the device) Oriented relative to a virtual stage having a), a request to display a virtual object within a first user interface area (eg, AR view) that includes at least a portion of the field of view of one or more cameras is received (16030 ). The first orientation corresponds to the orientation of the virtual object while the virtual object is displayed in each user interface at the time the request is received. For example, as described in connection with FIG. 11F, a virtual object 11002 within the user interface that includes the camera's field of view 6036 while the staging user interface 6010 (not including the camera's field of view) is displayed. A request to display is received. The orientation of the virtual object 11002 of FIG. 11G in which the virtual object 11002 is displayed within the user interface including the field of view 6036 of the cameras is that of FIG. 11F in which the virtual object 11002 is displayed within the staging user interface 6010. It corresponds to the orientation of the virtual object 11002. Displaying the virtual object in the first user interface (e.g., a displayed augmented reality view) in an orientation corresponding to the orientation of the virtual object as displayed in the (previously displayed) interface (e.g., staging user interface) is (e.g. , Providing visual feedback to the user indicating that the object manipulation input provided while the staging user interface is displayed can be used to establish the orientation of the object within the AR view. Providing improved visual feedback to the user (e.g., prior to placement of the object in a second orientation corresponding to the plane helps the user to provide appropriate inputs and to avoid attempts to provide input to manipulate the virtual object). Thereby) improving the operability of the device and making the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device faster and more efficiently.

In some embodiments, the first orientation is a predefined orientation (e.g., the orientation in which the virtual object is displayed when the virtual object is first displayed within each user interface that does not include at least a portion of the field of view of one or more cameras). Default orientation) (16032). Displaying a virtual object in a first user interface (e.g., a displayed augmented reality view) with a first set of visual properties and in a predefined orientation results in a translucent representation (e.g., according to an orientation established within the staging user interface). By allowing a pre-generated translucent representation of the virtual object to be displayed rather than rendering it), it reduces the power usage of the device and improves battery life.

In some embodiments, the virtual object is placed within a first user interface area (e.g., AR view) with a second set of visual properties and in a second orientation corresponding to a plane in the physical environment detected within the field of view of one or more cameras. During display, the device (e.g., as a result of a scaling input (e.g., a pinch or de-pinch gesture on the virtual object)) of the virtual object relative to the physical environment being captured within the field of view of one or more cameras A request to change from a first simulated physical size to a second simulated physical size (eg, from 80% of the default size to 120% of the default size, or vice versa) is detected (16034). For example, the input for reducing the simulated physical size of the virtual object 11002 is a pinch gesture as described in connection with Figs. 11N-11P. In response to detecting a request to change the simulated physical size of the virtual object, the device causes the first user to change the virtual object's simulated physical size from the first simulated physical size to a second simulated physical size. Progressively changing the displayed size of the representation of the virtual object within the interface area (e.g., the displayed size of the virtual object increases while the displayed size of the physical environment being captured within the field of view of one or more cameras remains unchanged). Or decrease), during the gradual change of the displayed size of the representation of the virtual object in the first user interface area, the simulated physical size of the virtual object has reached a predefined simulated physical size (e.g., 100% of the default size). Upon determination, the device generates a tactile output to indicate that the simulated physical size of the virtual object has reached a predefined simulated physical size (16036). For example, as described in connection with FIGS. 11N-11P, the displayed size of the representation of the virtual object 11002 gradually decreases in response to a pinch gesture input. In FIG. 11O, the displayed size of the representation of the virtual object 11002 is 100% of the size of the virtual object 11002 (e.g., within the user interface including the field of view 6036 of one or more cameras, as shown in FIG. When reaching the size of the virtual object 11002 as originally displayed), a tactile output as indicated by 11024 is generated. Generating a tactile output based on the determination that the simulated physical size of the virtual object has reached a predefined simulated physical size (e.g., additional input is required to return the simulated size of the virtual object to the predefined size). Provide feedback to the user indicating that they are not needed. Providing improved tactile feedback (e.g., by providing sensory information that makes the user aware that a predefined simulated physical size of a virtual object has been reached without cluttering the user interface with the displayed information). It improves the operability of the device, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, as a result of a second simulated physical size of the virtual object (e.g., a scaling input (e.g., a pinch or de-pinch gesture on the virtual object) that is distinct from the predefined simulated physical size), the default size 120%, or 80% of the default size) while displaying the virtual object in the first user interface area (e.g., AR view), the device detects a request to return the virtual object to a predefined simulated physical size. (Eg, detecting a tap or double tap input on a touch screen (eg, on a virtual object, or alternatively outside of a virtual object)) 16038.

For example, after a pinch input causes a reduction in the size of the virtual object 11002 (as described with respect to Figs. 11N to 11P), the double tap input results in a virtual (as described with respect to Fig. 11R). It is detected at a location corresponding to the object 11002. In response to detecting a request to return the virtual object to the predefined simulated physical size, the device is in the first user interface area according to the change of the simulated physical size of the virtual object to the predefined simulated physical size. Change the displayed size of the representation of the virtual object (e.g., the displayed size of the virtual object increases or decreases while the displayed size of the physical environment being captured within the field of view of one or more cameras remains unchanged) 16040 ). For example, in response to a double tap input described in connection with FIG. 11R, the size of the virtual object 11002 is determined by the size of the virtual object 11002 as displayed in FIG. 11I (the field of view 6036 of one or more cameras). Return to the size of the virtual object 11002 as originally displayed in the containing user interface). In some embodiments, upon determining that the simulated physical size of the virtual object has reached a predefined simulated physical size (e.g., 100% of the default size), the device determines that the simulated physical size of the virtual object is predefined. A tactile output is generated to indicate that the simulated physical size has been reached. Changing the displayed size of the virtual object to a predefined size in response to detecting a request to return the virtual object to the predefined simulated physical size (e.g., the input provided to adjust the display size is Rather than requiring the user to estimate when it is sufficient to display the object at a predefined simulated physical size, it is possible to predefine the object (by providing the option to precisely adjust the displayed size to the predefined simulated physical size). Reduces the number of inputs required to display in size. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, Reduces power usage and improves battery life.

In some embodiments, the device has a second orientation of the representation of the virtual object having a second set of visual properties according to the location and orientation of each of the one or more cameras relative to the physical environment (e.g., when object placement criteria are met). Select a plane to set the current position and orientation), where selecting a plane means that the representation of the virtual object is within the field of view of one or more cameras (e.g., as a result of the device being oriented in a first direction in the physical environment). When displayed over a first portion of the physical environment being captured (e.g., the base of a translucent object overlaps a plane within the first portion of the physical environment), depending on the determination that the object placement criteria have been met, the second set of visual properties is determined. A first plane among a plurality of planes detected in the physical environment within the field of view of one or more cameras as a plane for setting the second orientation of the representation of the virtual object having (e.g., more between the first plane on the display and the base of the object Choosing according to the greater proximity and greater proximity between the first plane and the first portion of the physical environment within the physical world; And when the representation of the virtual object is displayed over a second portion of the physical environment that is captured within the field of view of one or more cameras (e.g., as a result of the device being oriented in a second direction in the physical environment) (e.g., the base of the translucent object is Overlapping with the plane within the second part of the physical environment) the field of view of one or more cameras as a plane for establishing a second orientation of the representation of a virtual object having a second set of visual properties, upon determination that the object placement criteria have been met The second of the plurality of planes detected in the physical environment within the second plane (e.g., greater proximity between the second plane on the display and the base of the object and the greater between the second plane and the second part of the physical environment within the physical world) Selecting according to proximity), wherein the first portion of the physical environment is distinct from the second portion of the physical environment, and the first plane is distinguished from the second plane (16042). Selecting the first plane or the second plane as the plane on which the virtual object is to be set (e.g., without requiring user input to specify which of the many detected planes is the plane on which the virtual object is set) Reduces the number of inputs required to select a plane. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, Reduces power usage and improves battery life.

In some embodiments, the device displays a virtual object in a first user interface area (e.g., AR view) with a second set of visual attributes and a second orientation, while simultaneously displaying a snapshot affordance (e.g., camera shutter). Button) is displayed (16044). In response to activation of the snapshot affordance, the device generates a snapshot image that includes a current view of a representation of the virtual object at a location within the physical environment within the field of view of the one or more cameras, a second set of visual attributes and one or more. Capture 16046 with a second orientation corresponding to the plane in the physical environment detected within the field of view of the cameras. Displaying the snapshot affordance for capturing a snapshot image of the current view of the object reduces the number of inputs required to capture the snapshot image of the object. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, Reduces power usage and improves battery life.

In some embodiments, the device exits one or more control affordances (e.g., affordance to switch back to the staging user interface, AR viewer) with a representation of a virtual object having a second set of visual properties within the first user interface area. An affordance to exit, an affordance to capture a snapshot, etc.) are displayed (16048). For example, in FIG. 11J, a set of control units including a back control unit 6016, a toggle control unit 6018, and a sharing control unit 6020 are displayed. While displaying one or more control affordances with a representation of a virtual object having a second set of visual properties, the device indicates that the control-fading criteria are met (e.g., movement of the device and field of view of the cameras (With or without an update to) detect (16050) that no user input has been detected on the touch-sensitive surface for a threshold time. In response to detecting that the control unit fading criteria are met, the device provides one or more control unit affordances while continuing to display a representation of a virtual object having a second set of visual properties within a first user interface area that includes the field of view of the one or more cameras. Stop displaying them (16052). For example, as described in connection with FIGS. 11K and 11L, the controls 6016, 6018, 6020 gradually fade out and stop being displayed when no user input is detected for a threshold time. In some embodiments, after the control affordances fade out, tap input on the touch-sensitive surface or interaction with the virtual object causes the device to redisplay the control affordances simultaneously with the representation of the virtual object in the first user interface area. do. Automatically stopping displaying the controls in response to determining that the control fading criteria are met reduces the number of inputs required to stop displaying the controls. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, Reduces power usage and improves battery life.

In some embodiments, in response to a request to display the virtual object within the first user interface area, prior to displaying the representation of the virtual object over at least a portion of the field of view of one or more cameras included in the first user interface area, (e.g. In response to the determination that the calibration criteria are not met, the device is determined by the user, because there are not enough images from different viewing angles to generate dimensional and spatial relationship data for the physical environment captured within the field of view of one or more cameras. Display a prompt to move the device relative to the physical environment (e.g., as described in more detail below with reference to method 17000), and optionally, a first user interface area Display a calibration user interface object (e.g., a bouncy wireframe ball or cube that moves as the device moves) (e.g., the calibration user interface object is on a blurred image of the field of view of one or more cameras Overlaid)) 16054. Displaying a prompt for the user to move the device relative to the physical environment provides user visual feedback (e.g., indicating that movement of the device is necessary to obtain information to place the virtual object within the field of view of the camera(s)). To provide. Providing improved visual feedback to the user improves the operability of the device (e.g., by helping the user to provide a calibration input) and makes the user-device interface more efficient, which in addition allows the user to make the device faster. It reduces the power usage of the device and improves battery life by enabling it to be used efficiently.

It should be understood that the specific order in which the operations in FIGS. 16A-16G are described is merely an example and is not intended to indicate that the described order is the only order in which the operations may be performed. One of skill in the art will recognize various ways of reordering the operations described herein. Additionally, details of other processes described herein in connection with other methods described herein (e.g., methods 800, 900, 1000, 17000, 18000, 19000, 20000) are provided in FIGS. It should be noted that it is also applicable in a manner similar to method 16000 described above in connection with FIG. 16G. For example, contacts, inputs, virtual objects, user interface regions, fields of view, tactile outputs, movements, and/or animations described above in connection with method 16000 may optionally be described herein. Contacts, inputs, virtual objects, user interface regions, field of view described herein in connection with other methods described (e.g., methods 800, 900, 1000, 17000, 18000, 19000, 20000) S, tactile outputs, movements, and/or one or more of the characteristics of animations. For brevity, these details are not repeated here.

17A-17D are flowcharts illustrating a method 17000 of displaying a calibration user interface object that is dynamically animated according to movement of one or more cameras of the device. The method 17000 includes a display generating component (e.g., a display, a projector, a head-up display, etc.), one or more input devices (e.g., a touch-sensitive surface, or a touch serving as both a display-generating component and a touch-sensitive surface Screen display), one or more cameras (e.g., one or more rear cameras on the face of the device opposite the display and the touch-sensitive surface), and the attitude of the device (e.g., relative to the surrounding physical environment) Electronic device with one or more attitude sensors (e.g., accelerometers, gyroscopes, and/or magnetometers) to detect changes in orientation (e.g., rotation angle, yaw angle, and/or tilt angle) and position) (E.g., device 300 of FIG. 3, or portable multifunction device 100 of FIG. 1A). Some operations in method 17000 are selectively combined and/or the order of some operations is selectively changed.

The device includes a representation of the field of view of one or more cameras (e.g., the field of view captures at least a portion of the physical environment) of the physical environment (e.g., the physical environment around the device including one or more cameras) within a A request to display an augmented reality view is received (17002). In some embodiments, the request is a tap input detected on a button to switch from the staging view of the virtual object to the augmented reality view of the virtual object. In some embodiments, the request is a selection of an augmented reality affordance that is displayed following the representation of the virtual object within the two-dimensional user interface. In some embodiments, the request is activation of an augmented reality measurement application (eg, a measurement app that facilitates measurement of a physical environment). For example, the request is a tap input detected at toggle 6018 to display virtual object 11002 within field of view 6036 of one or more cameras, as described in connection with FIG. 12A.

In response to receiving a request to display an augmented reality view of the physical environment, the device displays a representation of the field of view of one or more cameras (e.g., the device displays a representation of the physical environment within the field of view of the one or more cameras when calibration criteria are not met. Display the blurred shape) (17004). For example, the device displays a blurred representation of the field of view 6036 of one or more cameras, as shown in FIG. 12E-1. Since there is not a sufficient amount of image data (e.g., from different viewing angles) to generate dimensional and spatial relationship data for the physical environment that is captured within the field of view of one or more cameras, the plane corresponding to the virtual object is Because it is not detected within the field of view of one or more cameras, and/or because there is not enough information to initiate or proceed with the plane detection based on the image data available from the cameras) the calibration criteria are not in the augmented reality view of the physical environment. Upon a determination that it is not satisfied, the device is in a physical environment (e.g., via a display generating component, and in a first user interface region that contains a representation of the field of view of one or more cameras (e.g., a blurred form of the field of view)). Displays a calibration user interface object (eg, a scan prompt object such as an elastic cube or wireframe object) that is dynamically animated according to the movement of one or more cameras of. For example, in FIGS. 12E-1 to 12I-1, a calibration user interface object 12014 is displayed. Animation of a calibration user interface object according to movement of one or more cameras is described, for example, with reference to FIGS. In some embodiments, analyzing the field of view of one or more cameras to detect one or more planes (e.g., floor, wall, table, etc.) within the field of view of the one or more cameras requires displaying a representation of the augmented reality view. Occurs when the initial portion of the input corresponding to is received. In some embodiments, the analysis occurs prior to receiving the request (eg, while the virtual object is displayed in the staging view). Displaying the calibration user interface object includes, through one or more attitude sensors, the attitude (e.g., position and/or orientation (e.g., rotation angle, tilt angle) of one or more cameras in the physical environment), while displaying the calibration user interface object. , Yaw angle)), and in response to detecting a change in the attitude of one or more cameras in the physical environment, a calibration user interface object according to the detected change in the attitude of one or more cameras in the physical environment. (Eg, a scan prompt object such as a resilient cube or wireframe object) at least one display parameter (eg, orientation, size, rotation, or position on the display). For example, FIGS. 12E-1 and 12F-1 corresponding to FIGS. 12E-2 and 12F-2 respectively show the lateral movement of the device 100 relative to the physical environment 5002, and of one or more cameras of the device. Illustrative of a corresponding change in the displayed field of view 6036. 12E-2 and 12F-2, the calibration user interface object 12014 rotates in response to movement of one or more cameras.

While displaying a calibration user interface object (e.g., a scan prompt object such as an elastic cube or wireframe object) that moves on the display in response to a detected change in the attitude of one or more cameras in the physical environment, the device Is detected (17006). For example, as described in connection with FIGS. 12E-12J, the device determines that calibration criteria are met in response to movement of the device occurring from FIGS. 12E-1 to 12I-1.

In response to detecting that the calibration criteria are met, the device stops displaying a calibration user interface object (eg, a scan prompt object such as an elastic cube or wireframe object) (17008). In some embodiments, after the device stops displaying the calibration user interface object, the device displays a representation of the camera's field of view without blurring. In some embodiments, the representation of the virtual object is displayed over the unblurred representation of the cameras' field of view. For example, in FIG. 12J, in response to movement of the device described in connection with FIGS. 12E-1 to 12I-1, the calibration user interface object 12014 is no longer displayed, and the virtual object 11002 is Displayed over the unblurred representation 6036 of the field of view of the(s). Adjusting the display parameter of the calibration user interface object according to the movement of one or more cameras (e.g., device cameras capturing the physical environment of the device) provides visual feedback (e.g., indicating that the movement of the device is required for calibration). Provides to the user. Providing improved visual feedback to the user improves the operability of the device and further enhances the user-device interface (e.g., by helping the user move the device in a way that provides the information needed to meet calibration criteria). It makes it efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, the request to display an augmented reality view of a physical environment (e.g., a physical environment around a device including one or more cameras) within a first user interface area containing a representation of the field of view of one or more cameras And a request to display a representation of a virtual three-dimensional object (eg, a virtual object having a three-dimensional model) in the augmented reality view of (17010). In some embodiments, the request is a tap input detected on a button to switch from the staging view of the virtual object to the augmented reality view of the virtual object. In some embodiments, the request is a selection of an augmented reality affordance that is displayed following the representation of the virtual object within the two-dimensional user interface. For example, in FIG. 12A, the input by the contact 12002 at a position corresponding to the toggle control unit 6018 is a virtual object in the user interface including the field of view 6036 of the cameras, as shown in FIG. 12B. It is a request to display (11002). Displaying the augmented reality view of the physical environment in response to a request to display the virtual object within the augmented reality view reduces the number of inputs required (eg, to display both the view of the physical environment and the virtual object). Reducing the number of inputs required to perform an operation improves the operability of the device (e.g., by helping the user provide a calibration input) and makes the user-device interface more efficient, which additionally allows the user to configure the device. It reduces device power usage and improves battery life by enabling faster and more efficient use.

In some embodiments, after the device stops displaying the calibration user interface object, the device displays a representation of the virtual three-dimensional object within the first user interface area that includes a representation of the field of view of one or more cameras (e.g., calibration criteria are met. After) display (17012). In some embodiments, in response to the request, after calibration is complete and the camera's field of view is displayed completely clearly, the virtual object is a predefined plane (e.g., a three-dimensional representation of the virtual object) identified in the field of view of one or more cameras. It falls in a predefined position and/or orientation relative to a physical surface (such as a vertical wall or horizontal floor surface) that can serve as a support plane for a. For example, in Fig. 12J, the device has stopped displaying the calibration user interface object 12014 that was displayed in Figs. Is displayed. Displaying the virtual object within the displayed augmented reality view after stopping displaying the calibration user interface object provides visual feedback (eg, to indicate that calibration criteria have been met). Providing improved visual feedback to the user can facilitate the operability of the device (e.g., by helping the user to provide appropriate inputs before calibration criteria are met and to avoid attempts to provide input to manipulate the virtual object). It improves and makes the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, the device displays a representation of the virtual three-dimensional object within the first user interface area simultaneously with the calibration user interface object (e.g., behind the calibration user interface object) (e.g., before the calibration criteria are met), the virtual The representation of the 3D object is a fixed position within the first user interface area during movement of one or more cameras in the physical environment (e.g., while the calibration user interface object is moved within the first user interface area according to movement of one or more cameras) (E.g., the virtual three-dimensional object is not placed at a location within the physical environment) (17014). For example, in FIGS. 12E-1 to 12I-1, the representation of the virtual object 1102 is displayed simultaneously with the calibration user interface object 12014. As the device 100 including one or more cameras moves (e.g., as illustrated in FIGS. 12E-1 and 12F-1 and the corresponding FIGS. 12E-2 and 12F-2), the virtual object 1102 Is maintained in a fixed position within the user interface including the field of view 6036 of one or more cameras. Displaying the virtual object concurrently with the calibration user interface object provides visual feedback (eg, to indicate the object being calibrated). Providing improved visual feedback to the user improves the operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide a calibration input corresponding to the plane in which the virtual object will be placed). This additionally reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, the request to display an augmented reality view of a physical environment (e.g., a physical environment around a device including one or more cameras) within a first user interface area that includes a representation of the field of view of one or more cameras is one or more. Without requesting display of a representation of any virtual three-dimensional object (e.g., a virtual object with a three-dimensional model) in the physical environment captured within the field of view of the cameras (e.g., one or more user interface objects and/or controls ( (E.g., contours of planes, objects, pointers, icons, markers, etc.)) and a request to display a representation of the field of view of one or more cameras (17016). In some embodiments, the request is a selection of an augmented reality affordance that is displayed following the representation of the virtual object within the two-dimensional user interface. In some embodiments, the request is activation of an augmented reality measurement application (eg, a measurement app that facilitates measurement of a physical environment). Requesting to display a representation of the field of view of one or more cameras without requesting display of a representation of any virtual 3D object (e.g., the same calibration user to indicate that calibration is required regardless of whether or not the virtual object is displayed) By using an interface object) to provide feedback. Providing improved feedback to the user improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device by allowing the user to use the device more quickly and efficiently. And improve battery life.

In some embodiments, in response to receiving a request to display an augmented reality view of the physical environment, the device displays a representation of the field of view of one or more cameras (e.g., the field of view of one or more cameras when calibration criteria are not met). Displays a blurred form of the physical environment within); Since there is a sufficient amount of image data (e.g., from different viewing angles) to generate dimensional and spatial relationship data for the physical environment that is captured within the field of view of one or more cameras, the plane corresponding to the virtual object is Because it was detected within the field of view of one or more cameras, and/or because there is sufficient information to initiate or proceed with the plane detection based on the image data available from the cameras), the calibration criteria are for an augmented reality view of the physical environment. Upon determining that it is satisfied, the device suspends (17018) display of the calibration user interface object (eg, a scan prompt object such as an elastic cube or wireframe object). In some embodiments, the scanning of the physical environment for the planes is initiated while the virtual three-dimensional object is displayed in the staging user interface, which allows the device (e.g., the field of view of the cameras to detect one or more planes within the physical space). In some environments (moved sufficiently to provide enough data to), it is possible to detect one or more planes within the physical space prior to displaying the augmented reality view, so that the calibration user interface does not need to be displayed. Suspending the display of the calibration user interface object in accordance with the determination that the calibration criteria are met for an augmented reality view of the physical environment (e.g., no calibration user interface object is the case that calibration criteria have been met and movement of the device is required for calibration). Provide visual feedback to the user indicating that they are not. Providing improved visual feedback to the user improves the operability of the device and makes the user-device interface more efficient (e.g., by helping the user avoid unnecessarily moving the device for calibration purposes). In addition, it reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, the device is simultaneously with a calibration user interface object (e.g., after a calibration user interface object) that provides information about actions that can be taken by the user to improve calibration of the augmented reality view. 1 In the user interface area, create a text object (e.g., a text description describing the currently detected error condition and/or a text prompt requesting a user action (e.g., to correct the detected error condition)) (e.g., calibration criteria). Before being satisfied) (17020). In some embodiments, the text object prompts for movement of the device, such as “excessive movement”, “low detail”, “move closer”, etc. With the detected error status) to the user. In some embodiments, the device updates the text object according to the user's actions during the calibration process and new error conditions detected based on the user's actions. Displaying text concurrently with a calibration user interface object provides visual feedback to the user (eg, providing a verbal indication of the type of movement required for calibration). Providing improved visual feedback to the user improves the operability of the device and improves the user-device interface (e.g., by helping the user provide appropriate inputs and reducing user mistakes when actuating/interacting with the device). This makes the device more efficient, which in addition, reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, in response to detecting that the calibration criteria are satisfied (e.g., that the criteria are satisfied before the calibration user interface object is displayed, or that the criteria are satisfied after the calibration user interface object is displayed and animated for a period of time) Thus, the device displays a visual indication of the plane detected in the physical environment captured within the field of view of one or more cameras (e.g., after stopping displaying the calibration user interface object if the calibration user interface object was initially displayed). (Eg, to display an outline around the detected plane, or to highlight the detected plane) 17022. For example, in FIG. 12J, the plane (floor surface 5038) is highlighted to indicate that the plane was detected in the physical environment 5002 as captured within the displayed field of view 6036 of one or more cameras. Displaying a visual indication of the detected plane provides visual feedback (eg, indicating that the plane was detected in the physical environment being captured by the device camera(s)). Providing improved visual feedback to the user improves the operability of the device and improves the user-device interface (e.g., by helping the user provide appropriate inputs and reducing user mistakes when actuating/interacting with the device). This makes the device more efficient, which in addition, reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, in response to receiving a request to display an augmented reality view of the physical environment, in accordance with a determination that the calibration criteria are not met and prior to displaying the calibration user interface object, the device (e.g., display creation component An animated prompt object (e.g., an animated prompt object containing a representation of the device moving relative to the representation of the plane) via, and in a first user interface area containing a representation of the field of view of one or more cameras (e.g. , A scan prompt object, such as a resilient cube or wireframe object) (e.g., movement of the device's representation relative to the representation of the plane represents the required movement of the device to be made by the user) (17024). For example, the animated prompt object includes a representation 12004 of the device 100 moving relative to the representation 12010 of the plane, as described in connection with FIGS. 12B-12D. In some embodiments, the device stops displaying the animated prompt object when the device detects movement of the device (e.g., indicating that the user has started moving the device in a manner that will enable calibration to proceed. ). In some embodiments, the device replaces the display of the animated prompt object with a calibration user interface object when the device detects movement of the device and before calibration is complete to further guide the user about the device's calibration. do. For example, as described in connection with FIGS. 12C to 12E, when movement of the device is detected (as shown in FIGS. 12C and 12D), an animation including a representation 12004 of the device 100 The displayed prompt stops displaying, and a calibration user interface object 12014 is displayed in FIG. 12E. Displaying an animated prompt object containing a representation of the moving device relative to the representation of the plane provides visual feedback to the user (eg, illustrating the type of device movement required for calibration). Providing improved visual feedback to the user improves the operability of the device and further enhances the user-device interface (e.g., by helping the user move the device in a way that provides the information needed to meet calibration criteria). It makes it efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, adjusting the at least one display parameter of the calibration user interface object according to the detected change in the attitude of one or more cameras in the physical environment is dependent on the first magnitude of movement of the one or more cameras in the physical environment. Moving the calibration user interface object by a first amount accordingly; And moving the calibration user interface object by a second amount according to a second amount of movement of the one or more cameras in the physical environment, wherein the first amount is distinct from the second amount (e.g., greater than the second amount), The first magnitude of movement is distinct from the second magnitude of movement (e.g., greater than the second magnitude) (e.g., the first and second magnitudes of movement are measured based on movement in the same direction in the physical environment) (17026). Moving the calibration user interface object by an amount corresponding to the size of the movement of one or more (device) cameras (e.g., indicating to the user that the movement of the calibration user interface object is a guide to the movement of the device required for calibration) Provides. Providing improved visual feedback to the user improves the operability of the device and improves the user-device interface (e.g., by helping the user provide appropriate inputs and reducing user mistakes when actuating/interacting with the device). This makes the device more efficient, which in addition, reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, adjusting the at least one display parameter of the calibration user interface object according to the detected change in the attitude of one or more cameras in the physical environment is such that the detected change in the attitude of the one or more cameras is of a first type. Determining that it corresponds to a movement (e.g., a left-right movement, such as a leftward, a rightward, or a back-and-forth, left-right movement) (and does not correspond to a second type of movement (eg, a vertical movement such as an upward, downward, or vertical movement) According to, moving the calibration user interface object based on a first type of movement (e.g., moving the calibration user interface object in a first manner (e.g., the calibration user around a vertical axis passing through the calibration user interface object) Rotating the interface object)); And moving the calibration user interface object based on the second type of movement, according to a determination that the detected change in the pose of the one or more cameras corresponds to a second type of movement (and does not correspond to a first type of movement). Suspending (e.g., suspending moving the calibration user interface object in a first manner, or holding the calibration user interface object stationary) (17028). For example, the left and right movement of the device 100 including one or more cameras (e.g., as described in relation to FIGS. 12F-1 and 12G-1, and FIGS. 12F-2 and 12G-2) is calibrated. While causing the user interface object 12014 to rotate, the vertical movement of the device 100 (e.g., as described in connection with FIGS. 12G-1 and 12H-1, and FIGS. 12G-2 and 12H-2) Does not cause the calibration user interface object 12014 to rotate. Withholding the movement of the calibration user interface object in accordance with the determination that the detected change in the attitude of the device camera(s) corresponds to a movement of the second type (e.g., movement of the second type of one or more cameras is not necessary for calibration). Provides visual feedback to the user indicating that they are not. Providing improved visual feedback to the user improves the operability of the device (e.g., by helping the user avoid providing unnecessary input) and makes the user-device interface more efficient, which in addition allows the user to It reduces the power usage of the device and improves battery life by making it possible to use the device more quickly and efficiently.

In some embodiments, adjusting the at least one display parameter of the calibration user interface object according to the detected change in the posture of the one or more cameras in the physical environment comprises: a characteristic display position of the calibration user interface object on the first user interface area. Moving (e.g., rotating and/or) the calibration user interface object according to a detected change in the pose of one or more cameras in the physical environment without changing (e.g., the position of the axis or the geometric center of the calibration user interface object on the display). Tilt) (e.g., while the calibration user interface object is fixed in a fixed position on the display, while the physical environment moves within the field of view of one or more cameras below the calibration user interface object) (17030). For example, in FIGS. 12E-1 to 12I-1, the calibration user interface object 12014 rotates while maintaining it in a fixed position with respect to the display 112. Moving a calibration user interface object without changing the display position of the properties of the calibration user interface object (e.g., indicating that the calibration user interface object is distinct from a virtual object placed at a certain location relative to the displayed augmented reality environment) is visual feedback. Provides. Providing improved visual feedback to the user improves the operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide appropriate inputs and reducing user input mistakes). In addition, it reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, adjusting the at least one display parameter of the calibration user interface object in accordance with the detected change in the attitude of one or more cameras in the physical environment comprises an axis perpendicular to the direction of movement of the one or more cameras in the physical environment. It includes rotating the calibration user interface object around the center (e.g., the calibration user interface object rotates around the z-axis when the device (e.g., including cameras) moves back and forth on the xy plane, or For example, a device (including cameras) rotates about a y-axis as it moves left and right along the x-axis (e.g., the x-axis is defined as, for example, a horizontal direction relative to the physical environment and is within the plane of the touch screen display Placed)) (17032). For example, in FIGS. 12E-1 to 12G-1, the calibration user interface object 12014 rotates around a vertical axis perpendicular to the left and right movement of the device shown in FIGS. 12E-2 to 12G-2. Rotating the calibration user interface object about an axis perpendicular to the movement of the device camera(s) provides visual feedback (e.g., indicating to the user that the movement of the calibration user interface object is a guide to the movement of the device required for calibration). do. Providing improved visual feedback to the user improves the operability of the device and improves the user-device interface (e.g., by helping the user provide appropriate inputs and reducing user mistakes when actuating/interacting with the device). This makes the device more efficient, which in addition, reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, adjusting the at least one display parameter of the calibration user interface object in accordance with the detected change in the attitude of one or more cameras in the physical environment includes the rate of change detected within the field of view of the one or more cameras (e.g. , Moving the calibration user interface object at a speed determined according to the moving speed of the physical environment (17034). Moving the calibration user interface object at a speed determined according to the change of the device camera(s) posture (e.g., visual feedback indicating to the user that the movement of the calibration user interface object is a guide to the movement of the device required for calibration) Provides. Providing improved visual feedback to the user improves the operability of the device and improves the user-device interface (e.g., by helping the user provide appropriate inputs and reducing user mistakes when actuating/interacting with the device). This makes the device more efficient, which in addition, reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, adjusting the at least one display parameter of the calibration user interface object according to the detected change in the posture of one or more cameras in the physical environment includes a change detected within the field of view of the one or more cameras (e.g., physical It includes moving the calibration user interface object in a direction determined according to the direction of the movement speed of the environment) (e.g., the device rotates the calibration user interface object clockwise for movement of the device from right to left, and Rotate the calibration user interface object counterclockwise for device movement to the right, or the device rotates the calibration user interface object counterclockwise for device movement from right to left and Rotate the calibration user interface object clockwise in response to the movement of the device) (17036). Moving the calibration user interface object in the direction determined according to the change of the device camera(s) posture (e.g., visual feedback indicating to the user that the movement of the calibration user interface object is a guide to the movement of the device required for calibration) Provides. Providing improved visual feedback to the user improves the operability of the device and improves the user-device interface (e.g., by helping the user provide appropriate inputs and reducing user mistakes when actuating/interacting with the device). This makes the device more efficient, which in addition, reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

It should be understood that the specific order in which the operations in FIGS. 17A-17D are described is merely an example and is not intended to indicate that the described order is the only order in which the operations may be performed. One of skill in the art will recognize various ways of reordering the operations described herein. Additionally, details of other processes described herein in connection with other methods described herein (e.g., methods 800, 900, 1000, 16000, 18000, 19000, 20000) are provided in FIGS. It should be noted that it is also applicable in a manner similar to method 17000 described above with respect to FIG. 17D. For example, contacts, inputs, virtual objects, user interface regions, fields of view, tactile outputs, movements, and/or animations described above with respect to method 17000 may optionally be described herein. Contacts, inputs, virtual objects, user interface regions, field of view described herein in connection with other methods described (e.g., methods 800, 900, 1000, 16000, 18000, 19000, 20000) S, tactile outputs, movements, and/or one or more of the characteristics of animations. For brevity, these details are not repeated here.

18A-18I are flow charts illustrating a method 18000 of constraining rotation of a virtual object about an axis. The method 18000 includes a display generating component (e.g., a display, a projector, a head-up display, etc.), one or more input devices (e.g., a touch-sensitive surface, or a touch serving as both a display-generating component and a touch-sensitive surface). Screen display), one or more cameras (e.g., one or more rear cameras on the face of the device opposite the display and the touch-sensitive surface), and the attitude of the device (e.g., relative to the surrounding physical environment) Electronic device with one or more attitude sensors (e.g., accelerometers, gyroscopes, and/or magnetometers) to detect changes in orientation (e.g., rotation angle, yaw angle, and/or tilt angle) and position) (E.g., device 300 of FIG. 3, or portable multifunction device 100 of FIG. 1A). Some operations in method 18000 are selectively combined and/or the order of some operations is selectively changed.

The device displays, by the display generating component, a representation of the first viewpoint of the virtual three-dimensional object in the first user interface area (eg, staging user interface or augmented reality user interface) (18002). For example, the virtual object 11002 is shown in the staging user interface 6010, as shown in FIG. 13B.

While displaying the representation of the first viewpoint of the virtual 3D object in the first user interface area on the display, the device displays a portion of the virtual 3D object that is not visible from the first viewpoint of the virtual 3D object. A first input corresponding to a request to rotate the object relative to a display (e.g., a display plane corresponding to a display generating component such as the plane of a touch screen display) (e.g., one finger or two A swipe input (by finger contacts), or a pivot input (eg, two-finger rotation, or one finger contact pivots around another finger contact) is detected (18004). For example, the request is an input as described in connection with FIGS. 13B and 13C or an input as described in connection with FIGS. 13E and 13F.

In response to detecting the first input, the first input is three-dimensional about a first axis (e.g., a first axis horizontally parallel to the plane of the display (e.g., xy plane), e.g., x-axis). Upon determining that it responds to the request to rotate the object, the device may determine the size of the first input (e.g., the vertical axis (e.g., y-axis) of the touch-sensitive surface (e.g., the corresponding xy plane relative to the xy plane of the display)). ), which is determined based on the speed and/or distance of the swipe input) and is virtual by an amount constrained by the limit on movement that limits the rotation of the virtual three-dimensional object by rotation above a threshold amount about the first axis. Rotate a 3D object about a first axis (e.g., a rotation about a first axis is limited to a range of +/-30 degrees about a first axis, and a rotation beyond that range is the magnitude of the first input) Is prohibited regardless of) (18006). For example, as described in connection with FIGS. 13E-13G, the rotation of the virtual object 11002 is constrained by a limit. The first input is to rotate the 3D object around a second axis different from the first axis (e.g., a second axis parallel to the plane of the display (e.g., xy plane), e.g., y-axis). Upon determining that it corresponds to the request, the device swipes along the horizontal axis (e.g., x-axis) of the size of the first input (e.g., the touch-sensitive surface (e.g., the corresponding xy plane relative to the xy plane of the display)). The virtual 3D object is rotated about the second axis by an amount determined based on the speed and/or distance of the input), and in the case of an input having a magnitude above each threshold, the device will rotate the second axis by a rotation above the threshold. Rotate the virtual 3D object for. In some embodiments, for rotation about the second axis, the device imposes a constraint on rotation that is greater than the constraint on rotation about the first axis (e.g., a three-dimensional object is Allowed to rotate). In some embodiments, in the case of rotation about the second axis, the device does not impose constraints on the rotation, allowing the three-dimensional object to freely rotate around the second axis (e.g., movement of one or more contacts). In the case of an input having a sufficiently large size, such as a fast or long swipe input including, the three-dimensional object may rotate more than 360 degrees about the second axis). For example, in response to the input described in connection with FIGS. 13E to 13G, the rotation amount of the virtual object 11002 larger than the rotation amount of the virtual object 11002 centered on the x-axis with reference to FIGS. 13B and 13C It occurs around the y-axis in response to the described input. Depending on whether the input is a request to rotate the object about a first axis or a request to rotate the object about a second axis, determining whether to rotate the object by an amount constrained by a threshold amount or rotate the object by a threshold amount is different. Improves the ability to control types of rotational motions. Providing additional control options without cluttering the user interface with the displayed additional controls improves the operability of the device and makes the user-device interface more efficient.

In some embodiments, in response to detecting the first input, the first input is a first of a contact across the touch-sensitive surface in a first direction (e.g., y-direction, vertical direction on the touch-sensitive surface). According to the determination that the first movement of the contact in the first direction comprises a movement and meets the first criteria for rotating the representation of the virtual object about a first axis-the first criteria are to be met so that the first criteria are satisfied. Includes a requirement that the first input includes a movement above a first threshold amount in a first direction (e.g., the device is about a first axis until the device detects a movement above the first threshold amount in the first direction). Does not initiate rotation of the 3D object) -, the device 3D around a first axis (e.g., x-axis, horizontal axis parallel to the display, or horizontal axis passing through the virtual object) Determine that it corresponds to the request to rotate the object; That the first input comprises a second movement of the contact across the touch-sensitive surface in a second direction (e.g., x-direction, horizontal direction on the touch-sensitive surface) and that the second movement of the contact in the second direction Depending on the determination that the second criteria for rotating the representation of the virtual object about the second axis are met-the second criteria allow the first input to move in the second direction by more than a second threshold amount in order for the second criteria to be satisfied. Include the requirement to include (e.g., the device does not initiate rotation of the three-dimensional object about the second axis until the device detects a movement in excess of the second threshold amount in the second direction) -, the device 1 determines that the input corresponds to a request to rotate the 3D object about a second axis (e.g., a vertical axis parallel to the display, or a vertical axis passing through the virtual object), the first threshold being the second threshold Greater (e.g., the user has to trigger a rotation about the horizontal axis (e.g., for the user) rather than swiping in the horizontal direction to trigger a rotation about the vertical axis (e.g., to rotate an object) To tilt the object forward or backward) it is necessary to swipe in the vertical direction by a larger amount) (18008). Depending on whether the input is a request to rotate the object around a first axis or a request to rotate around a second axis, determining whether to rotate the object by an amount constrained by a threshold amount or by an amount above the threshold amount Improves the ability to control different types of rotational actions in response to an input corresponding to a request to rotate an object. Providing additional control options without cluttering the user interface with the displayed additional controls improves the operability of the device and makes the user-device interface more efficient.

In some embodiments, the rotation of the virtual 3D object about the first axis is a characteristic value (e.g., swipe distance, or swipe speed) of the first input parameter of the first input and the virtual 3 around the first axis. It occurs as a first degree of correspondence between the amount of rotation applied to the dimensional object, and the rotation of the virtual 3D object about the second axis is a characteristic value of the first input parameter of the second input gesture. The second degree of correspondence occurs between the wipe distance, or swipe speed) and the amount of rotation applied to the virtual 3D object centered on the second axis, and the first degree of correspondence is the virtual for the first input parameter than the second degree of correspondence. A smaller rotation of the three-dimensional object is involved (eg, rotation about a first axis has greater friction or jamming than rotation about a second axis) 18010. For example, the first amount of rotation of the virtual object 11002 is a swipe with a swipe distance d ₁ for rotation about the y-axis (as described in connection with FIGS. 13B and 13C ). The second rotation amount of the virtual object 11002 that occurs in response to the input and is smaller than the first rotation amount is a swipe for rotation about the x-axis (as described in connection with FIGS. 13E-13G). Occurs in response to a swipe input with a distance d ₁ . Depending on whether the input is a request to rotate the object around a first axis or a request to rotate around a second axis, rotating the virtual object to a greater or lesser degree of rotation in response to the input rotates the object. Improves the ability to control different types of rotational actions in response to an input corresponding to a request to let. Providing additional control options without cluttering the user interface with the displayed additional controls improves the operability of the device and makes the user-device interface more efficient.

In some embodiments, the device detects the termination of the first input (e.g., the input includes movement of one or more contacts on the touch-sensitive surface, and detecting the termination of the first input from the touch-sensitive surface. Detecting liftoff of one or more contacts) 18012. After detecting the end of the first input (e.g., in response to it), the device is based on the magnitude of the first input prior to detecting the end of the input (e.g., based on the speed of movement of the contact just before liftoff of the contact). Thus) the 3D object is continuously rotated, which, depending on the determination that the 3D object is rotating about the first axis, is proportional to the size of the rotation of the 3D object about the first axis. Slowing the rotation of the object about (eg, slowing the rotation of the three-dimensional object about a first axis based on a first simulated physical parameter, such as a simulated friction having a first coefficient of friction); And slowing the rotation of the object about the second axis by a second amount proportional to the size of the rotation of the 3D object about the second axis according to a determination that the 3D object is rotating about the second axis (e.g. , Slowing the rotation of the three-dimensional object about the second axis based on a second simulated physical parameter, such as a simulated friction having a second coefficient of friction less than the first coefficient of friction), wherein the second quantity is It is different from the first amount (18014). For example, in FIGS. 13C and 13D, the virtual object 11002 continues to rotate after the lift-off of the contact 13002 that caused the rotation of the virtual object 11002 as described in connection with FIGS. 13B and 13C. In some embodiments, the second amount is greater than the first amount. In some embodiments, the second amount is less than the first amount. Depending on whether the input is a request to rotate the object about a first axis or a request to rotate the object about a second axis, after detecting the end of the input, the rotation of the virtual object is slowed by a first amount or a second amount. This provides visual feedback indicating that rotational motions are applied differently to the virtual object for rotation about the first and second axes. Providing improved visual feedback to the user (e.g., prior to placement of the object in a second orientation corresponding to the plane helps the user to provide appropriate inputs and to avoid attempts to provide input to manipulate the virtual object). Thereby) improving the operability of the device and making the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device faster and more efficiently.

In some embodiments, the device detects the termination of the first input (e.g., the input includes movement of one or more contacts on the touch-sensitive surface, and detecting the termination of the first input from the touch-sensitive surface. And detecting the liftoff of one or more contacts) 18016. After detecting the end of the first input (e.g., in response to it), upon determining that the 3D object has been rotated beyond the respective rotation threshold about the first axis, the device will rotate the 3D object about the first axis. Reverse at least a portion of; Upon determining that the three-dimensional object has not been rotated beyond the respective rotation threshold about the first axis, the device withholds (18018) reversing the rotation of the three-dimensional object about the first axis. (E.g., stopping the rotation of the 3D object about the first axis and/or the amount of the 3D object about the first axis in the direction of the motion of the input by a size determined by the size of the input before detecting the end of the input. To keep spinning). For example, as described in connection with FIGS. 13E to 13G, after the virtual object 11002 has rotated beyond the rotation threshold, as illustrated by FIGS. 13G and 13H, the rotation of the virtual object 11002 Is reversed. In some embodiments, the amount of reversal of rotation of a three-dimensional object is determined based on how far the three-dimensional object has rotated beyond each rotation threshold (e.g., the rotation of the three-dimensional object has rotated beyond each rotation threshold. A larger amount with respect to the first axis by 3 if the amount by which the rotation of the three-dimensional object has rotated beyond each rotation threshold is greater compared to a smaller amount that reverses the rotation about the first axis if the amount is smaller. The rotation of the dimensional object is reversed). In some embodiments, the reversal of rotation is driven by a simulated physical parameter, such as an elastic effect pulling with a greater force, and additionally the three-dimensional object is rotated about the first axis beyond each rotation threshold. In some embodiments, the reversal of rotation is made in a direction of rotation that is determined based on the direction of rotation about the first axis rotated beyond each rotation threshold (e.g., a 3D object is rotated so that the top of the object is displayed. If moved inward or backward, the reversal of rotation is to rotate the top of the object forward out of the display; and/or; if the object is rotated so that the top of the 3D object is rotated forward outside the display, the reversal of rotation is Rotating the top back into the display and/or if the three-dimensional object is rotated so that the right side of the object has moved back into the display, the reversal of the rotation is to rotate the right side of the object forward out of the display; If the object is rotated so that the left side of the 3D object is rotated forward outside the display, the reversal of the rotation is to rotate the left side of the object backward into the display). In some embodiments, for example, if the rotation about the second axis is constrained to each range of angles, similar rubberbanding (e.g., conditional reversal of rotation) will result in rotation about the second axis. Is done for In some embodiments, for example, if rotation about the second axis is not constrained to cause the three-dimensional object to rotate 360 degrees by the device, (e.g., the device may set a rotation threshold for rotation about the second axis). Because it is not imposed) rubber banding is not performed for rotation about the second axis. Depending on whether the object has been rotated beyond the rotation threshold, after detecting the end of the input, at least part of the rotation of the 3D object about the first axis is reversed, or a part of the rotation of the 3D object about the first axis. Withholding reversing provides visual feedback indicating a rotation threshold applicable to the rotation of the virtual object. Providing improved visual feedback to the user improves the operability of the device and further enhances the user-device interface (e.g., by helping the user avoid attempts to provide input to rotate the virtual object beyond the rotation threshold). It makes it efficient, which in addition reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, the first input is about a third axis different from the first and second axes (e.g., a third axis perpendicular to the plane of the display (e.g., xy plane), such as the z-axis). Upon determining that it responds to the request to rotate the 3D object, the device withholds rotating the virtual 3D object about the 3rd axis (e.g. rotation about the z-axis is prohibited, The request to rotate the object by the way is ignored by the device) (18020). In some embodiments, the device provides an alarm (eg, a tactile output indicating a failure of an input). Retaining rotation of the virtual object upon determination that the rotation input corresponds to a request to rotate the virtual object about the third axis provides visual feedback indicating that rotation about the third axis is restricted. Providing improved visual feedback to the user improves the operability of the device and improves the user-device interface (e.g., by helping the user avoid attempts to provide input to rotate the virtual object about a third axis). It makes it more efficient, which in addition reduces the device's power usage and improves battery life by enabling users to use the device more quickly and efficiently.

In some embodiments, the device displays a representation of the shadow cast by the virtual 3D object while displaying the representation of the first viewpoint of the virtual 3D object in the first user interface area (eg, staging user interface) ( 18022). The device changes the shape of the representation of the shadow according to the rotation of the virtual 3D object about the first axis and/or the second axis. For example, the shape of the shadow 13006 of the virtual object 11002 changes from FIG. 13B to FIG. 13F as the virtual object 11002 rotates. In some embodiments, the shadow is shifted and changed in shape to indicate the current orientation of the virtual object relative to the invisible reference plane in a staging user interface that supports the predefined lower surface of the virtual object. In some embodiments, the surface of the virtual three-dimensional object appears to reflect light from a simulated light source positioned in a predefined direction within the virtual space represented in the staging user interface. Varying the shape of the shadow according to the rotation of the virtual object provides visual feedback (eg, indicating the virtual plane (eg, stage of the staging view) in which the virtual object is oriented). Providing improved visual feedback to the user improves the operability of the device (e.g., by helping the user determine the appropriate orientation for the swipe input to cause a rotation about the first or second axis). And make the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, while rotating the virtual 3D object within the first user interface area, according to the determination that the virtual 3D object is displayed as a second viewpoint revealing a predefined lower portion of the virtual 3D object, the device is Displaying the expression of the shadow as the expression of the second viewpoint of the virtual 3D object is reserved (18024). For example, the device does not display the shadow of the virtual object when the virtual object is viewed from below (eg, as described in connection with FIGS. 13G-13I). Suspending the display of the shadow of the virtual object upon the determination that the lower part of the virtual object is displayed (e.g., indicating that the object has rotated to a position no longer corresponding to the virtual plane (e.g., the stage of the staging view)) Provide feedback. Providing improved visual feedback to the user improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device faster and more efficiently, thereby reducing the power usage of the device. And improve battery life.

In some embodiments, after rotating the virtual 3D object within the first user interface area (eg, staging view), the device has a second input corresponding to the request to reset the virtual 3D object within the first user interface area. (For example, the second input is a double tap on the first user interface area) (18026). In response to detecting the second input, the device is configured with a predefined original viewpoint (e.g., the first viewpoint, or (e.g., the first viewpoint is within the staging user interface) of the virtual three-dimensional object within the first user interface region). In the case of the displayed time point after user manipulation), the display (e.g., through rotation and resizing of the virtual object) of the default start time distinct from the first time point) is displayed (e.g., in response to a double tap, the device Reset the orientation of the virtual object to a predefined original orientation (eg, upright, with the front face facing the user and the lower face seated on a predefined reference plane) 18028. For example, FIGS. 13I and 13J show that the view point of the virtual object 11102 is changed (as a result of the rotation input described in connection with FIGS. 13B to 13G) of the virtual object 11102 shown in FIG. The same as the viewpoint) illustrates an input to change to the original viewpoint of FIG. In some embodiments, in response to detecting the second input corresponding to the instruction to reset the virtual 3D object, the device further resizes the virtual 3D object to reflect the default display size of the virtual 3D object. . In some embodiments, a double tap input resets both the orientation and size of the virtual object in the staging user interface, while the double tap input resets only the size, not the orientation of the virtual object in the augmented reality user interface. In some embodiments, the device requires that a double tap is for the virtual object to resize the virtual object in the augmented reality user interface, while the device requires double taps detected on the virtual object and double taps detected around the virtual object. Reorients and resizes the virtual object in response to the taps. In an augmented reality view, a single finger swipe drags a virtual object rather than rotating it (eg, unlike in a staging view). Displaying the virtual object's predefined original viewpoint in response to detecting a request to reset the virtual object (e.g., an input provided to adjust the properties of the object returns the object to the predefined original viewpoint). It improves the operability of the device and makes the user-device interface more efficient (by providing the option to reset the object rather than requiring the user to estimate when). Reducing the number of inputs required to perform an operation improves the operability of the device, which in addition reduces the device's power usage and improves battery life by enabling users to use the device more quickly and efficiently. Let it.

In some embodiments, while displaying a virtual 3D object within a first user interface area (e.g., a staging user interface), the device detects a third input corresponding to the request to resize the virtual 3D object (e.g. , The third input is a pinch or de-pinch gesture for the virtual object expressed on the first user interface area, and the third input is the criteria for initiating the resize operation (e.g., with reference to (method 19000) below Has a size that meets the original or augmented criteria), as described in more detail in (18030). In response to detecting the third input, the device adjusts the size of the representation of the virtual 3D object in the first user interface area according to the size of the input (18032). For example, in response to an input comprising a de-pinch gesture (eg, as described in connection with FIGS. 6N and 6O ), the size of the virtual object 11002 is reduced. In some embodiments, the device displays an indicator for indicating the current zoom level of the virtual object when the size of the representation of the virtual three-dimensional object is adjusted. In some embodiments, the device stops displaying the indicator of the zoom level at the end of the third input. Resizing the virtual object according to the size of the input for resizing the object improves the operability of the device (eg, by providing an option to resize the object to a desired amount). Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, while adjusting the size of the representation of the virtual 3D object in the first user interface area (e.g., staging user interface), the device determines that the size of the virtual 3D object is a predefined default value of the virtual 3D object. It is detected that the display size has been reached (18034). In response to detecting that the size of the virtual 3D object has reached the predefined default display size of the virtual 3D object, the device tactiles to indicate that the virtual 3D object is displayed at the predefined default display size. An output (eg, a separate tactile output) is generated (18036). 11O is provided in response to detecting that the size of the virtual object 11002 has reached a previously predefined size of the virtual object 11002 (e.g., as described in connection with FIGS. 11M-11O). Provides an example of a tactile output 11024 to be used. In some embodiments, the device generates the same tactile output when the size of the virtual object is reset to the default display size in response to a double tap input. Generating a tactile output upon determination that the size of the virtual object has reached a predefined default display size (e.g., that no additional input is required to return the simulated size of the virtual object to the predefined size) To provide feedback to the user. Providing improved tactile feedback (e.g., by providing sensory information that makes the user aware that a predefined simulated physical size of a virtual object has been reached without cluttering the user interface with the displayed information). It improves the operability of the device, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, a visual indication of the zoom level (eg, a slider indicating a value corresponding to the current zoom level) is displayed within the first user interface area (eg, a staging user interface). As the size of the representation of the virtual 3D object is adjusted, the visual display of the zoom level is adjusted according to the adjusted size of the representation of the virtual 3D object.

In some embodiments, while displaying a representation of the third viewpoint of the virtual three-dimensional object within the first user interface area (e.g., staging user interface), the device may be configured with one or more cameras (e.g., cameras embedded within the device). A fourth input corresponding to a request to display a virtual 3D object in a second user interface area (eg, augmented reality user interface) including a field of view is detected (18042). In response to detecting the fourth input, the device displays, via a display generating component, a representation of the virtual object over at least a portion of the field of view of the one or more cameras included in the second user interface area (e.g., The field of view is displayed in response to a request to display a virtual object within the second user interface area), and the field of view of the one or more cameras is a view of the physical environment in which the one or more cameras are located (18044). Displaying the representation of the virtual object is to place the virtual three-dimensional object at a predefined angle around a first axis (e.g., an axis horizontally parallel to the plane of the display (e.g., xy plane), such as the x-axis). Rotation (eg, at a default yaw angle, such as zero degrees; or at an angle aligned with (eg, parallel to) the plane detected in the physical environment captured within the field of view of one or more cameras). In some embodiments, the device displays an animation in which the three-dimensional object gradually rotates at a predefined angle with respect to the first axis. Maintaining the current angle of the virtual three-dimensional object with respect to a second axis (eg, an axis parallel in a direction perpendicular to the plane of the display (eg, x-y plane), such as the y-axis). In response to a request to display the virtual object within the field of view of one or more cameras (e.g., without requiring additional input to reposition the virtual object to a predefined orientation with respect to the plane), the virtual object is pre-adjusted about the first axis. Rotating to a defined angle improves the operability of the device. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, while displaying the representation of the fourth viewpoint of the virtual 3D object in the first user interface area (e.g., staging user interface), the device is a 2D user including a 2D representation of the virtual 3D object. A fifth input corresponding to the request to return to the interface is detected (18046). In response to detecting the fifth input, the device creates a virtual three-dimensional object (e.g., a two-dimensional user interface and a virtual three-dimensional Rotate) before displaying the two-dimensional representation of the object; After the virtual 3D object is rotated to show the representation of each viewpoint corresponding to the 2D representation of the virtual 3D object, the 2D representation of the virtual 3D object is displayed (18048). In some embodiments, the device displays an animation in which the 3D object is gradually rotated to show a representation of the viewpoint of the virtual 3D object corresponding to the 2D representation of the virtual 3D object. In some embodiments, the device also resizes the virtual 3D object during or after rotation to match the size of the 2D representation of the virtual 3D object displayed within the 2D user interface. In some embodiments, an animated transition is displayed within the 2D user interface to show a rotated virtual 3D object that moves towards a location of a 2D representation (e.g., a thumbnail image of a virtual object) and settles at that location. do. Rotating the virtual 3D object to a point in time corresponding to the 2D representation of the virtual 3D object in response to an input for returning to displaying the 2D representation of the virtual 3D object (e.g., Provide visual feedback). Providing improved visual feedback to the user (e.g., by helping the user to provide appropriate inputs and to avoid attempts to provide input to rotate the 2D object along an axis where rotation of the 2D object is not available). It improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, prior to displaying the representation of the first viewpoint of the virtual 3D object, the device may display a representation of the view of the virtual 3D object from each viewpoint (e.g., a 2D image corresponding to the virtual 3D object and A user interface including a representation (eg, a thumbnail or an icon) of a virtual 3D object including the same static representation) is displayed (18050). While displaying the representation of the virtual 3D object, the device detects a request to display the virtual 3D object (eg, a tap input or other selection input for the representation of the virtual 3D object) (18052). In response to detecting a request to display the virtual 3D object, the device replaces the display of the representation of the virtual 3D object with the rotated virtual 3D object to match each viewpoint of the representation of the virtual 3D object (18054). ). 11A-11E provide an example of a user interface 5060 that displays a representation of a virtual object 11002. In response to a request to display the virtual object 11002, the display of the user interface 5060, as described in connection with FIG. ) Is replaced by the display. The viewpoint of the virtual object 11002 of FIG. 11E is the same as the viewpoint of the representation of the virtual object 11002 of FIG. 11A. In some embodiments, the representation of the virtual three-dimensional object is scaled up (eg, to a size that matches the size of the virtual three-dimensional object) before it is replaced with the virtual three-dimensional object. In some embodiments, the virtual three-dimensional object is initially displayed at the size of the representation of the virtual three-dimensional object, and then the scale is increased. In some embodiments, during the transition from the representation of the virtual 3D object to the virtual 3D object, the device gradually enlarges the representation of the virtual 3D object and crossfades the representation of the virtual 3D object to the virtual 3D object. (cross fade), then the virtual 3D object is gradually enlarged to create a smooth transition between the representation of the virtual 3D object and the virtual 3D object. In some embodiments, the initial position of the virtual three-dimensional object is selected to correspond to the position of the representation of the virtual three-dimensional object. In some embodiments, the representation of the virtual three-dimensional object is shifted to a location selected to correspond to the location where the virtual three-dimensional object is to be displayed. Replacing the display of the (2D) representation of the virtual 3D object with a virtual 3D object rotated to match the viewpoint of the (2D) representation (e.g., the 3D object is the same as the 2D representation of the virtual 3D object Provides visual feedback indicating that it is an object. Providing improved visual feedback to the user improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device faster and more efficiently, thereby reducing the power usage of the device. And improve battery life.

In some embodiments, prior to displaying the first user interface, the device displays (18056) a two-dimensional user interface that includes a two-dimensional representation of a virtual three-dimensional object. While displaying a two-dimensional user interface that includes a two-dimensional representation of a virtual three-dimensional object, the device displays preview criteria (e.g., preview criteria) at a location on the touch-sensitive surface corresponding to the two-dimensional representation of the virtual three-dimensional object. That the strength of the click input exceeds a first strength threshold (e.g., a light click strength threshold) and/or the preview criteria require the duration of the click input to exceed a first duration threshold). A first portion of the touch input (eg, an increase in the intensity of the contact) is detected (18058). In response to detecting the first portion of the touch input that meets the preview criteria, the device displays a preview of the virtual three-dimensional object that is larger than the two-dimensional representation of the virtual three-dimensional object (e.g., the preview is The object is animated to show different viewpoints) (18060). In some embodiments, the device displays an animation in which the three-dimensional object is gradually magnified (eg, based on the duration or pressure of the input or based on a predetermined speed of the animation). Displaying a preview of a virtual three-dimensional object (e.g., without replacing the display of the currently displayed user interface with a different user interface) can result in (e.g., the user displaying the virtual three-dimensional object and between user interfaces. It improves the operability of the device) by making it possible to return to observing a two-dimensional representation of a virtual three-dimensional object without providing an input for navigating. Reducing the number of inputs required to perform an operation improves the operability of the device, which additionally reduces the device's power usage and increases battery life by enabling users to use the device more quickly and efficiently. Improve.

In some embodiments, while displaying the preview of the virtual three-dimensional object, the device detects (18062) a second portion of the touch input (eg, by the same continuously held contact). In response to detecting the second portion of the touch input, the second portion of the touch input causes menu display criteria (e.g., menu display criteria require the contact to move in a predefined direction (e.g., upward) by a threshold amount). Depending on the determination to satisfy the virtual object, the device may have a plurality of selectable options (e.g., a sharing menu) corresponding to a plurality of actions associated with the virtual object (e.g., various means of sharing the virtual object with another device or user). Sharing options); Depending on the determination that the second portion of the touch input meets staging criteria (e.g., the staging criteria require that the strength of the contact exceed a second threshold strength (e.g., a deep press strength threshold) greater than the first threshold strength). , The device replaces the display of the 2D user interface including the 2D representation of the virtual 3D object with the first user interface including the virtual 3D object (18064). Depending on whether the staging criteria are satisfied, displaying a menu associated with a virtual object or displaying a two-dimensional user interface including a two-dimensional representation of a virtual three-dimensional object is replaced with a first user interface including a virtual three-dimensional object Doing enables the performance of a number of different types of actions in response to an input. Enabling the performance of a number of different types of actions using the first type of input increases the efficiency with which the user can perform these actions, thereby improving the operability of the device, which additionally allows the user to It reduces the device's power usage and improves battery life by making it possible to use the device more quickly and efficiently.

In some embodiments, the first user interface includes a plurality of controls (eg, buttons for switching to a real-world view, for going back, and so on) (18066). Before displaying the first user interface, the device displays a two-dimensional user interface including a two-dimensional representation of a virtual three-dimensional object (18068). In response to detecting a request to display the virtual three-dimensional object in the first user interface, the device displays the virtual three-dimensional object in the first user interface without displaying a set of one or more control units associated with the virtual three-dimensional object. and; After displaying the virtual 3D object in the first user interface, the device displays a set of one or more control units (18070). For example, as described in connection with FIGS. 11A-11E, the display of user interface 5060 including a two-dimensional representation of virtual object 11002 is displayed before staging user interface 6010. In response to a request to display the virtual object 11002 in the staging user interface 6010 (as described in connection with FIG. 11A ), the virtual object 11002 is , 6020) (as shown in Figs. 11B and 11C). In Figures 11D and 11E, the controls 6016, 6018, 6020 of the staging user interface 6010 are faded in to the view within the user interface. In some embodiments, a set of one or more control units is configured to display a virtual 3D object in an augmented reality environment in which the virtual 3D object is disposed at a fixed position with respect to a plane detected within the field of view of one or more cameras of the device. Includes a control unit for. In some embodiments, in response to detecting a request to display a virtual three-dimensional object in the first user interface, the virtual three-dimensional object is not ready to be displayed in the first user interface (e.g., the first user interface. Depending on the determination that the 3D model of the virtual object is not fully loaded at the time when is ready to be displayed (e.g., the loading time of the virtual object is longer than the threshold time (e.g., perceptible and significant to the user)), The device displays a portion of the first user interface (eg, a background window of the first user interface) without displaying a plurality of controls on the first user interface; Upon determining that the virtual 3D object is ready to be displayed in the first user interface (e.g., after a portion of the first user interface is displayed without controls), the device creates the virtual 3D object in the first user interface ( Display (e.g., fade in); The device displays the control units (eg, by fade in) after the virtual 3D object is displayed in the first user interface. In response to detecting a request to display the virtual three-dimensional object within the first user interface, and that the virtual three-dimensional object is ready to be displayed (e.g., when the first user interface is ready to be displayed) According to the determination that the 3D model has been loaded) (e.g., the loading time of the virtual object is shorter than the threshold time (e.g., it is not perceptible to the user and can be ignored)), the device Display the first user interface in a state of at The device displays (eg, without fade-in) a virtual 3D object in the first user interface together with a plurality of control units. In some embodiments, if a staging user interface exists to return to the 2D user interface (eg, in response to a request to "go back"), the virtual 3D object is converted to a 2D representation of the virtual 3D object. The controls are first faded out before becoming. Displaying the controls after displaying the virtual three-dimensional object in the user interface provides visual feedback (eg, indicating that controls that manipulate the virtual object are unavailable for the time required to load the virtual object). Providing improved visual feedback to the user improves the operability of the device (e.g., by helping the user avoid providing input to manipulate the object while manipulating actions are unavailable during loading time for the virtual object). And make the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

It should be understood that the specific order in which the operations in FIGS. 18A-18I are described is merely an example and is not intended to indicate that the described order is the only order in which the operations may be performed. One of skill in the art will recognize various ways of reordering the operations described herein. Additionally, details of other processes described herein in connection with other methods described herein (e.g., methods 800, 900, 1000, 16000, 17000, 19000, 20000) are provided in FIGS. It should be noted that it is also applicable in a manner similar to method 18000 described above with respect to FIG. 18I. For example, contacts, inputs, virtual objects, user interface regions, fields of view, tactile outputs, movements, and/or animations described above with respect to method 18000 may optionally be described herein. Contacts, inputs, virtual objects, user interface regions, field of view described herein in connection with other methods described (e.g., methods 800, 900, 1000, 17000, 18000, 19000, 20000) S, tactile outputs, movements, and/or one or more of the characteristics of animations. For brevity, these details are not repeated here.

19A-19H are flow charts illustrating a method 19000 of increasing the second threshold size of movement required for the second object manipulation behavior, upon determining that the first threshold size of movement is satisfied for the first object manipulation behavior. admit. The method 19000 includes a touch screen serving as a display generating component (e.g., display, projector, head-up display, etc.) and a touch-sensitive surface (e.g., a touch-sensitive surface, or both a display-generating component and a touch-sensitive surface). (E.g., device 300 of FIG. 3, or portable multifunction device 100 of FIG. Some operations in method 19000 are selectively combined and/or the order of some operations is selectively changed.

The device, through the display generating component, performs a first object manipulation behavior (e.g., of a user interface object centered on each axis) in response to inputs that meet the first gesture recognition criteria (e.g., rotation criteria). Rotation) and a second object manipulation behavior (e.g., a translation of a user interface object or a user interface object) performed in response to inputs that meet the second gesture recognition criteria (e.g., one of translation criteria and scaling criteria) A first user interface area (eg, a user interface area including a representation of a virtual object) including a user interface object associated with a plurality of object manipulation behaviors including one of the scaling adjustments of (19002) is displayed. For example, the displayed virtual object 11002 may be rotated about each axis (e.g., as described in connection with FIGS. 14B-14E), as described in connection with FIGS. 14K-14M. Such), and manipulation behaviors including scaling (eg, as described in connection with FIGS. 14G-14I ).

While displaying the first user interface area, the device detects a first portion of the input to the user interface object, including detecting movement of one or more contacts across the touch-sensitive surface (e.g., the device Detecting one or more contacts at locations on the touch-sensitive surface corresponding to the display location of the user interface object), while the one or more contacts are detected on the touch-sensitive surface, the device performs first gesture recognition criteria and a second gesture. Evaluate the movement of one or more contacts to both recognition criteria (19004).

In response to detecting the first portion of the input, the device updates the appearance of the user interface object based on the first portion of the input, which is the first gesture recognition criterion before the first portion of the input meets the second gesture recognition criteria. Change the appearance of the user interface object according to the first object manipulation behavior based on the first portion of the input (e.g., based on the orientation and/or size of the first portion of the input) upon determining that the recognition criteria are met. (Eg, rotating a user interface object); And by increasing the threshold for the second gesture recognition criteria (e.g., without changing the appearance of the user interface object according to the second object manipulation behavior) (e.g., a movement parameter (e.g., movement distance) in the second gesture recognition criteria) , By increasing the threshold required for speed, etc.) (19006). For example, in FIG. 14E, the virtual object 1102 has been rotated according to a determination that the rotation criteria have been satisfied (before the scaling criteria are satisfied), and the threshold ST for the scaling criteria is increased to ST'. In some embodiments, before the criteria for recognizing a gesture for rotating an object are met, for recognizing a gesture for translation or scaling (assuming that the criteria for translation or scaling have not been previously met) It is relatively easy to initiate a translation or scaling operation on an object by meeting the criteria. Once the criteria for recognizing a gesture to rotate an object are met, it becomes more difficult to initiate a translation or scaling operation for the object (e.g., the criteria for translation and scaling are increased thresholds for the movement parameter). Updated), object manipulation is biased towards manipulation behavior corresponding to gestures already recognized and used to manipulate these objects. Upon determining that the input meets the second gesture recognition criteria prior to meeting the first gesture recognition criteria, the device is based on the first portion of the input (e.g., based on the direction and/or size of the first portion of the input). ) Changing the appearance of the user interface object according to the second object manipulation behavior (eg, translating the user interface object or resizing the user interface object); By increasing the threshold for the first gesture recognition criteria (e.g., without changing the appearance of the user interface object according to the first object manipulation behavior) (e.g., the movement parameter (e.g., movement distance, Update the first gesture recognition criteria) by increasing the threshold required for speed, etc. For example, in FIG. 14I, the size of the virtual object 1102 is increased according to the determination that the scaling criteria are satisfied (before the rotation criteria are satisfied), and the threshold RT for the rotation criteria is increased to RT′. In some embodiments, before the criteria for recognizing a gesture for translating or scaling an object are met, a gesture for rotation (assuming that the criteria for recognizing a gesture for rotating an object have not been previously met) It is relatively easy to initiate a rotational operation on an object by meeting the criteria for recognizing a. Once the criteria for recognizing a gesture to translate or scale an object are met, it becomes more difficult to initiate a rotational motion on the object (e.g., the criteria for rotating the object are updated with an increased threshold for the movement parameter). ), object manipulation behaviors are biased towards manipulation behaviors corresponding to gestures already recognized and used to manipulate these objects. In some embodiments, the appearance of the user interface object changes dynamically and continuously according to the values of each movement parameter of the input (e.g., showing different sizes, positions, viewpoints, reflections, shadows, etc.) ). In some embodiments, the device is dependent on a movement parameter (e.g., each movement parameter for each type of operating behavior) and an appearance of the user interface object (e.g., each aspect of the appearance for each type of operating behavior). It follows a preset correspondence between changes made to (eg, each correspondence to each type of operating behavior). Increasing the first threshold for the input movement required for the first object manipulation when the input movement increases beyond the second threshold for the second object manipulation (e.g., to provide an input for performing the first object manipulation Improve the operability of the device) by helping the user avoid accidentally performing the second object manipulation during the attempt. Improving the user's ability to control the manipulation of different types of objects improves the operability of the device and makes the user-device interface more efficient.

In some embodiments, after updating the appearance of the user interface object based on the first portion of the input, the device (e.g., the same continuously held contacts within the first portion of the input, or within the first portion of the input). Detect a second portion of the input (19008) by different contacts that are detected after the end of the contacts (eg, liftoff). In some embodiments, the second portion of the input is detected based on successively detected inputs that are for the user interface object. In response to detecting the second portion of the input, the device updates the appearance of the user interface object based on the second portion of the input, which means that the first portion of the input met the first gesture recognition criteria and the second portion of the input. Depending on the determination that the portion does not meet the updated second gesture recognition criteria, (e.g., regardless of whether the second portion of the input meets the first gesture recognition criteria or the original second gesture recognition criteria) (e.g. , Based on the second portion of the input without changing the appearance of the user interface object according to the second object manipulation behavior (e.g., even if the second portion of the input satisfies the original second gesture recognition criteria before being updated) Changing the appearance of the user interface object according to the first object manipulation behavior) based on the direction and/or size of the second portion of the input; And according to the determination that the first portion of the input meets the second gesture recognition criteria and the second portion of the input does not meet the updated first gesture recognition criteria, (e.g., the second portion of the input meets the second gesture recognition criteria. User interface according to the first object manipulation behavior (e.g., even if the second part of the input satisfies the original first gesture recognition before being updated) regardless of whether it meets the first gesture recognition criteria Including changing the appearance of a user interface object according to a second object manipulation behavior based on a second portion of the input (e.g., based on the orientation and/or size of the second portion of the input) without changing the appearance of the object Do (19010).

In some embodiments, while the appearance of the user interface object is changed according to the first object manipulation behavior based on the second portion of the input after the first portion of the input meets the first gesture recognition criteria, the second portion of the input The portion is the second gesture recognition before the second gesture recognition criteria are updated (e.g., with the original threshold(s) for the movement parameter(s) of the input in the second gesture recognition criteria before the threshold(s) is increased) Include an input that meets the criteria (eg, the second portion of the input does not contain an input that meets the updated second gesture recognition criteria) (19012).

In some embodiments, while the appearance of the user interface object is changed according to the second object manipulation behavior based on the second part of the input after the first part of the input meets the second gesture recognition criteria, the second part of the input The portion is the first gesture recognition before the first gesture recognition criteria are updated (e.g., with the original threshold(s) for the movement parameter(s) of the input in the first gesture recognition criteria before the threshold(s) is increased) Include an input that meets the criteria (eg, the second portion of the input does not contain an input that meets the updated first gesture recognition criteria) (19014).

In some embodiments, while the appearance of the user interface object is changed according to the first object manipulation behavior based on the second portion of the input after the first portion of the input meets the first gesture recognition criteria, the second portion of the input The portion does not include an input that meets the first gesture recognition criteria (e.g., with the original threshold(s) for the movement parameter(s) of the input within the first gesture recognition criteria) (19016). For example, after the first gesture recognition criteria are satisfied once, the input no longer needs to continue to meet the first gesture recognition criteria to cause a first object manipulation behavior.

In some embodiments, while the appearance of the user interface object is changed according to the second object manipulation behavior based on the second part of the input after the first part of the input meets the second gesture recognition criteria, the second part of the input The portion does not include an input that meets the second gesture recognition criteria (e.g., with the original threshold(s) for the movement parameter(s) of the input in the second gesture recognition criteria) (19018). For example, after the second gesture recognition criteria are satisfied once, the input no longer needs to continue to meet the second gesture recognition criteria to cause a second object manipulation behavior. Performing the first object manipulation behavior when the second portion of the input includes an increasing movement above an increased threshold (e.g., by meeting increased criteria without requiring the user to provide a new input). After performing the first object manipulation, the operability of the device is improved by providing the user with the ability to intentionally perform the second object manipulation. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, updating the appearance of the user interface object based on the second portion of the input comprises: a first gesture recognition in which the first portion of the input meets second gesture recognition criteria and the second portion of the input is updated. According to the determination that the criteria are satisfied, the appearance of the user interface object is changed according to the first object manipulation behavior based on the second part of the input, and the user interface object according to the second object manipulation behavior based on the second part of the input. Changing the appearance of; And according to a determination that the first portion of the input meets the first gesture recognition criteria and the second portion of the input meets the updated second gesture recognition criteria, according to the first object manipulation behavior based on the second portion of the input. And changing the appearance of the user interface object, and changing the appearance of the user interface object according to the second object manipulation behavior based on the second portion of the input (19020). For example, after the first gesture recognition criteria are met first and then the input meets the updated second gesture recognition criteria, the input can now cause both first and second object manipulation behaviors. For example, after the second gesture recognition criteria are met first and then the input meets the updated first gesture recognition criteria, the input can now cause both first and second object manipulation behaviors. Updating the object according to the first object manipulation behavior and the second object manipulation behavior in response to a portion of the input detected after the second gesture recognition criteria and the updated first gesture recognition criteria are satisfied (e.g., the user Improves operability of the device) by providing the user with the ability to freely manipulate the object using the first object manipulation and the second object manipulation after satisfying the increased threshold without requiring to provide an input. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, after updating the appearance of the user interface object based on the second portion of the input (e.g., after both the first gesture recognition criteria and the updated second gesture recognition criteria are met, or After both the 2 gesture recognition criteria and the updated first gesture recognition criteria have been met), the device may (e.g., the same continuously maintained contacts within the first and second portion of the input, or the first portion of the input). And (19022) a third portion of the input) by different contacts detected after termination (eg, liftoff) of the contacts in the second portion. In response to detecting the third portion of the input, the device updates the appearance of the user interface object based on the third portion of the input, which is based on the third portion of the input and according to the first object manipulation behavior. Changing the appearance of; And changing the appearance of the user interface object according to the second object manipulation behavior based on the third portion of the input (19024). For example, after both the first gesture recognition criteria and the updated second gesture recognition criteria are satisfied, or after both the second gesture recognition criteria and the updated first gesture recognition criteria are satisfied, the input May subsequently cause both the first and second object manipulation behaviors regardless of the thresholds in the original or updated first and second gesture recognition criteria. Updating the object according to the first object manipulation behavior and the second object manipulation behavior in response to a portion of the input detected after the second gesture recognition criteria and the updated first gesture recognition criteria are satisfied (e.g., the user The ability to freely manipulate an object using first object manipulation and second object manipulation after demonstrating the intention to perform the first type of object manipulation by satisfying an increased threshold without requiring to provide input. To improve the operability of the device). Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, the third portion of the input does not include an input that meets the first gesture recognition criteria or an input that meets the second gesture recognition criteria (19026). For example, after both the first gesture recognition criteria and the updated second gesture recognition criteria are satisfied, or after both the second gesture recognition criteria and the updated first gesture recognition criteria are satisfied, the input May subsequently cause both the first and second object manipulation behaviors regardless of the thresholds in the original or updated first and second gesture recognition criteria. Updating the object according to the first object manipulation behavior and the second object manipulation behavior in response to a portion of the input detected after the second gesture recognition criteria and the updated first gesture recognition criteria are satisfied (e.g., the user Improves the operability of the device) by providing the user with the ability to freely manipulate the object using the first object manipulation and the second object manipulation after satisfying the reinforced criteria without requiring to provide input. . Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, the plurality of object manipulation behaviors are performed in response to inputs meeting third gesture recognition criteria (eg, scaling criteria) (eg, about each axis). Rotation of the user interface object) (19028). Updating the appearance of the user interface object based on the first portion of the input indicates that the first portion of the input satisfies the first gesture recognition criteria prior to meeting the second gesture recognition criteria or meeting the third gesture recognition criteria. Depending on the decision, changing the appearance of the user interface object according to the first object manipulation behavior based on the first portion of the input (e.g., based on the direction and/or size of the first portion of the input) (e.g. Rotating the interface object); And by increasing the threshold for the second gesture recognition criteria (e.g., without changing the appearance of the user interface object according to the second object manipulation behavior) (e.g., a movement parameter (e.g., movement distance) in the second gesture recognition criteria) , By increasing the threshold required for speed, etc.) (19030). For example, before the criteria for recognizing a gesture for rotating an object are met, criteria for recognizing a gesture for translation or scaling (assuming that the criteria for translation or scaling have not been previously met) By fulfilling it is relatively easy to initiate a translation or scaling operation on an object. Once the criteria for recognizing a gesture to rotate an object are met, it becomes more difficult to initiate a translation or scaling operation for the object (e.g., the criteria for translation and scaling are increased thresholds for the movement parameter). Updated), object manipulation is biased towards manipulation behavior corresponding to gestures already recognized and used to manipulate these objects. The device updates the third gesture recognition criteria by increasing the threshold for the third gesture recognition criteria (e.g., by increasing the threshold required for the movement parameter (eg, travel distance, speed, etc.) in the third gesture recognition criteria). . For example, before the criteria for recognizing a gesture for rotating an object are met, criteria for recognizing a gesture for translation or scaling (assuming that the criteria for translation or scaling have not been previously met) By fulfilling it is relatively easy to initiate a translation or scaling operation on an object. Once the criteria for recognizing a gesture to rotate an object are met, it becomes more difficult to initiate a translation or scaling operation for the object (e.g., the criteria for translation and scaling are increased thresholds for the movement parameter). Updated), object manipulation is biased towards manipulation behavior corresponding to gestures already recognized and used to manipulate these objects. Upon determining that the input meets the first gesture recognition criteria or meets the second gesture recognition criteria prior to meeting the third gesture recognition criteria, the device is based on the first portion of the input (e.g., the first portion of the input). Changing the appearance of the user interface object (eg, translating the user interface object or resizing the user interface object) according to the second object manipulation behavior (based on the direction and/or size of); By increasing the threshold for the first gesture recognition criteria (e.g., without changing the appearance of the user interface object according to the first object manipulation behavior) (e.g., the movement parameter (e.g., movement distance, Update the first gesture recognition criteria) by increasing the threshold required for speed, etc. For example, before the criteria for recognizing a gesture to translate or scale an object are met, (assuming that the criteria for recognizing a gesture to rotate an object have not been met before), recognize a gesture for rotation. It is relatively easy to initiate a rotational operation on an object by meeting the criteria for: Once the criteria for recognizing a gesture to translate or scale an object are met, it becomes more difficult to initiate a rotational motion on the object (e.g., the criteria for rotating the object are updated with an increased threshold for the movement parameter). ), object manipulation behaviors are biased towards manipulation behaviors corresponding to gestures already recognized and used to manipulate these objects. In some embodiments, the appearance of the user interface object changes dynamically and continuously according to the values of each movement parameter of the input (e.g., showing different sizes, positions, viewpoints, reflections, shadows, etc.) ). In some embodiments, the device is dependent on a movement parameter (e.g., each movement parameter for each type of operating behavior) and an appearance of the user interface object (e.g., each aspect of the appearance for each type of operating behavior). It follows a preset correspondence between changes made to (eg, each correspondence to each type of operating behavior). The device updates the third gesture recognition criteria by increasing the threshold for the third gesture recognition criteria (e.g., by increasing the threshold required for the movement parameter (eg, travel distance, speed, etc.) in the third gesture recognition criteria). . For example, before the criteria for recognizing a gesture for rotating an object are met, criteria for recognizing a gesture for translation or scaling (assuming that the criteria for translation or scaling have not been previously met) By fulfilling it is relatively easy to initiate a translation or scaling operation on an object. Once the criteria for recognizing a gesture to rotate an object are met, it becomes more difficult to initiate a translation or scaling operation on the object (e.g., the criteria for translation and scaling are increased thresholds for the movement parameter). Updated), object manipulation is biased towards manipulation behavior corresponding to gestures already recognized and used to manipulate these objects. Upon determination that the input satisfies the third gesture recognition criteria prior to meeting the first gesture recognition criteria or meeting the second gesture recognition criteria, the device is based on the first portion of the input (e.g., the first portion of the input). Changing the appearance of the user interface object (eg, resizing the user interface object) according to the third object manipulation behavior (based on the direction and/or size of); (E.g., without changing the appearance of the user interface object according to the first object manipulation behavior and the second object manipulation behavior) by increasing the threshold for the first gesture recognition criteria (e.g., in the first gesture recognition criteria) The first gesture recognition criteria are updated by increasing the threshold required for the movement parameter (eg, travel distance, speed, etc.). For example, before the criteria for recognizing a gesture to translate or scale an object are met, (assuming that the criteria for recognizing a gesture to rotate an object have not been met before), recognize a gesture for rotation. It is relatively easy to initiate a rotational operation on an object by meeting the criteria for: Once the criteria for recognizing a gesture to translate or scale an object are met, it becomes more difficult to initiate a rotational motion on the object (e.g., the criteria for rotating the object are updated with an increased threshold for the movement parameter). ), object manipulation behaviors are biased towards manipulation behaviors corresponding to gestures already recognized and used to manipulate these objects. In some embodiments, the appearance of the user interface object changes dynamically and continuously according to the values of each movement parameter of the input (e.g., showing different sizes, positions, viewpoints, reflections, shadows, etc.) ). In some embodiments, the device is dependent on a movement parameter (e.g., each movement parameter for each type of operating behavior) and an appearance of the user interface object (e.g., each aspect of the appearance for each type of operating behavior). It follows a preset correspondence between changes made to (eg, each correspondence to each type of operating behavior). The device updates the second gesture recognition criteria by increasing the threshold for the second gesture recognition criteria (e.g., by increasing the threshold required for the movement parameter (eg, travel distance, speed, etc.) in the second gesture recognition criteria). . For example, before the criteria for recognizing a gesture for rotating an object are met, criteria for recognizing a gesture for translation or scaling (assuming that the criteria for translation or scaling have not been previously met) By fulfilling it is relatively easy to initiate a translation or scaling operation on an object. Once the criteria for recognizing a gesture to rotate an object are met, it becomes more difficult to initiate a translation or scaling operation for the object (e.g., the criteria for translation and scaling are increased thresholds for the movement parameter). Updated), object manipulation is biased towards manipulation behavior corresponding to gestures already recognized and used to manipulate these objects. Updating the object according to the third object manipulation behavior in response to a portion of the input detected only when the corresponding third gesture recognition criteria are satisfied (e.g., input for performing a first object manipulation or a second object manipulation Improve the operability of the device) by helping the user avoid accidentally performing third object manipulation while attempting to provide. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, the plurality of object manipulation behaviors includes a third object manipulation behavior performed in response to inputs meeting the third gesture recognition criteria, and the first portion of the input is the first gesture recognition criteria or the first 2 The third gesture recognition criteria have not been satisfied before the gesture recognition criteria are satisfied, and the device determines the third gesture recognition criteria after the first portion of the input meets the first gesture recognition criteria or the second gesture recognition criteria. The third gesture recognition criteria were updated by increasing the threshold, and the second portion of the input did not meet the updated third gesture recognition criteria before meeting the updated first gesture recognition criteria or the updated second gesture recognition criteria. (E.g., the device updated the third gesture recognition criteria by increasing the threshold for the third gesture recognition criteria after the first portion of the input meets one of the first or second gesture recognition criteria) (19032) . In response to detecting the third portion of the input, the third portion of the input (eg, regardless of whether the third portion of the input satisfies the (eg, updated or original) first or second gesture recognition criteria) Upon determination that the portion meets the updated third gesture recognition criteria, the device (e.g., even if the third portion of the input does not meet the original first and second gesture recognition criteria) and Based on the third part of the input (e.g., based on the direction and/or size of the third part of the input) to the third object manipulation behavior while changing the appearance of the user interface object according to the second object manipulation behaviors. Accordingly, the appearance of the user interface object is changed (19034). Upon determining that the third portion of the input does not meet the updated third gesture recognition criteria, the device (e.g., (e.g., the third portion of the input does not meet the original first and second gesture recognition criteria). Although not) changing the appearance of the user interface object according to the third object manipulation behavior based on the third part of the input) while changing the appearance of the user interface object according to the first and second object manipulation behaviors is reserved. In response to a portion of the input detected after the second gesture recognition criteria, the updated first gesture recognition criteria, and the updated third gesture recognition criteria are satisfied, the first object manipulation behavior, the second object manipulation behavior, and the second 3 Updating the object according to the object manipulation behavior (e.g., not requiring the user to provide a new input, but after establishing the intent to perform all three object manipulation types by satisfying increased thresholds. , By providing the user with the ability to freely manipulate objects using second and third object manipulation types) to improve the operability of the device. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, the third portion of the input met the updated third gesture recognition criteria (19036). After updating the appearance of the user interface object based on the third part of the input (e.g., after both the first gesture recognition criteria and the updated second and third gesture recognition criteria are satisfied, or the second gesture recognition) After both the criteria and the updated first and third gesture recognition criteria are met), the device may (e.g., maintain the same continuously held contacts within the first, second and third portion of the input, or Detect (19038) a fourth portion of the input) by different contacts detected after termination (eg, lift-off) of the contacts in the first portion, the second portion, and the third portion. In response to detecting the fourth portion of the input, the device updates the appearance of the user interface object based on the fourth portion of the input, which is based on the fourth portion of the input and according to the first object manipulation behavior. Changing the appearance of; Changing the appearance of the user interface object according to the second object manipulation behavior based on the fourth portion of the input; And changing the appearance of the user interface object according to the third object manipulation behavior based on the fourth portion of the input (19040). For example, after the first gesture recognition criteria and the updated second and third gesture recognition criteria are satisfied, or after the second gesture recognition criteria and the updated first and third gesture recognition criteria are satisfied, the input May subsequently cause all three types of manipulation behaviors regardless of the thresholds in the original or updated first, second, and third gesture recognition criteria.

In some embodiments, the fourth portion of the input does not include an input that meets the first gesture recognition criteria, an input that meets the second gesture recognition criteria, or an input that meets the third gesture recognition criteria (19042). For example, after the first gesture recognition criteria and the updated second and third gesture recognition criteria are satisfied, or after the second gesture recognition criteria and the updated first and third gesture recognition criteria are satisfied, the input May subsequently cause all three types of manipulation behaviors regardless of the thresholds in the original or updated first, second, and third gesture recognition criteria. Requiring multiple simultaneously detected contacts for a gesture (e.g., by helping the user avoid accidentally performing object manipulation while providing an input with fewer than the required number of simultaneously detected contacts) Improves. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, both the first gesture recognition criteria and the second gesture recognition criteria (and the third gesture recognition criteria) are satisfied with the first number of simultaneously detected contacts (e.g., two contacts). ) Is requested (19044). In some embodiments, a single finger gesture may also be used for translation, and the single finger translation threshold is lower than the two finger translation threshold. In some embodiments, the original movement threshold and the updated movement threshold set for the two finger translation gesture are movement by centroid of 40 points and 70 points of contacts, respectively. In some embodiments, the original movement threshold and the updated movement threshold set for the two finger rotation gesture are rotational movement by contact of 12 degrees and 18 degrees, respectively. In some embodiments, the original movement threshold and the updated movement threshold set for the two finger scaling gesture are 50 points (distance between contacts) and 90 points, respectively. In some embodiments, the threshold set for a single finger drag gesture is 30 points.

In some embodiments, the first object manipulation behavior changes the zoom level or displayed size of the user interface object (e.g., a pinch gesture (e.g., movement of contacts towards each other) is (e.g., original or After being recognized based on the updated) first gesture recognition criteria, the object is resized by a pinch gesture), and the second object manipulation behavior changes the rotation angle of the user interface object (e.g., twist/ After a pivot gesture (e.g., movement around a common trajectory of contacts) is recognized by the second gesture recognition criteria (e.g., original or updated), the user about an outer or inner axis by a twist/pivot gesture Change the observation point of the interface object) (19046). For example, the first object manipulation behavior changes the displayed size of the virtual object 11002 as described in connection with FIGS. 14G to 14I, and the second object manipulation behavior is described in relation to FIGS. 14B to 14E. As described above, the rotation angle of the virtual object 11002 is changed. In some embodiments, the second object manipulation behavior changes the zoom level or displayed size of the user interface object (e.g., a pinch gesture (e.g., movement of contacts towards each other) is (e.g., original or After being recognized based on the updated) second gesture recognition criteria, the object is resized by a pinch gesture), and the first object manipulation behavior changes the rotation angle of the user interface object (e.g., twist/ After a pivot gesture (e.g., movement around a common trajectory of contacts) is recognized by the first gesture recognition criteria (e.g., original or updated), the user about an outer or inner axis by a twist/pivot gesture Change the viewpoint of the interface object).

In some embodiments, the first object manipulation behavior changes the zoom level or displayed size of the user interface object (e.g., a pinch gesture (e.g., movement of contacts towards each other) is (e.g., original or After being recognized based on the updated) first gesture recognition criteria, the object is resized by a pinch gesture), and the second object manipulation behavior changes the position of the user interface object in the first user interface area (e.g. For example, after a one-finger or two-finger drag gesture (e.g., movement of the contacts in each direction) is recognized by the (e.g., original or updated) second gesture recognition criteria, the user by the drag gesture Drag the interface object) (19048). For example, the first object manipulation behavior changes the displayed size of the virtual object 11002 as described in connection with FIGS. 14G to 14I, and the second object manipulation behavior is described in relation to FIGS. 14B to 14E. As described above, the location of the virtual object 11002 in the user interface is changed. In some embodiments, the second object manipulation behavior changes the zoom level or displayed size of the user interface object (e.g., a pinch gesture (e.g., movement of contacts towards each other) is (e.g., original or After being recognized based on the updated) second gesture recognition criteria, the object is resized by a pinch gesture), and the first object manipulation behavior changes the position of the user interface object in the first user interface area (e.g. For example, after a one-finger or two-finger drag gesture (e.g., movement of the contacts in each direction) is recognized by the (e.g., original or updated) first gesture recognition criteria, the user by the drag gesture Drag the interface object).

In some embodiments, the first object manipulation behavior changes the position of the user interface object within the first user interface area (e.g., a one-finger or two-finger drag gesture (e.g., in each direction of the contacts). Movement) is recognized by the first gesture recognition criteria (e.g., original or updated), then dragging the user interface object by a drag gesture), the second object manipulation behavior changes the rotation angle of the user interface object (E.g., a twist/pivot gesture (e.g., a movement around a common trajectory of contacts) is recognized by the (e.g., original or updated) second gesture recognition criteria, then a twist/pivot gesture By changing the observation point of the user interface object around the outer or inner axis) (19050). For example, the first object manipulation behavior changes the position of the virtual object 11002 in the user interface as described in connection with FIGS. 14B to 14E, and the second object manipulation behavior is related to FIGS. 14B to 14E. Thus, the rotation angle of the virtual object 11002 is changed as described. In some embodiments, the second object manipulation behavior changes the position of the user interface object within the first user interface area (e.g., a one-finger or two-finger drag gesture (e.g., in each direction of the contacts)). Movement) is recognized by the second gesture recognition criteria (e.g., original or updated), then dragging the user interface object by a drag gesture), the first object manipulation behavior changes the rotation angle of the user interface object (E.g., a twist/pivot gesture (e.g., a movement around a common trajectory of contacts) is recognized by the (e.g., original or updated) first gesture recognition criteria, then a twist/pivot gesture Changes the viewing point of the user interface object around an external or internal axis).

In some embodiments, the first portion of the input and the second portion of the input are provided by a plurality of continuously held contacts (19052). The device uses first gesture recognition criteria and second gesture recognition criteria (e.g., original thresholds) to initiate additional first and second object manipulation behaviors after detecting the liftoff of the plurality of continuously held contacts. To) re-established (19054). For example, after lift-off of contacts, the device re-establishes gesture recognition thresholds for rotation, translation, and scaling for the newly detected touch input. Re-establishing the threshold for the input movement after the input is terminated by the liftoff of the contacts (e.g., by resetting the increased movement thresholds each time a new input is provided, reduces the degree of input required to perform object manipulation. Thereby improving the operability of the device. Reducing the amount of input required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby enabling the device to It reduces the power usage of the battery and improves battery life.

In some embodiments, the first gesture recognition criteria correspond to rotation about a first axis, and the second gesture recognition criteria correspond to rotation about a second axis orthogonal to the first axis (19056). In some embodiments, instead of updating the thresholds for different types of gestures, the update also includes different subtypes of manipulation behavior within the type of manipulation behavior corresponding to the recognized gesture type (e.g., twist/pivot gesture). For example, it applies to thresholds set for rotation about a first axis versus rotation about a different axis. For example, once a rotation about a first axis is recognized and performed, the threshold set for rotation about a different axis is updated (e.g., increased) and by subsequent inputs to trigger rotation about a different axis. It must be overcome. Increasing the threshold for the input motion required to rotate the object about the first axis when the input motion increases above the threshold for the input motion required to rotate the object about the second axis (e.g., the first It improves the operability of the device) by helping the user avoid accidentally rotating the object about the second axis while attempting to rotate the object about the axis. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby allowing the device to It reduces the power usage of the battery and improves battery life.

It should be understood that the specific order in which the operations in FIGS. 19A-19H are described is merely an example and is not intended to indicate that the described order is the only order in which the operations may be performed. One of skill in the art will recognize various ways of reordering the operations described herein. Additionally, details of other processes described herein in connection with other methods described herein (e.g., methods 800, 900, 1000, 16000, 17000, 18000, 20000) are provided in FIGS. It should be noted that it is also applicable in a manner similar to method 19000 described above with respect to FIG. 19H. For example, contacts, inputs, virtual objects, user interface regions, fields of view, tactile outputs, movements, and/or animations described above with respect to method 19000 may optionally be described herein. Contacts, inputs, virtual objects, user interface regions, field of view described herein in connection with other methods described (e.g., methods 800, 900, 1000, 16000, 17000, 18000, 20000) S, tactile outputs, movements, and/or one or more of the characteristics of animations. For brevity, these details are not repeated here.

20A-20F are flow diagrams illustrating a method 20000 of generating an audio alert upon determining that movement of the device causes the virtual object to move outside of the displayed field of view of one or more device cameras. The method 20000 includes a display generating component (e.g., a display, a projector, a head-up display, etc.), one or more input devices (e.g., a touch-sensitive surface, or a touch serving as both a display-generating component and a touch-sensitive surface). Screen display), one or more audio output generators, and an electronic device with one or more cameras (eg, device 300 of FIG. 3, or portable multifunction device 100 of FIG. 1A ). Some operations in method 20000 are selectively combined and/or the order of some operations is selectively changed.

The device responds to a request (e.g., in response to a request to place a virtual object within an augmented reality view of the physical environment around the device including the camera (e.g., responsive to a tap on a "real world" button displayed with a staging view of the virtual object). A)), via the display generating component, a first user interface area that contains a representation of the field of view of one or more cameras (e.g., the first user interface area displays an augmented reality view of the physical environment around the device including the camera). User interface), wherein displaying includes maintaining a first spatial relationship between the representation of the virtual object and the plane detected in the physical environment captured within the field of view of one or more cameras (e.g. , The virtual object is displayed on the display in an orientation and position such that a fixed angle between the representation of the virtual object and the plane is maintained (e.g., the virtual object remains in a fixed position on the plane or appears to roll along the viewing plane. ))(20002). For example, as shown in FIG. 15V, a virtual object 11002 is displayed within a user interface area that includes a field of view 6036 of one or more cameras.

The device detects (20004) movement of the device (eg, lateral movement and/or rotation of a device including one or more cameras) that adjusts the field of view of one or more cameras. For example, as described in connection with FIGS. 15V and 15W, movement of device 100 adjusts the field of view of one or more cameras.

In response to detecting movement of the device that adjusts the field of view of the one or more cameras, the device causes a first spatial relationship between the virtual object and the plane detected within the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted. , Orientation and/or position) to adjust the display of the representation of the virtual object within the first user interface area (e.g., between the representation of the virtual object and the plane detected in the physical environment captured within the field of view of one or more cameras) Because the spatial relationship remains fixed during movement of the device relative to the physical environment), the movement of the device causes a virtual object in excess of a threshold (e.g., 100%, 50%, or 20%) to be displayed in the field of view of one or more cameras. Depending on the decision to cause the part to move out of the way, the device generates, via one or more audio output generators, a first audio alert (e.g., a voice alert indicating that a virtual object above a threshold is no longer displayed in the camera view). Do (20006). For example, as described in connection with FIG. 15W, in response to movement of the device 100 that causes the virtual object 11002 to move out of the displayed portion of the field of view 6036 of one or more cameras, an audio alert (15118) is created. Generating the audio output in response to a determination that movement of the device causes the virtual object to move out of the displayed augmented reality view provides feedback indicating the extent to which movement of the device has affected the display of the virtual object relative to the augmented reality view. Provide to the user. Providing improved feedback to the user (e.g., without distracting the display with additional displayed information and without requiring the user to see the display, allows the user to know whether the virtual object has moved out of the display). Improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and reduces battery power consumption by allowing users to use the device more quickly and efficiently. Improves lifespan.

In some embodiments, outputting the first audio alert includes generating an audio output indicating an amount of a virtual object that remains visible on the displayed portion of the field of view of the one or more cameras (e.g., The amount of object is measured for the total size of the virtual object from the current observation point (e.g., 20%, 25%, 50%, etc.)) (e.g., the audio output is "object x is 20% visible (object x is 20%). It is visible)") (20008). For example, as described in connection with FIGS. 15X and 15Y, the movement of the device 100 causes the virtual object 11002 to move partially out of the displayed portion of the field of view 6036 of one or more cameras. In response, an audio alert 15126 is generated that includes a notification 15128 indicating "chair is 90 percent visible, occupying 20 percent of the screen". Generating an audio output indicating the amount of a virtual object visible within the displayed augmented reality view provides feedback to the user (eg, the degree to which movement of the device has changed the degree to which the virtual object is visible). Improved feedback (e.g., by providing information that allows the user to perceive whether a virtual object has moved away from the display without disturbing the display with additional displayed information and without requiring the user to see the display) Providing the user with a device improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the device's power usage and reduces battery life by enabling users to use the device more quickly and efficiently. Improves.

In some embodiments, outputting the first audio alert includes generating an audio output indicating an amount of the displayed portion of the field of view obscured by the virtual object (e.g., augmented reality of the physical environment occupied by the virtual object). The amount of view (e.g. 20%, 25%, 50%, etc.)) (e.g. the audio output is a notification saying "object x occupying 15% of the world view" Including) (20010). In some embodiments, the audio output also includes a description of an action performed by the user that causes changes in the display state of the virtual object. For example, the audio output is "device moved to the left; object x is 20% visible, occupying 15% of the world view (device moved to the left; object x is 20% visible, occupying 15% of the world view). It includes a notification that says". For example, in FIG. 15Y, an audio alert 15126 including a notification 15128 indicating "chair is 90 percent visible, occupying 20 percent of the screen" is generated. Generating an audio output indicating the amount of augmented reality view obscured by the virtual object provides feedback to the user (eg, indicating the degree to which movement of the device has changed the degree to which the augmented reality view is obscured). Providing improved feedback to the user (e.g., information that allows the user to perceive the size of the virtual object relative to the display without confusing the display with additional displayed information and without requiring the user to see the display. Improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and shortens battery life by enabling users to use the device more quickly and efficiently. Improve.

In some embodiments, the device detects an input by contact at a location on a touch-sensitive surface corresponding to a representation of the field of view of one or more cameras (e.g., tapping on a portion of the touch screen that displays an augmented reality view of the physical environment. Detecting an input or double tap input) (20012). In response to detecting the input, and upon determining that the input is detected at a first location on the touch-sensitive surface corresponding to a first portion of the field of view of one or more cameras not occupied by the virtual object, the device An alert (eg, a click or buzz indicating that the location of the virtual object was not found within the tapped area) is generated (20014). For example, as described in connection with FIG. 15Z, responding to an input detected at a location on the touch screen 112 corresponding to a portion of the field of view 6036 of one or more cameras not occupied by the virtual object 11002 Thus, the device generates an audio alert 15130. In some embodiments, in response to detecting the input, generating a second audio alert in response to a determination that the input is detected at a second location corresponding to a second portion of the field of view of one or more cameras occupied by the virtual object. I withhold to do. In some embodiments, instead of generating a second audio alert indicating that the user has not found the location of the virtual object, the device generates a different audio alert indicating that the user has found the location of the virtual object. In some embodiments, instead of generating a second audio alert, the device provides an audio notification describing the action being performed on the virtual object (eg, “Object x selected”, “Object x is resized to”). a default size", "Object x is rotated to a default orientation", etc.), or an audio notification describing the state of the virtual object (e.g. , Prints "Object x, 20% visible, occupying 15% of the world view"). Generating an audio output in response to an input detected at a location corresponding to a portion of a displayed augmented reality view not occupied by the virtual object (e.g., the input is (e.g., acquiring information about the virtual object and/or operating Provide feedback to the user (indicating that it should be provided in a different location) to perform the action. Providing improved feedback to the user (e.g., without cluttering the display with additional displayed information and without requiring the user to see the display, allows the user to know whether the input has been successfully associated with the virtual object). Improves the device's operability and makes the user-device interface more efficient, which additionally reduces the device's power usage by enabling users to use the device more quickly and efficiently. Improves battery life.

In some embodiments, outputting the first audio alert is an action performed on the virtual object (e.g., prior to generating the audio output, the device determines the currently selected action and indicates the user's intention to perform the currently selected action. And generating an audio output indicating a result state of the virtual object after the operation is performed and an operation is performed in response to an input to be checked (eg, a double tap) (20016). For example, the audio output is "device moved to the left; object x is 20% visible, occupying 15% of the world view", "object x is rotated clockwise by 30 degrees; object is 50 degrees rotated around the y-axis (Object x is rotated clockwise by 30 degrees; object is rotated 50 degrees around y-axis)", or "object x enlarged by 20% and occupies 50% of the world view (object x is rotated by 20%) It is enlarged and takes up 50% of the real world view)". For example, as described in connection with FIGS. 15Ah and 15Ai, in response to performing a rotation operation on the virtual object 11002, "Chair is rotated by five degrees counterclockwise. Chair is now rotated by zero degrees relative to" An audio alert 15190 is generated that includes a notification 15192 indicating "the screen is rotated 5 degrees counterclockwise. The chair is now rotated 0 degrees relative to the screen". Generating an audio output representing an action performed on a virtual object provides feedback to the user indicating how the provided input affects the virtual object. Providing improved feedback to the user (e.g., without distracting the display with additional displayed information and without requiring the user to see the display, information that makes the user aware of how the action has changed the virtual object) Improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and shortens battery life by enabling users to use the device more quickly and efficiently. Improve.

In some embodiments, the resulting state of the virtual object after performing the operation is described as an audio output in the first audio alert with respect to a frame of reference corresponding to the physical environment captured within the field of view of one or more cameras (e.g. , After manipulating the object in response to a touch-based gesture or movement of the device, the device (e.g., 30 degrees relative to the initial position/orientation of the virtual object when the virtual object is initially placed into the augmented reality view of the physical environment) Create a voice over describing the new state of the object rotated, rotated 60 degrees, or moved to the left) (20018). For example, as described in connection with FIGS. 15Ah and 15Ai, in response to performing a rotation operation on the virtual object 11002, "Chair is rotated by five degrees counterclockwise. Chair is now rotated by zero degrees relative to" An audio alert 15190 is generated that includes a notification 15192 indicating "the screen". In some embodiments, the action includes movement of the device relative to the physical environment (e.g., causing movement of the virtual object relative to a representation of a portion of the physical environment that is captured within the field of view of one or more cameras), and the voice over Describe the new state of the virtual object in response to the device's movement relative to the physical environment. Generating an audio output indicating the state of the virtual object after the operation is performed on the object provides feedback to the user that allows the user to recognize how the operation has changed the virtual object. Providing improved feedback to the user (e.g., without distracting the display with additional displayed information and without requiring the user to see the display, information that makes the user aware of how the action has changed the virtual object) Improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and shortens battery life by enabling users to use the device more quickly and efficiently. Improve.

In some embodiments, the device detects additional movement of the device (e.g., lateral movement and/or rotation of a device comprising one or more cameras) that further adjusts the field of view of one or more cameras after generation of the first audio alert. Do (20020). For example, as described in connection with FIGS. 15W and 15X, movement of the device 100 (adjustment of the field of view of one or more cameras occurring in response to movement of the device 100 from FIGS. 15V and 15W) Afterwards) the field of view of one or more cameras is further adjusted. In response to detecting the further movement of the device that further adjusts the field of view of the one or more cameras, the device controls the detection between the virtual object and the plane detected within the field of view of the one or more cameras as the field of view of the one or more cameras is further adjusted. 1 Adjust the display of the representation of the virtual object in the first user interface area according to the spatial relationship (e.g., orientation and/or position), and (e.g., the plane and virtual detected in the physical environment captured within the field of view of one or more cameras) Since the spatial relationship between the representations of the objects remains fixed during movement of the device relative to the physical environment), further movement of the device causes the virtual object to exceed a second threshold amount (e.g., 50%, 80%, or 100%). Upon a determination to cause the one or more cameras to move into the displayed portion of the field of view, the device, via one or more audio output generators, provides a third audio alert (e.g., a notification indicating that a virtual object above a threshold is moved back into the camera view). An audio output including a) is generated (20022). For example, as described in connection with FIG. 15X, in response to movement of the device 100 causing the virtual object 11002 to move within the displayed portion of the field of view 6036 of one or more cameras, (e.g., a notification , An audio alarm (including "Chair is now projected in the world, 100 percent visible, occupying 10 percent of the screen"). 15122) is created. Generating the audio output in response to the determination that movement of the device causes the virtual object to move into the displayed augmented reality view provides feedback to the user indicating the extent to which movement of the device has affected the display of the virtual object relative to the augmented reality view. to provide. Providing improved feedback to the user (e.g., making the user aware of whether a virtual object has moved into the display without disturbing the display with additional displayed information and without requiring the user to see the display) Information) improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and reduces battery life by enabling users to use the device more quickly and efficiently. Improves.

In some embodiments, while displaying a representation of the virtual object in the first user interface area and the first object manipulation type among a plurality of object manipulation types applicable to the virtual object is currently selected for the virtual object, the device is Detects a request to switch to another object manipulation type applicable to the object (e.g., at a location on the touch-sensitive surface corresponding to a portion of the first user interface area displaying a representation of the field of view of one or more cameras (e.g. Detecting a swipe input by contact) including movement of the contact to (20024). For example, as described in connection with FIG. 15AG, while the clockwise rotation control unit 15170 is currently selected, the counterclockwise rotation control unit 15180 (to rotate the virtual object 15160 counterclockwise). A swipe input to switch to) is detected. In response to detecting a request to switch to another object manipulation type applicable to the virtual object, the device causes an audio output naming a second object manipulation type of the plurality of object manipulation types applicable to the virtual object (e.g., the audio output is "rotate object around x-axis", "resize object", or "move object on the plane", etc. ), the second object manipulation type is distinct from the first object manipulation type (20026). For example, in FIG. 15Ah, in response to detection of a request described in connection with FIG. 15Ag, an audio alert 15182 is generated that includes a notification 15184 ("selected: rotate counterclockwise"). In some embodiments, the device iterates through a predefined list of applicable object manipulation types in response to successive swipe inputs in the same direction. In some embodiments, in response to detecting a swipe input in the opposite direction from a swipe input that immediately precedes it, the device performs a previously known object manipulation type applicable to the virtual object (e.g., an object manipulation known just before the last). Produces an audio output containing a notification naming the type). In some embodiments, the device does not display a corresponding control for each object manipulation type applicable to the virtual object (e.g., actions initiated by gestures (e.g., rotate, resize, translate, etc.) There are no buttons or controls displayed for). Generating an audio output in response to a request to switch the object manipulation type provides feedback to the user indicating that the switch operation has been performed. Providing improved feedback to the user (e.g., by providing information confirming that the switching input has been successfully performed without disturbing the display with additional displayed information and without requiring the user to see the display. ) It improves the operability of the device and makes the user-device interface more efficient, which additionally reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

Audio output naming a second object manipulation type among a plurality of object manipulation types applicable to a virtual object (eg, audio output is “rotate object around x-axis”, “resize object”, or “move object on the plane” After generating the notification, etc.), the device detects a request to execute an object manipulation behavior corresponding to the currently selected object manipulation type (e.g., a first user interface that displays a representation of the field of view of one or more cameras). Detecting a double tap input by contact at a location on the touch-sensitive surface corresponding to a portion of the area (20028). For example, as described in connection with FIG. 15Ah, a double tap input for rotating the virtual object 11002 counterclockwise is detected. In response to detecting a request to perform an object manipulation behavior corresponding to the currently selected object manipulation type, the device executes the object manipulation behavior corresponding to the second object manipulation type (e.g., 5 Rotate by degrees, increase the size of the object by 5%, or move the object by 20 pixels on the plane (e.g., display the representation of the virtual object in the first user interface area according to the second object manipulation type) Adjusted) (20030). For example, in FIG. 15Ai, in response to detection of the request described in connection with FIG. 15Ah, the virtual object 11002 is rotated counterclockwise. In some embodiments, the device, in addition to executing the object manipulation behavior corresponding to the second object manipulation type, a notification indicating the object manipulation behavior executed on the virtual object and the resulting state of the virtual object after execution of the object manipulation behavior Outputs an audio output including. For example, in FIG. 15Ai, an audio output 15190 including a notification 15192 ("Chair rotated by five degrees counterclockwise. Chair is now rotated by zero degrees relative to the screen") is generated. Performing an object manipulation action in response to an input detected while an action is being selected provides additional control options for performing the action (e.g., by providing a tap input rather than requiring a two-touch input, allowing the user to perform the action. To make). Providing additional control options for providing input without cluttering the user interface with additional displayed controls (e.g., for manipulating objects to users with limited ability to provide multiple contact gestures) Options) improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and reduces battery life by enabling users to use the device more quickly and efficiently. Improves.

In some embodiments, in response to detecting a request to switch to another object manipulation type applicable to the virtual object, upon determining that the second object manipulation type is a continuously adjustable manipulation type, the device To indicate that the type is a continuously adjustable operation type, generate an audio alert with an audio output naming the second object manipulation type (eg, an audio notification naming the second object manipulation type (eg “rotate object clockwise around the y axis") then outputs an audio output called "adjustable"); The device may (e.g., after detect a double tap input by contact at a location on the touch-sensitive surface corresponding to a portion of the first user interface area displaying a representation of the field of view of the one or more cameras) of the field of view of one or more cameras. Detecting a request to execute an object manipulation behavior corresponding to the second object manipulation type, including detecting a swipe input at a location on the touch-sensitive surface corresponding to a portion of the first user interface area displaying the representation; In response to detecting a request to execute the object manipulation behavior corresponding to the second object manipulation type, the device executes the object manipulation behavior corresponding to the second object manipulation type by an amount corresponding to the size of the swipe input (e.g. , Depending on whether the size of the swipe input is a first quantity or a second quantity greater than the first quantity, the virtual object is rotated 5 degrees or 10 degrees around the y axis, or the size of the object is 5% or 10 %, or move the object by 20 or 40 pixels on the plane) (20032). For example, as described in connection with FIGS. 15J and 15K, while the clockwise rotation control unit 15038 is currently selected, a swipe input for switching to the zoom control unit 15064 is detected. An audio alert 15066 is created that includes a notification 15068 ("scale: adjustable"). 15K and 15L, a swipe input for zooming in on the virtual object 11002 is detected, and in response to the input, a zoom operation is performed on the virtual object 11002 (Fig. In the illustrative examples of 15K and 15L, an input for continuously adjustable operation is detected while the staging view interface 6010 is displayed, but in a portion of the first user interface area displaying a representation of the field of view of one or more cameras. It will be appreciated that a similar input can be detected at a location on the corresponding touch-sensitive surface). In some embodiments, the device, in addition to executing the second object manipulation behavior, includes an audio indicating the amount of the object manipulation behavior performed on the virtual object and the resulting state of the virtual object after execution of that amount of object manipulation behavior. Print a notification. Performing an object manipulation action in response to a swipe input provides additional control options for performing the action (eg, allowing the user to perform the action by providing a swipe input rather than requiring a two-touch input). Providing additional control options for providing input without cluttering the user interface with additional displayed controls (e.g., for manipulating objects to users with limited ability to provide multiple contact gestures) Options) improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and reduces battery life by enabling users to use the device more quickly and efficiently. Improves.

In some embodiments, prior to displaying the representation of the virtual object in the first user interface area, the device displays the representation of the virtual object in the second user interface area (eg, staging user interface), the second user interface area It does not contain a representation of the field of view of one or more cameras (e.g., the second user interface area is the virtual object manipulated (e.g., the second user interface area) without maintaining a fixed relationship to the plane detected in the physical environment captured within the A staging user interface that can be rotated, resized, and moved) (20034). While displaying the representation of the virtual object in the second user interface area, and while the first operation among a plurality of operations applicable to the virtual object is currently selected for the virtual object, the device (e.g., the virtual object in the second user interface area Object manipulation type applicable to (e.g., resize, rotate, tilt, etc.) or user interface actions applicable to virtual objects in the second user interface area (e.g., return to 2D user interface, object to augmented reality in physical environment) To detect a request to switch to another action applicable to the virtual object (including a request to switch to fall into view) (e.g., detecting the request is at a location on the touch-sensitive surface corresponding to the first user interface area). And detecting a swipe input by contact (eg, including movement of the contact in a horizontal direction) (20036). For example, as described in connection with FIGS. 15F and 15G, while the staging user interface 6010 is displayed and the tilt-down control unit 15022 is currently selected, for switching to the clockwise rotation control unit 15038 Swipe input is detected. In response to detecting a request to switch to another operation applicable to the virtual object in the second user interface area, the device outputs an audio output naming a second operation of the plurality of operations applicable to the virtual object (e.g., the audio output is "rotate object around x-axis", "resize object", "tilt the object toward the display", or "display object in the augmented reality view" )", etc.), the second operation being distinct from the first operation (20038). In some embodiments, the device iterates through a predefined list of applicable actions in response to successive swipe inputs in the same direction. For example, in FIG. 15G, in response to detection of a request described in connection with FIG. 15F, an audio alert 15040 including a notification 15042 ("selected: rotate clockwise button") is generated. Generating an audio output naming the selected operation type in response to a request to switch the operation type provides feedback to the user indicating that the switching input was successfully received. Generating an audio output naming the selected operation type in response to a request to switch the operation type provides feedback to the user indicating that the switching input was successfully received. Providing improved feedback to the user (e.g., without cluttering the display with additional displayed information and without requiring the user to see the display) provides information that makes the user aware when the selected control has changed. By providing) improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the device's power usage and improves battery life by enabling users to use the device faster and more efficiently. Let it.

In some embodiments, prior to displaying the representation of the virtual object in the first user interface area, the representation of the virtual object is placed in a second user interface area (e.g., a staging user interface) that does not contain a representation of the field of view of one or more cameras. During display (e.g., the second user interface area is a staging user interface in which virtual objects can be manipulated (e.g., rotated, resized, and moved) without maintaining a fixed relationship to a plane within the physical environment), The device detects a request to display a representation of the virtual object within a first user interface area that contains a representation of the field of view of one or more cameras (e.g., the currently selected action is "display the object in the augmented reality view. Display object)" and in response to a swipe input (e.g., received just before a double tap input), the device immediately outputs an audio notification naming the currently selected action and then detects a double tap input) (20040 ). For example, as described in connection with FIGS. 15P-15V, including a representation of the field of view 6036 of one or more cameras while the staging user interface 6010 is displayed and the toggle control 6018 is selected. A double tap input for displaying the representation of the virtual object 11002 in the user interface area is detected. In response to detecting a request to display a representation of the virtual object within a first user interface area that contains a representation of the field of view of one or more cameras, the device is configured to detect a plane and a virtual object in the physical environment captured within the field of view of one or more cameras. Display the representation of the virtual object in the first user interface area according to the first spatial relationship between the representation of the object (e.g., the rotation angle of the virtual object in the staging view when the virtual object falls into the physical environment expressed in the augmented reality view) And the size is maintained within the augmented reality view, and the tilt angle is reset within the augmented reality view according to the orientation of the plane detected in the physical environment captured within the field of view); The device generates a fourth audio alert indicating that the virtual object is placed within the augmented reality view in relation to the physical environment being captured within the field of view of one or more cameras. For example, in response to an input to display a representation of the virtual object 11002 within a user interface area that includes a representation of the field of view 6036 of one or more cameras, as described in connection with FIG. 15V, the virtual object ( 11002) is displayed within the user interface area including the field of view 6036 of one or more cameras, and a notification 15116 ("chair is now projected in the world, 100 percent visible, occupying 10 percent of the screen"). A containing audio alert 15114 is generated. Generating an audio output in response to a request to place the object within the augmented reality view provides feedback to the user indicating that the operation of placing the virtual object has been successfully executed. Providing improved feedback to the user (e.g., making the user aware that the object is displayed within the augmented reality view without disturbing the display with additional displayed information and without requiring the user to see the display) Information) improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and reduces battery life by enabling users to use the device more quickly and efficiently. Improves.

In some embodiments, the third audio alert indicates information about the appearance of the virtual object for a portion of the field of view of one or more cameras (eg, the third audio alert is “object x is placed in the world, object x is 30 % visible, occupying 90% of the screen (contains audio output with a notification saying "object x is placed in the real world, object x is 30% visible, occupies 90% of the screen)" (20042) . For example, as described in connection with FIG. 15V, an audio alert 15114 including a notification 15116 ("chair is now projected in the world, 100 percent visible, occupying 10 percent of the screen") is generated do. Generating an audio output representing the appearance of the virtual object visible to the displayed augmented reality view provides feedback to the user (e.g., indicating the extent to which the placement of the object within the augmented reality view affects the appearance of the virtual object). . Providing improved feedback to the user (e.g., making the user aware of how an object is displayed within an augmented reality view without disturbing the display with additional displayed information and without requiring the user to see the display) Improves the operability of the device and makes the user-device interface more efficient, which in addition reduces the power usage of the device and reduces battery power consumption by allowing users to use the device more quickly and efficiently. Improves lifespan.

In some embodiments, the device generates (20044) a tactile output with the placement of the virtual object within the augmented reality view in relation to the physical environment captured within the field of view of one or more cameras. For example, when an object is placed on a plane that is detected within the field of view of the cameras, the device generates a tactile output indicating the object's landing on the plane. In some embodiments, the device generates a tactile output when the object reaches a predefined default size during resizing of the object. In some embodiments, the device is configured for each operation performed on the virtual object (e.g., each rotation by a preset angle, dragging the virtual object onto a different plane, original orientation and/or Produces tactile outputs, such as resizing to size. In some embodiments, these tactile outputs precede corresponding audio alerts that describe the action being performed and the resulting state of the virtual object. For example, as described in connection with FIG. 15V, a tactile output 15118 is generated with the placement of the virtual object 11002 within the field of view 6036 of one or more cameras. Generating a tactile output along with the placement of the virtual object in relation to the physical environment captured by one or more cameras provides feedback to the user (e.g., indicating that the action to place the virtual object has been successfully executed) . Providing improved feedback to the user improves the operability of the device (e.g., by providing sensory information that allows the user to recognize that the placement of a virtual object has occurred without disturbing the user interface with the displayed information). And makes the user-device interface more efficient, which in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, the device, simultaneously with the representation of the field of view of one or more cameras, at a first location within the first user interface area (e.g., among a plurality of controls displayed at different locations within the first user interface area). The first control unit is displayed (20046). In accordance with a determination that the controller fading criteria are met (e.g., when the first user interface area is displayed for at least a threshold time while no touch input is detected on the touch-sensitive surface), the device Stop displaying the first control in the first user interface area (e.g., with all other controls in the first user interface area) while maintaining display of the representation of the field of view of one or more cameras in the interface area Controls are not redisplayed when the user moves the device relative to the physical environment) (20048). While displaying the first user interface area without displaying the first control unit in the first user interface area, the device receives a touch input at each location on the touch-sensitive surface corresponding to the first location in the first user interface area. Detect (20050). In response to detecting the touch input, the device outputs an audio output specifying an action corresponding to the first control unit (eg, “go back to staging view” or “rotate object around the y-axis(y -Rotate the object around the axis)") and generates a fifth audio alarm (20052). In some embodiments, the device also redisplays the first control at the first location in response to detecting the touch input. In some embodiments, upon redisplaying the control unit and making a touch input at the normal position of the control unit on the display, making the control unit the currently selected control unit uses a series of swipe inputs once the user knows the positions of the control units on the display. It provides a way to access controls, faster than scanning through the available controls. Automatically stopping displaying the control unit in response to determining that the control fading criteria are met reduces the number of inputs required to stop displaying the control unit. Reducing the number of inputs required to perform an operation improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, Reduces power usage and improves battery life.

It should be understood that the specific order in which the operations in FIGS. 20A-20F are described is merely an example and is not intended to indicate that the described order is the only order in which the operations may be performed. One of skill in the art will recognize various ways of reordering the operations described herein. Additionally, details of other processes described herein in connection with other methods described herein (e.g., methods 800, 900, 1000, 16000, 17000, 18000, 20000) are provided in FIGS. It should be noted that it is also applicable in a manner similar to method 20000 described above with respect to FIG. 20F. For example, the contacts, inputs, virtual objects, user interface regions, fields of view, tactile outputs, movements, and/or animations described above in connection with method 20000 are optionally, herein Contacts, inputs, virtual objects, user interface regions, field of view described herein in connection with other methods described (e.g., methods 800, 900, 1000, 16000, 17000, 18000, 19000) S, tactile outputs, movements, and/or one or more of the characteristics of animations. For brevity, these details are not repeated here.

8A to 8E, 9A to 9D, 10A to 10D, 16A to 16G, 17A to 17D, 18A to 18I, 19A to 19H, and 20A to 20F. The operations described above are, optionally, implemented by the components shown in FIGS. 1A and 1B. For example, display operations 802, 806, 902, 906, 910, 1004, 1008, 16004, 17004, 18002, 19002, 20002; Detection operations 804, 904, 908, 17006, 18004, 19004, 20004; Change operation 910, receive operations 1002, 1006, 16002, 17002; Stop operations 17008; Rotating operation 18006; Update operation 19006; Adjustment operation 20006; And the generating operation 20006 is optionally implemented by the event classifier 170, the event recognizer 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects a contact on the touch-sensitive display 112, and the event dispatcher module 174 delivers event information to the application 136-1. Each event recognizer 180 of the application 136-1 compares the event information with the respective event definitions 186, and the first contact (or rotation of the device) at a first location on the touch-sensitive surface is Determine whether it corresponds to a predefined event or sub-event, such as selection of an object on the user interface, or rotation of the device from one orientation to another. When each predefined event or subevent is detected, the event recognizer 180 activates an event handler 190 associated with detection of the event or subevent. The event handler 190 optionally updates the application internal state 192 by using or calling the data updater 176 or the object updater 177. In some embodiments, event handler 190 accesses each GUI updater 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes may be implemented based on the components shown in FIGS. 1A and 1B.

The foregoing description has been described with reference to specific embodiments, for purposes of explanation. However, the exemplary discussions above are not intended to characterize or limit the invention to the precise forms disclosed. Many modifications and variations are possible in terms of the above teachings. To best explain the principles of the invention and its practical applications, thereby enabling those skilled in the art to best use the invention and the various described embodiments with various modifications as suitable for the particular application contemplated. Examples have been selected and described.

Claims (209)

As a method,
In a device with a display, a touch-sensitive surface, and one or more cameras:
Displaying a representation of a virtual object in a first user interface area on the display;
While displaying the first representation of the virtual object in the first user interface area on the display, detecting a first input by contact at a location on the touch-sensitive surface corresponding to the representation of the virtual object on the display step; And
In response to detecting the first input by the contact:
Depending on the determination that the first input by the contact meets first criteria:
Displaying a second user interface area on the display, including replacing a display of at least a portion of the first user interface area with a representation of the field of view of the one or more cameras;
And continuously displaying a representation of the virtual object while switching from displaying the first user interface area to displaying the second user interface area. The method of claim 1, wherein the first criteria are satisfied criteria when the contact is maintained with a movement less than a threshold amount for at least a predefined time at the location on the touch-sensitive surface corresponding to the representation of the virtual object. Containing, method. The method of claim 1,
The device includes one or more sensors that detect intensities of contacts with the touch-sensitive surface;
The first criteria include criteria that are satisfied when the characteristic strength of the contact increases beyond a first strength threshold. 2. The method of claim 1, wherein the first criteria include criteria that are satisfied when the movement of the contact meets predefined movement criteria. The method according to any one of claims 1 to 4,
The device comprises one or more tactile output generators;
The method comprises, in response to detecting the first input by the contact, the first criterion by the first input, according to a determination that the first input by the contact has met the first criteria. And outputting, by the one or more tactile output generators, a tactile output to indicate satisfaction of the people. The method according to any one of claims 1 to 5,
In response to detecting at least an initial portion of the first input, analyzing the field of view of the one or more cameras to detect one or more planes within the field of view of the one or more cameras; And
After detecting each plane within the field of view of the one or more cameras, determining the size and/or position of the representation of the virtual object based on the relative position of the respective plane with respect to the field of view of the one or more cameras. Including, how. The method of claim 6, wherein analyzing the field of view of the one or more cameras to detect the one or more planes within the field of view of the one or more cameras is on the touch-sensitive surface corresponding to the representation of the virtual object on the display. Initiated in response to detection of a contact at the location. The method of claim 6, wherein analyzing the field of view of the one or more cameras to detect the one or more planes within the field of view of the one or more cameras comprises the first criteria being satisfied by the first input by the contact. Initiated in response to detecting that. The method of claim 6, wherein analyzing the field of view of the one or more cameras to detect the one or more planes within the field of view of the one or more cameras while an initial portion of the first input does not meet the first criteria. Initiated in response to detecting that plane-detection trigger criteria are met. The method according to any one of claims 6 to 9,
Displaying a representation of the virtual object in the second user interface area in a respective manner such that the virtual object is oriented at a predefined angle with respect to each plane detected within the field of view of the one or more cameras. , Way. The method of claim 10,
The device comprises one or more tactile output generators;
The method, in response to detecting the respective plane within the field of view of the one or more cameras, generates a tactile output to indicate detection of the respective plane within the field of view of the one or more cameras, the one or more tactile outputs. Outputting, by output generators. The method of claim 10 or 11,
The device comprises one or more tactile output generators;
The above method,
While switching from displaying the first user interface area to displaying the second user interface area, the representation of the virtual object transitions into the second user interface area to a predefined position for each plane. displaying the animation as it becomes (transition); And
In addition to displaying the representation of the virtual object at the predefined angle with respect to each of the planes, it indicates displaying the virtual object at the predefined angle with respect to the respective plane within the second user interface area. And outputting, by the one or more tactile output generators, a tactile output for production. 13. The method of claim 12, wherein the tactile output has a tactile output profile corresponding to a characteristic of the virtual object. The method according to any one of claims 10 to 13,
Detecting movement of the device to adjust the field of view of the one or more cameras while displaying the representation of the virtual object within the second user interface area; And
In response to detecting movement of the device, the second user interface region according to a fixed spatial relationship between the virtual object and each plane within the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted Adjusting the representation of the virtual object within. The animation according to any one of claims 1 to 14, wherein the representation of the virtual object is continuously displayed while switching from displaying the first user interface area to displaying the second user interface area. Displaying a. The method according to any one of claims 1 to 15,
While displaying the second user interface area on the display, detecting a second input by a second contact, the second input detecting movement of the second contact along a first path across the display Contains -; And
In response to detecting the second input by the second contact, moving the representation of the virtual object within the second user interface area along a second path corresponding to the first path, Way. The method of claim 16, further comprising: adjusting the size of the representation of the virtual object as the representation of the virtual object moves along the second path based on the movement of the contact and each plane corresponding to the virtual object. Containing, method. The method of claim 16,
Maintaining a first size of the representation of the virtual object as the representation of the virtual object moves along the second path;
Detecting an end of the second input by the second contact; And
In response to detecting the termination of the second input by the second contact:
Arranging the representation of the virtual object at a drop-off position in the second user interface area;
And displaying the representation of the virtual object at the falling position in the second user interface area with a second size distinct from the first size. 19. A method according to any one of claims 16 to 18, according to a determination that the movement of the second contact along the first path across the display meets second criteria:
Stopping displaying the second user interface area containing a representation of the field of view of the one or more cameras; And
And re-displaying the first user interface area with the representation of the virtual object. The method of claim 19,
At a time corresponding to re-displaying the first user interface area, from displaying the representation of the virtual object in the second user interface area to displaying the representation of the virtual object in the first user interface area. And displaying an animated transition. 21. The method of any one of claims 16-20, wherein as the second contact moves along the first path, each of the one or more identified within the field of view of the one or more cameras corresponding to the current location of the contact. Changing the visual appearance of the planes of the. 22. The method according to any one of the preceding claims, in response to detecting the first input by the contact, in response to a determination that the first input by the contact meets third criteria, the second input. And displaying a third user interface area on the display, including replacing a display of at least a portion of the first user interface area. The display of at least a portion of the first user interface area according to any one of claims 1 to 22, wherein in response to a determination that the first input by the contact does not meet the first criteria. Maintaining a display of the first user interface area without replacing with a representation of the field of view of the cameras. As a computer system,
display;
Touch-sensitive surface;
One or more cameras;
One or more processors; And
And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs,
Displaying a representation of a virtual object in a first user interface area on the display;
While displaying the first representation of the virtual object in the first user interface area on the display, detecting a first input by contact at a location on the touch-sensitive surface corresponding to the representation of the virtual object on the display for; And
In response to detecting the first input by the contact:
Depending on the determination that the first input by the contact meets first criteria:
Displaying a second user interface area on the display, including replacing a display of at least a portion of the first user interface area with a representation of the field of view of the one or more cameras;
And instructions for continuously displaying a representation of the virtual object while switching from displaying the first user interface area to displaying the second user interface area. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display, a touch-sensitive surface, and one or more cameras, cause the computer system to:
Displaying a representation of a virtual object in a first user interface area on the display;
While displaying the first representation of the virtual object in the first user interface area on the display, detecting a first input by contact at a location on the touch-sensitive surface corresponding to the representation of the virtual object on the display doing; And
In response to detecting the first input by the contact:
Depending on the determination that the first input by the contact meets first criteria:
Displaying a second user interface area on the display, including replacing a display of at least a portion of the first user interface area with a representation of the field of view of the one or more cameras;
A computer-readable storage medium comprising instructions for causing to continuously display a representation of the virtual object while switching from displaying the first user interface area to displaying the second user interface area. As a computer system,
display;
Touch-sensitive surface;
One or more cameras;
Means for displaying a representation of a virtual object within a first user interface area on the display;
By contact at a location on the touch-sensitive surface corresponding to the representation of the virtual object on the display, enabled while displaying a first representation of the virtual object in the first user interface area on the display Means for detecting a first input; And
Means enabled in response to detecting the first input by the contact, the means enabled in response to detecting the first input by the contact,
Enabled according to a determination that the first input by the contact meets first criteria,
Displaying a second user interface area on the display, including replacing a display of at least a portion of the first user interface area with a representation of the field of view of the one or more cameras;
And means for continuously displaying a representation of the virtual object while switching from displaying the first user interface area to displaying the second user interface area. An information processing apparatus for use in a computer system having a display, a touch-sensitive surface, and one or more cameras, comprising:
Means for displaying a representation of a virtual object within a first user interface area on the display;
A first input by contact at a location on the touch-sensitive surface corresponding to a representation of the virtual object on the display, enabled while displaying a first representation of the virtual object in the first user interface area on the display Means for detecting an error; And
Means enabled in response to detecting the first input by the contact, the means enabled in response to detecting the first input by the contact,
Enabled according to a determination that the first input by the contact meets first criteria,
Displaying a second user interface area on the display, including replacing a display of at least a portion of the first user interface area with a representation of the field of view of the one or more cameras;
And means for continuously displaying a representation of the virtual object while switching from displaying the first user interface area to displaying the second user interface area. As a computer system,
display;
Touch-sensitive surface;
One or more cameras;
One or more processors; And
24. A computer system comprising a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any one of claims 1 to 23. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display, a touch-sensitive surface, and one or more cameras, cause the computer system to cause the computer system to A computer readable storage medium comprising instructions for causing to perform the method of claim 1. A graphical user interface on a computer system having a display, a touch-sensitive surface, one or more cameras, a memory, and one or more processors for executing one or more programs stored in the memory, the graphical user interface of any one of the preceding claims. A graphical user interface comprising user interfaces displayed according to the method. As a computer system,
display;
Touch-sensitive surface;
One or more cameras; And
A computer system comprising means for performing the method of claim 1. An information processing apparatus for use in a computer system having a display, a touch-sensitive surface, and one or more cameras, comprising:
An information processing apparatus comprising means for performing the method of any one of claims 1 to 23. As a method,
In a device with a display, a touch-sensitive surface, and one or more cameras:
Displaying a first representation of a virtual object in a first user interface area on the display;
While displaying the first representation of the virtual object in the first user interface area on the display, a first contact by a first contact at a location on the touch-sensitive surface corresponding to the first representation of the virtual object on the display Detecting an input;
A second user interface different from the first user interface area in response to detecting the first input by the first contact and according to a determination that the first input by the first contact meets first criteria Displaying a second representation of the virtual object in an area;
Detecting a second input while displaying a second representation of the virtual object in the second user interface area; And
In response to detecting the second input:
According to a determination that the second input corresponds to a request to manipulate the virtual object in the second user interface area, the display attribute of the second representation of the virtual object in the second user interface area based on the second input Change it;
In response to a determination that the second input corresponds to a request to display the virtual object in an augmented reality environment, displaying a third representation of the virtual object along with a representation of the field of view of the one or more cameras. The method of claim 33, wherein the first criteria include criteria that are satisfied when the first input includes a tap input by the first contact at a location on the touch-sensitive surface corresponding to a virtual object indicator. Way. 34. The method of claim 33, wherein the first criteria are such that the first contact is to be maintained with less than a threshold amount of movement for at least a predefined threshold time at the location on the touch-sensitive surface corresponding to the first representation of the virtual object. The method, including criteria that are satisfied when. The method of claim 33,
The device includes one or more sensors that detect intensities of contacts with the touch-sensitive surface;
The first criteria include criteria that are satisfied when the characteristic strength of the first contact increases above a first strength threshold. The method of claim 33,
In response to detecting the first input by the first contact and according to a determination that the first input by the first contact meets second criteria-the second criteria are determined by the first input being the touch Requiring to include movement of the first contact for exceeding a threshold distance in a direction crossing the responsive surface, including scrolling the first user interface area in a direction corresponding to a direction of movement of the first contact How to. The method according to any one of claims 33 to 37,
In response to detecting the first input by the first contact and according to a determination that the first input by the first contact meets third criteria, the virtual Displaying a third representation of the object. The method of claim 38,
The device comprises one or more device orientation sensors;
The method includes, in response to detecting the first input by the first contact, determining, by the one or more device orientation sensors, a current device orientation of the device;
Wherein the third criteria require that the current device orientation be within a first range of orientations in order for the third criteria to be met. 40. The method according to any of claims 33 to 39, wherein at least one display attribute of the second representation of the virtual object is applied to the third representation of the virtual object. The method according to any one of claims 33 to 40,
In response to detecting at least an initial portion of the first input by the first contact:
Activating the one or more cameras; And
And analyzing the field of view of the one or more cameras to detect one or more planes within the field of view of the one or more cameras. The method of claim 41,
The device comprises one or more tactile output generators;
The method, in response to detecting each plane within the field of view of the one or more cameras, generates a tactile output to indicate detection of each plane within the field of view of the one or more cameras, the one or more tactile output generators. By way of, outputting. The method according to any one of claims 33 to 42, wherein the size of the third representation of the virtual object on the display is a position in the field of view of the one or more cameras having a fixed spatial relationship with the third representation of the virtual object. A distance between the one or more cameras and a simulated real-world size of the virtual object. The method of any one of claims 33 to 43, wherein the second input corresponding to a request to display the virtual object in an augmented reality environment comprises an input of dragging a second representation of the virtual object. , Way. The method of any one of claims 33 to 44, wherein while displaying a second representation of the virtual object within the second user interface area, each of the criteria for redisplaying the first user interface area is met. Detecting a fourth input; And
In response to detecting the fourth input:
Stopping displaying a second representation of the virtual object within the second user interface area;
And re-displaying the first representation of the virtual object within the first user interface area. The method according to any one of claims 33 to 45,
While displaying the third representation of the virtual object with the representation of the field of view of the one or more cameras, detecting a fifth input that meets respective criteria for redisplaying the second user interface area; And
In response to detecting the fifth input:
Stopping displaying the third representation of the virtual object and the representation of the field of view of the one or more cameras;
And re-displaying the second representation of the virtual object within the second user interface area. 47. The method of any one of claims 33 to 46, wherein each criterion for redisplaying the first user interface area while displaying the third representation of the virtual object along with the representation of the field of view of the one or more cameras. Detecting a sixth input that satisfies them; And
In response to detecting the sixth input:
Stopping displaying the third representation of the virtual object and the representation of the field of view of the one or more cameras;
And re-displaying the first representation of the virtual object within the first user interface area. The method according to any one of claims 33 to 47,
In response to detecting the first input by the first contact and according to a determination that the input by the contact meets the first criteria, a first representation of the virtual object in the first user interface area is Including displaying an animation converted to a second representation of the virtual object in the second user interface area, when transitioning from displaying the first user interface area to displaying the second user interface area, the And continuously displaying the virtual object. The method according to any one of claims 33 to 48,
In response to detecting the second input by the second contact and according to a determination that the second input by the second contact corresponds to a request to display the virtual object in the augmented reality environment, the second And displaying an animation in which the second representation of the virtual object in the user interface area is converted into a third representation of the virtual object in a third user interface area including the field of view of the one or more cameras, And continuously displaying the virtual object when transitioning from displaying an area to displaying the third user interface area comprising a field of view of the one or more cameras. As a computer system,
display;
Touch-sensitive surface;
One or more cameras;
One or more processors; And
And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs,
Displaying a first representation of a virtual object in a first user interface area on the display;
While displaying the first representation of the virtual object in the first user interface area on the display, a first contact by a first contact at a location on the touch-sensitive surface corresponding to the first representation of the virtual object on the display For detecting an input;
In response to detecting the first input by the first contact and in accordance with a determination that the input by the first contact meets first criteria, within a second user interface area different from the first user interface area For displaying a representation of the virtual object;
While displaying a second representation of the virtual object within the second user interface area, detecting a second input; And
In response to detecting the second input:
According to a determination that the second input corresponds to a request to manipulate the virtual object in the second user interface area, the display attribute of the second representation of the virtual object in the second user interface area based on the second input Change it;
A computer comprising instructions for displaying a third representation of the virtual object along with a representation of the field of view of the one or more cameras according to a determination that the second input corresponds to a request to display the virtual object in an augmented reality environment system. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display, a touch-sensitive surface, and one or more cameras, cause the computer system to:
Displaying a first representation of a virtual object in a first user interface area on the display;
While displaying the first representation of the virtual object in the first user interface area on the display, a first contact by a first contact at a location on the touch-sensitive surface corresponding to the first representation of the virtual object on the display To detect the input;
In response to detecting the first input by the first contact and in accordance with a determination that the input by the first contact meets first criteria, within a second user interface area different from the first user interface area Display a representation of the virtual object;
Detecting a second input while displaying a second representation of the virtual object in the second user interface area; And
In response to detecting the second input:
According to a determination that the second input corresponds to a request to manipulate the virtual object in the second user interface area, the display attribute of the second representation of the virtual object in the second user interface area based on the second input To change it;
A computer comprising instructions for causing, in response to a determination that the second input corresponds to a request to display the virtual object in an augmented reality environment, to display a third representation of the virtual object along with a representation of the field of view of the one or more cameras Readable storage medium. As a computer system,
display;
Touch-sensitive surface;
One or more cameras;
Means for displaying a first representation of a virtual object within a first user interface area on the display;
To a first contact at a location on the touch-sensitive surface corresponding to the first representation of the virtual object on the display, enabled while displaying the first representation of the virtual object in the first user interface area on the display Means for detecting a first input by;
A second user different from the first user interface area, enabled in response to detecting the first input by the first contact and in accordance with a determination that the input by the first contact meets first criteria Means for displaying a representation of the virtual object within an interface area;
Means for detecting a second input, enabled while displaying a second representation of the virtual object within the second user interface area; And
Means enabled in response to detecting the second input, the means enabled in response to detecting the second input,
A second representation of the virtual object in the second user interface area based on the second input, enabled in response to a determination that the second input corresponds to a request to manipulate the virtual object in the second user interface area Means for changing the display property of the device; And
Means for displaying a third representation of the virtual object along with a representation of the field of view of the one or more cameras, enabled in response to a determination that the second input corresponds to a request to display the virtual object in an augmented reality environment. Ham-containing, computer system. An information processing apparatus for use in a computer system having a display, a touch-sensitive surface, and one or more cameras, comprising:
Means for displaying a first representation of a virtual object within a first user interface area on the display;
To a first contact at a location on the touch-sensitive surface corresponding to the first representation of the virtual object on the display, enabled while displaying the first representation of the virtual object in the first user interface area on the display Means for detecting a first input by;
A second user different from the first user interface area, enabled in response to detecting the first input by the first contact and in accordance with a determination that the input by the first contact meets first criteria Means for displaying a representation of the virtual object within an interface area;
Means for detecting a second input, enabled while displaying a second representation of the virtual object within the second user interface area; And
Means enabled in response to detecting the second input, the means enabled in response to detecting the second input,
A second representation of the virtual object in the second user interface area based on the second input, enabled in response to a determination that the second input corresponds to a request to manipulate the virtual object in the second user interface area Means for changing the display property of the device; And
Means for displaying a third representation of the virtual object along with a representation of the field of view of the one or more cameras, enabled in accordance with a determination that the second input corresponds to a request to display the virtual object in an augmented reality environment. Ham-comprising, information processing apparatus. As a computer system,
display;
Touch-sensitive surface;
One or more cameras;
One or more processors; And
A computer system comprising a memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of any one of claims 33-49. A computer-readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display, a touch-sensitive surface, and one or more cameras, cause the computer system to cause the A computer readable storage medium comprising instructions for causing to perform the method of claim 1. A graphical user interface on a computer system having a display, a touch-sensitive surface, one or more cameras, a memory, and one or more processors for executing one or more programs stored in the memory, the graphical user interface of any one of claims 33 to 49 A graphical user interface comprising user interfaces displayed according to the method. As a computer system,
display;
Touch-sensitive surface;
One or more cameras; And
A computer system comprising means for performing the method of any of claims 33-49. An information processing apparatus for use in a computer system having a display, a touch-sensitive surface, and one or more cameras, comprising:
An information processing apparatus comprising means for performing the method of any of claims 33-49. As a method,
On devices with displays and touch-sensitive surfaces:
Receiving a request to display a first user interface including a first item;
In response to the request to display the first user interface, displaying the first user interface together with the expression of the first item-displaying the first user interface,
In accordance with a determination that the first item corresponds to each virtual 3D object, displaying a representation of the first item with a visual indication indicating that the first item corresponds to each of the first virtual 3D objects ; And
In accordance with the determination that the first item does not correspond to each virtual 3D object, including the step of displaying a representation of the first item without the visual indication;
Receiving a request to display a second user interface including a second item after displaying a representation of the first item; And
In response to the request to display the second user interface, displaying the second user interface together with the expression of the second item-displaying the second user interface,
According to a determination that the second item corresponds to each virtual 3D object, displaying a representation of the second item together with the visual indication indicating that the second item corresponds to each second virtual 3D object step; And
And, according to a determination that the second item does not correspond to each virtual three-dimensional object, displaying a representation of the second item without the visual indication. The method of claim 59,
The device comprises one or more device orientation sensors;
Displaying the representation of the first item together with the visual indication indicating that the first item corresponds to each first virtual three-dimensional object,
In response to detecting a movement of the device resulting in a change from a first device orientation to a second device orientation, a movement of the first item corresponding to a change from the first device orientation to the second device orientation is performed. Displaying. 61. The method of claim 59 or 60, wherein displaying the representation of the first item with the visual indication indicating that the first item corresponds to each first virtual three-dimensional object comprises:
In response to detecting a first input by a first contact scrolling the first user interface while the representation of the first item is displayed in the first user interface:
Translating the representation of the first item on the display according to the scrolling of the first user interface; And
Rotating the representation of the first item with respect to a plane defined by the first user interface according to a direction in which the first user interface is scrolled. 62. The method of any one of claims 59-61, comprising displaying the representation of the third item while displaying the representation of the first item together with the visual indication within the first user interface, wherein the The representation of the third item is displayed without the visual indication to indicate that the third item does not correspond to a virtual three-dimensional object. 63. The method of any one of claims 59-62, comprising displaying the representation of the fourth item while displaying the representation of the second item together with the visual indication within the second user interface, wherein the The representation of the fourth item is displayed without the visual indication to indicate that the fourth item does not correspond to each virtual three-dimensional object. The method according to any one of claims 59 to 63,
The first user interface corresponds to a first application;
The second user interface corresponds to a second application that is distinct from the first application;
Wherein the representation of the first item displayed with the visual indication and the representation of the second item displayed with the visual indication share a predefined set of visual properties and/or behavioral properties. 65. The method of any of claims 59-64, wherein the first user interface is an internet browser application user interface and the first item is an element of a web page. 65. The method of any of claims 59-64, wherein the first user interface is an email application user interface and the first item is an attachment to an email. 65. The method of any of claims 59-64, wherein the first user interface is a messaging application user interface and the first item is an attachment or element within a message. 65. The method of any of claims 59-64, wherein the first user interface is a file management application user interface and the first item is a file preview object. 65. The method of any of claims 59-64, wherein the first user interface is a map application user interface and the first item is a representation of a point of interest in a map. The method according to any one of claims 59 to 69, wherein the visual indication that the first item corresponds to each virtual three-dimensional object occurs without requiring an input for the representation of each three-dimensional object. An animation of the first item. The method according to any one of claims 59 to 70,
The device comprises one or more cameras;
The above method,
While displaying the representation of the second item with the visual indication indicating that the second item corresponds to each virtual three-dimensional object, a second item at a location on the touch-sensitive surface corresponding to the representation of the second item Detecting a second input by two contacts;
In response to detecting the second input by the second contact and according to a determination that the second input by the second contact meets first criteria:
Displaying a third user interface area on the display, including replacing a display of at least a portion of the second user interface with a representation of the field of view of the one or more cameras; And
And continuously displaying the second virtual three-dimensional object while switching from displaying the second user interface to displaying the third user interface area. The method according to any one of claims 59 to 70,
The device comprises one or more cameras;
The above method,
While displaying the second item with the visual indication that the second item corresponds to the second virtual three-dimensional object, a third at a position on the touch-sensitive surface corresponding to the representation of the second item Detecting a third input by contact;
In response to detecting the third input by the third contact and according to a determination that the third input by the third contact meets first criteria, within a fourth user interface different from the second user interface Displaying the second virtual 3D object;
Detecting a fourth input while displaying each of the virtual three-dimensional objects in the fourth user interface; And
In response to detecting the fourth input:
According to a determination that the fourth input corresponds to a request to manipulate the second virtual 3D object in the fourth user interface, the second virtual 3D object in the fourth user interface is Change display properties;
In accordance with a determination that the fourth input corresponds to a request to display the second virtual object in an augmented reality environment, displaying the second virtual 3D object together with a representation of the field of view of the one or more cameras, Way. As a computer system,
display;
Touch-sensitive surface;
One or more processors; And
And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs,
For receiving a request to display a first user interface including a first item;
In response to the request to display the first user interface, for displaying the first user interface together with the expression of the first item,
According to a determination that the first item corresponds to each virtual three-dimensional object, displaying a representation of the first item with a visual indication indicating that the first item corresponds to each of the first virtual three-dimensional objects;
According to a determination that the first item does not correspond to each virtual three-dimensional object, displaying a representation of the first item without the visual indication;
After displaying the representation of the first item, receiving a request to display a second user interface including a second item; And
In response to the request to display the second user interface, for displaying the second user interface together with the expression of the second item,
According to the determination that the second item corresponds to each virtual three-dimensional object, displaying a representation of the second item together with the visual indication indicating that the second item corresponds to each second virtual three-dimensional object, and ;
A computer system comprising instructions for displaying a representation of the second item without the visual indication, upon determining that the second item does not correspond to each virtual three-dimensional object. A computer-readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display and a touch-sensitive surface, cause the computer system to:
Receive a request to display a first user interface including a first item;
In response to the request to display the first user interface, causing the first user interface to be displayed together with the expression of the first item,
According to a determination that the first item corresponds to each virtual 3D object, display a representation of the first item together with a visual indication indicating that the first item corresponds to each of the first virtual 3D objects, and ;
In accordance with a determination that the first item does not correspond to each virtual 3D object, causing to display the representation of the first item without the visual indication;
Receiving a request to display a second user interface including a second item after displaying the representation of the first item; And
In response to the request to display the second user interface, causing the second user interface to be displayed together with the expression of the second item,
According to a determination that the second item corresponds to each virtual 3D object, display a representation of the second item together with the visual indication indicating that the second item corresponds to each second virtual 3D object and;
A computer-readable storage medium comprising instructions for causing, upon determining that the second item does not correspond to each virtual three-dimensional object, to display a representation of the second item without the visual indication. As a computer system,
display;
Touch-sensitive surface;
Means for receiving a request to display a first user interface including a first item;
Means for displaying the first user interface with a representation of the first item, enabled in response to the request to display the first user interface,
The expression of the first item is displayed along with a visual indication that the first item corresponds to each first virtual 3D object, which is enabled according to a determination that the first item corresponds to each virtual 3D object Means for displaying; And
Means for displaying a representation of the first item without the visual indication, enabled in response to a determination that the first item does not correspond to each virtual three-dimensional object;
Means for receiving a request to display a second user interface including a second item, which is enabled after displaying a representation of the first item; And
Means for displaying the second user interface with a representation of the second item, enabled in response to the request to display the second user interface,-displaying the second user interface with a representation of the second item Means for displaying,
Expression of the second item with the visual indication indicating that the second item corresponds to each of the second virtual 3D objects, enabled according to a determination that the second item corresponds to each virtual 3D object Means for displaying; And
And means for displaying a representation of the second item without the visual indication, enabled in accordance with a determination that the second item does not correspond to each virtual three-dimensional object. An information processing apparatus for use in a computer system having a display and a touch-sensitive surface, comprising:
Means for receiving a request to display a first user interface including a first item;
Means for displaying the first user interface with a representation of the first item, enabled in response to the request to display the first user interface,
The expression of the first item is displayed along with a visual indication that the first item corresponds to each first virtual 3D object, which is enabled according to a determination that the first item corresponds to each virtual 3D object Means for displaying; And
Means for displaying a representation of the first item without the visual indication, enabled in response to a determination that the first item does not correspond to each virtual three-dimensional object;
Means for receiving a request to display a second user interface including a second item, which is enabled after displaying a representation of the first item; And
Means for displaying the second user interface with a representation of the second item, enabled in response to the request to display the second user interface,-displaying the second user interface with a representation of the second item Means for displaying,
Expression of the second item with the visual indication indicating that the second item corresponds to each of the second virtual 3D objects, enabled according to a determination that the second item corresponds to each virtual 3D object Means for displaying; And
And means for displaying a representation of the second item without the visual indication, enabled in response to a determination that the second item does not correspond to each virtual three-dimensional object. As a computer system,
display;
Touch-sensitive surface;
One or more processors; And
A computer system comprising a memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of any one of claims 59-72. A computer-readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display and a touch-sensitive surface, cause the computer system to cause the computer system of any one of claims 59 to 72. A computer-readable storage medium containing instructions that cause a method to be performed. A graphical user interface on a computer system having a display, a touch-sensitive surface, a memory, and one or more processors for executing one or more programs stored in the memory, which is displayed according to the method of any one of claims 59-72. Graphical user interface, including user interfaces. As a computer system,
display;
Touch-sensitive surface; And
A computer system comprising means for performing the method of any of claims 59-72. An information processing apparatus for use in a computer system having a display and a touch-sensitive surface, comprising:
An information processing apparatus comprising means for performing the method of any one of claims 59-72. As a method,
In a device with a display generating component, one or more input devices, and one or more cameras:
Receiving a request to display a virtual object within a first user interface area that includes at least a portion of the field of view of the one or more cameras; And
In response to the request to display the virtual object in the first user interface area, through the display generating component, the representation of the virtual object on at least a portion of the field of view of the one or more cameras included in the first user interface area Displaying a field of view of the one or more cameras is a view of a physical environment in which the one or more cameras are located, and displaying a representation of the virtual object comprises
Upon determination that object placement criteria are not met-the object placement criteria require that the placement location for the virtual object be identified within the field of view of the one or more cameras in order for the object placement criteria to be satisfied -, a first set of Displaying a representation of the virtual object with visual properties and in a first orientation independent of which portion of the physical environment is displayed within the field of view of the one or more cameras; And
According to a determination that the object placement criteria are satisfied, a second set of visual attributes distinct from the first set of visual attributes and a first corresponding to a plane in the physical environment detected within the field of view of the one or more cameras. And displaying the representation of the virtual object in two orientations. The method of claim 82,
And detecting that the object placement criteria are met while the representation of the virtual object is displayed in the first set of visual properties and in the first orientation. The method of claim 83,
In response to detecting that the object placement criteria are met, the second set of visual attributes from moving from the first orientation to the second orientation and having the first set of visual attributes, via the display generating component. Displaying an animated transition showing a representation of the virtual object that changes to have properties. The method of claim 83 or 84, wherein detecting that the object placement criteria are met,
Detecting that a plane has been identified within the field of view of the one or more cameras;
Detecting less than a threshold amount of movement between the device and the physical environment for at least a threshold time; And
And detecting that at least a predetermined time has elapsed since receiving the request to display the virtual object within the first user interface area. The method according to any one of claims 82 to 85,
The first movement of the one or more cameras while the representation of the virtual object over a first portion of the physical environment captured within the field of view of the one or more cameras is displayed in the first set of visual properties and the first orientation. Detecting; And
In response to detecting a first movement of the one or more cameras, the virtual object in the first orientation and the first set of visual properties over a second portion of the physical environment captured within the field of view of the one or more cameras. Displaying a representation of the physical environment, wherein the second portion of the physical environment is distinct from the first portion of the physical environment. The method according to any one of claims 82 to 86,
The second movement of the one or more cameras while the representation of the virtual object is displayed in the second orientation and the second set of visual properties over a third portion of the physical environment captured within the field of view of the one or more cameras. Detecting; And
In response to detecting the second movement of the device, the physical environment as captured within the field of view of the one or more cameras is moved according to the second movement of the device, and the second orientation of the one or more cameras The second set of visual properties and the second orientation over a third portion of the physical environment captured within the field of view of the one or more cameras, while continuing to correspond to the plane within the physical environment detected within the field of view. Maintaining a display of the representation of the virtual object. The method according to any one of claims 82 to 87,
Representation of the virtual object in the second orientation corresponding to the plane within the physical environment detected within the field of view of the one or more cameras and with the second set of visual properties, upon determination that the object placement criteria are met. And generating a tactile output in conjunction with displaying. The method of any one of claims 82-88,
The field of view of the one or more cameras while displaying the representation of the virtual object with the second set of visual properties and in the second orientation corresponding to the plane within the physical environment detected within the field of view of the one or more cameras Receiving an update relating to at least the position or orientation of the plane within the physical environment detected within; And
In response to receiving the update regarding at least the position or orientation of the plane within the physical environment detected within the field of view of the one or more cameras, adjust at least the position and/or orientation of the representation of the virtual object according to the update. A method comprising the step of: The method according to any one of claims 82 to 89,
The first set of visual attributes include a first size and a first translucency level;
Wherein the second set of visual attributes comprises a second size distinct from the first size and a second translucency level lower than the first translucency level. The method according to any one of claims 82 to 90,
Displaying the virtual object in the first user interface area including at least a portion of the field of view of the one or more cameras while the virtual object is displayed in each user interface that does not include at least a portion of the field of view of the one or more cameras. The request is received,
Wherein the first orientation corresponds to an orientation of the virtual object while the virtual object is displayed within the respective user interface at the time the request is received. 91. The method of any of claims 82-90, wherein the first orientation corresponds to a predefined orientation. The method according to any one of claims 82 to 92,
While displaying the virtual object in the first user interface area in the second orientation corresponding to the plane in the physical environment detected within the field of view of the one or more cameras and with the second set of visual properties, the Detecting a request to change the simulated physical size of a virtual object from a first simulated physical size to a second simulated physical size for the physical environment captured within the field of view of the one or more cameras; And
In response to detecting the request to change the simulated physical size of the virtual object:
The displayed size of the representation of the virtual object in the first user interface area is gradually changed according to the gradual change of the simulated physical size of the virtual object from the first simulated physical size to the second simulated physical size. and;
During the gradual change of the displayed size of the representation of the virtual object in the first user interface area, according to the determination that the simulated physical size of the virtual object has reached a predefined simulated physical size, the simulation of the virtual object And generating a tactile output to indicate that the predefined physical size has reached the predefined simulated physical size. The method of claim 93,
While displaying the virtual object in a first user interface area with a second simulated physical size of the virtual object distinct from the predefined simulated physical size, returning the virtual object to the predefined simulated physical size Detecting a request to do so; And
In response to detecting a request to return the virtual object to the predefined simulated physical size, the first user according to a change of the simulated physical size of the virtual object to the predefined simulated physical size Changing the displayed size of the representation of the virtual object within an interface area. The method according to any one of claims 82 to 94,
Selecting a plane for setting a second orientation of the representation of the virtual object having the second set of visual properties according to the position and orientation of each of the one or more cameras with respect to the physical environment, the plane The steps to select are,
The virtual object having the second set of visual properties, in response to a determination that the object placement criteria have been met when the representation of the virtual object is displayed over a first portion of the physical environment captured within the field of view of the one or more cameras Selecting a first plane from among a plurality of planes detected in the physical environment within the field of view of the one or more cameras as the plane for setting a second orientation of the representation of And
The virtual object having the second set of visual properties, in accordance with a determination that the object placement criteria have been met when the representation of the virtual object is displayed over a second portion of the physical environment captured within the field of view of the one or more cameras Selecting a second plane of the plurality of planes detected in the physical environment within the field of view of the one or more cameras as the plane for setting a second orientation of the representation of the first part of the physical environment Is distinct from the second portion of the physical environment, and the first plane is distinct from the second plane. The method according to any one of claims 82 to 95,
Displaying a snapshot affordance at the same time as displaying the virtual object in the first user interface area with the second set of visual attributes and the second orientation; And
In response to activation of the snapshot affordance, a snapshot image comprising a current view of a representation of the virtual object at a certain placement location within the physical environment within the field of view of the one or more cameras is provided with the second set of visual properties. And capturing in the second orientation corresponding to the plane within the physical environment detected within the field of view of the one or more cameras. The method according to any one of claims 82 to 96,
Displaying one or more control affordances together with the representation of the virtual object having the second set of visual properties in the first user interface area;
Detecting that control-fading criteria are satisfied while displaying the one or more control unit affordances together with the representation of the virtual object having the second set of visual properties; And
In response to detecting that the controller fading criteria are satisfied, while continuing to display a representation of the virtual object having the second set of visual properties within the first user interface area including the field of view of the one or more cameras, the And stopping displaying one or more control affordances. The method according to any one of claims 82 to 97,
In response to the request to display the virtual object in the first user interface area: Before displaying the representation of the virtual object over at least a portion of the field of view of the one or more cameras included in the first user interface area, calibration ( calibration) upon determining that criteria are not met, displaying a prompt by a user to move the device to the physical environment. As a computer system,
Display creation component;
One or more input devices;
One or more cameras;
One or more processors; And
And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs,
Receiving a request to display a virtual object within a first user interface area that includes at least a portion of the field of view of the one or more cameras; And
In response to the request to display the virtual object in the first user interface area, through the display generating component, the representation of the virtual object on at least a portion of the field of view of the one or more cameras included in the first user interface area For displaying-the field of view of the one or more cameras is a view of the physical environment in which the one or more cameras are located, and displaying a representation of the virtual object,
Upon determination that object placement criteria are not met-the object placement criteria require that the placement location for the virtual object be identified within the field of view of the one or more cameras in order for the object placement criteria to be satisfied -, a first set of Displaying a representation of the virtual object with visual properties and in a first orientation independent of which portion of the physical environment is displayed within the field of view of the one or more cameras; And
According to a determination that the object placement criteria are satisfied, a second set of visual attributes distinct from the first set of visual attributes and a first corresponding to a plane in the physical environment detected within the field of view of the one or more cameras. Comprising displaying the representation of the virtual object in two orientations-comprising instructions. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display generating component, one or more input devices, and one or more cameras, cause the computer system to:
Receiving a request to display a virtual object within a first user interface area that includes at least a portion of the field of view of the one or more cameras; And
In response to the request to display the virtual object in the first user interface area, through the display generating component, the representation of the virtual object on at least a portion of the field of view of the one or more cameras included in the first user interface area -The field of view of the one or more cameras is a view of the physical environment in which the one or more cameras are located, and displaying the representation of the virtual object,
Upon determination that object placement criteria are not met-the object placement criteria require that the placement location for the virtual object be identified within the field of view of the one or more cameras in order for the object placement criteria to be satisfied -, a first set of Displaying a representation of the virtual object with visual properties and in a first orientation independent of which portion of the physical environment is displayed within the field of view of the one or more cameras; And
According to a determination that the object placement criteria are satisfied, a second set of visual attributes distinct from the first set of visual attributes and a first corresponding to a plane in the physical environment detected within the field of view of the one or more cameras. And displaying the representation of the virtual object in two orientations-containing instructions. As a computer system,
Display creation component;
One or more input devices;
One or more cameras;
Means for receiving a request to display a virtual object within a first user interface area that includes at least a portion of the field of view of the one or more cameras; And
In response to the request to display the virtual object in the first user interface area, through the display generating component, the representation of the virtual object on at least a portion of the field of view of the one or more cameras included in the first user interface area Means enabled to display a view of the one or more cameras is a view of the physical environment in which the one or more cameras are located, and the means enabled to display a representation of the virtual object,
Upon determination that object placement criteria are not met-the object placement criteria require that the placement location for the virtual object be identified within the field of view of the one or more cameras in order for the object placement criteria to be satisfied -, a first set of Means enabled to display a representation of the virtual object with visual properties and in a first orientation independent of which portion of the physical environment is displayed within the field of view of the one or more cameras; And
According to a determination that the object placement criteria are satisfied, a second set of visual attributes distinct from the first set of visual attributes and a first corresponding to a plane in the physical environment detected within the field of view of the one or more cameras. A computer system comprising means enabled to display a representation of the virtual object in two orientations. An information processing apparatus for use in a computer system having a display generating component, one or more input devices, and one or more cameras, comprising:
Means for receiving a request to display a virtual object within a first user interface area that includes at least a portion of the field of view of the one or more cameras; And
In response to the request to display the virtual object in the first user interface area, through the display generating component, the representation of the virtual object on at least a portion of the field of view of the one or more cameras included in the first user interface area Means enabled to display a view of the one or more cameras is a view of the physical environment in which the one or more cameras are located, and the means enabled to display a representation of the virtual object,
Upon determination that object placement criteria are not met-the object placement criteria require that the placement location for the virtual object be identified within the field of view of the one or more cameras in order for the object placement criteria to be satisfied -, a first set of Means enabled to display a representation of the virtual object with visual properties and in a first orientation independent of which portion of the physical environment is displayed within the field of view of the one or more cameras; And
According to a determination that the object placement criteria are satisfied, a second set of visual attributes distinct from the first set of visual attributes and a first corresponding to a plane in the physical environment detected within the field of view of the one or more cameras. And means enabled to display a representation of the virtual object in two orientations. As a computer system,
Display creation component;
One or more input devices;
One or more cameras;
One or more processors; And
A computer system comprising a memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of any one of claims 82-98. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display generating component, one or more input devices, and one or more cameras, cause the computer system to cause the A computer-readable storage medium comprising instructions for causing to perform the method of any one of claims 98 to 98. A graphical user interface on a computer system having a display generating component, one or more input devices, one or more cameras, a memory, and one or more processors for executing one or more programs stored in the memory, comprising: A graphical user interface comprising user interfaces displayed according to the method of claim 1. As a computer system,
Display creation component;
One or more input devices;
One or more cameras; And
A computer system comprising means for performing the method of any of claims 82-98. An information processing apparatus for use in a computer system having a display generating component, one or more input devices, and one or more cameras, comprising:
An information processing apparatus comprising means for performing the method of any one of claims 82-98. As a method,
A device, wherein the device has a display generating component, one or more input devices, one or more cameras, and one or more attitude sensors for detecting changes in the attitude of the device comprising the one or more cameras.
Receiving a request to display an augmented reality view of a physical environment within a first user interface area that includes a representation of the field of view of the one or more cameras;
In response to receiving the request to display an augmented reality view of the physical environment, displaying a representation of the field of view of the one or more cameras, and upon determining that calibration criteria are not met for the augmented reality view of the physical environment, Displaying a calibration user interface object that is dynamically animated according to movement of one or more cameras in the physical environment-displaying the calibration user interface object,
Detecting, through the one or more attitude sensors, a change in the attitude of one or more cameras in the physical environment while displaying the calibration user interface object; And
In response to detecting a change in posture of one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object according to the detected change in posture of one or more cameras in the physical environment Includes -;
Detecting that the calibration criteria are satisfied while displaying the calibration user interface object moving on the display according to the detected change of the posture of one or more cameras in the physical environment; And
And in response to detecting that the calibration criteria are met, stopping displaying the calibration user interface object. 111. The method of claim 108, wherein the request to display an augmented reality view of the physical environment in the first user interface area comprising a representation of the field of view of the one or more cameras is a virtual three-dimensional object in the augmented reality view of the physical environment. And a request to display an expression. The method of claim 109,
And displaying a representation of the virtual three-dimensional object within the first user interface area comprising a representation of the field of view of the one or more cameras after stopping displaying the calibration user interface object. The method of claim 109 or 110,
And simultaneously displaying a representation of the virtual 3D object in the first user interface area with the calibration user interface object, wherein the representation of the virtual 3D object is performed during movement of the one or more cameras in the physical environment. 1 method, maintained in a fixed position within the user interface area. 111. The physical environment of claim 108, wherein the request to display an augmented reality view of the physical environment within the first user interface area comprising a representation of the field of view of the one or more cameras is captured within the field of view of the one or more cameras. And a request to display a representation of the field of view of the one or more cameras without requesting display of a representation of any virtual three-dimensional object within. The method according to any one of claims 108 to 112,
In response to receiving the request to display an augmented reality view of the physical environment, displaying a representation of the field of view of the one or more cameras, and upon determining that the calibration criteria are met for an augmented reality view of the physical environment, And suspending display of the calibration user interface object. The method according to any one of claims 108 to 113,
Displaying a text object in the first user interface area at the same time as the calibration user interface object providing information on actions that can be taken by the user to improve the calibration of the augmented reality view. How to. The method according to any one of claims 108 to 114,
In response to detecting that the calibration criteria are met, displaying a visual indication of a plane detected in the physical environment that is captured within the field of view of the one or more cameras. The method according to any one of claims 108 to 115,
In response to receiving the request to display an augmented reality view of the physical environment:
In accordance with a determination that the calibration criteria are not met and prior to displaying the calibration user interface object, displaying an animated prompt object comprising a representation of the device moving relative to a representation of a plane. The method of any one of claims 108-116, wherein adjusting at least one display parameter of the calibration user interface object according to the detected change in the posture of one or more cameras in the physical environment comprises:
Moving the calibration user interface object by a first amount according to a first magnitude of movement of one or more cameras in the physical environment; And
And moving the calibration user interface object by a second amount according to a second amount of movement of one or more cameras in the physical environment, wherein the first amount is distinguished from the second amount, and the first amount of the movement The size is distinct from the second size of the movement. The method of any one of claims 108 to 117, wherein adjusting at least one display parameter of the calibration user interface object according to the detected change in the posture of one or more cameras in the physical environment,
Moving the calibration user interface object based on the first type of movement according to a determination that the detected change in the attitude of the one or more cameras corresponds to a first type of movement; And
In accordance with a determination that the detected change in the posture of the one or more cameras corresponds to the second type of movement, withholding from moving the calibration user interface object based on the second type of movement. . The method of any one of claims 108 to 118, wherein adjusting at least one display parameter of the calibration user interface object according to the detected change in the posture of one or more cameras in the physical environment,
Moving the calibration user interface object according to the detected change of the posture of one or more cameras in the physical environment without changing the characteristic display position of the calibration user interface object on the first user interface area . The method of any one of claims 108 to 119, wherein adjusting at least one display parameter of the calibration user interface object according to the detected change in the posture of one or more cameras in the physical environment,
Rotating the calibration user interface object about an axis perpendicular to a direction of movement of one or more cameras in the physical environment. The method of any one of claims 108 to 120, wherein adjusting at least one display parameter of the calibration user interface object according to the detected change in the posture of one or more cameras in the physical environment,
Moving the calibration user interface object at a rate determined in accordance with the rate of change detected within the field of view of the one or more cameras. The method of any one of claims 108 to 121, wherein adjusting at least one display parameter of the calibration user interface object according to the detected change in the posture of one or more cameras in the physical environment,
Moving the calibration user interface object in a direction determined according to a direction of change detected within the field of view of the one or more cameras. As a computer system,
Display creation component;
One or more input devices;
One or more cameras;
One or more attitude sensors;
One or more processors; And
And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs,
Receiving a request to display an augmented reality view of a physical environment within a first user interface area comprising a representation of the field of view of the one or more cameras;
In response to receiving the request to display an augmented reality view of the physical environment, displaying a representation of the field of view of the one or more cameras, and upon determining that calibration criteria are not met for the augmented reality view of the physical environment, For displaying a calibration user interface object that is dynamically animated according to the movement of one or more cameras in the physical environment-displaying the calibration user interface object,
While displaying the calibration user interface object, detecting, through the one or more attitude sensors, a change in the attitude of one or more cameras in the physical environment; And
In response to detecting a change in the attitude of one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object according to the detected change in the attitude of one or more cameras in the physical environment. Contains -;
Detecting that the calibration criteria are satisfied while displaying the calibration user interface object moving on the display according to the detected change in the posture of one or more cameras in the physical environment; And
A computer system comprising instructions for stopping displaying the calibration user interface object in response to detecting that the calibration criteria are met. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display generating component, one or more input devices, one or more cameras, and one or more pose sensors, the computer The system,
Receiving a request to display an augmented reality view of a physical environment within a first user interface area that includes a representation of the field of view of the one or more cameras;
In response to receiving the request to display an augmented reality view of the physical environment, displaying a representation of the field of view of the one or more cameras, and upon determining that calibration criteria are not met for the augmented reality view of the physical environment, Displaying a calibration user interface object that is dynamically animated according to movement of one or more cameras in the physical environment-Displaying the calibration user interface object comprises:
While displaying the calibration user interface object, detecting, through the one or more attitude sensors, a change in the attitude of one or more cameras in the physical environment; And
In response to detecting a change in the attitude of one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object according to the detected change in the attitude of one or more cameras in the physical environment. Contains -;
Detecting that the calibration criteria are satisfied while displaying the calibration user interface object moving on the display according to the detected change in the posture of one or more cameras in the physical environment; And
A computer-readable storage medium comprising instructions for causing, in response to detecting that the calibration criteria are met, to stop displaying the calibration user interface object. As a computer system,
Display creation component;
One or more input devices;
One or more cameras;
One or more attitude sensors;
Means for receiving a request to display an augmented reality view of a physical environment within a first user interface area comprising a representation of the field of view of the one or more cameras;
In response to receiving the request to display an augmented reality view of the physical environment, displaying a representation of the field of view of the one or more cameras, and upon determining that calibration criteria are not met for the augmented reality view of the physical environment, Means enabled to display a calibration user interface object that is dynamically animated according to the movement of one or more cameras in the physical environment-displaying the calibration user interface object,
While displaying the calibration user interface object, detecting, through the one or more attitude sensors, a change in the attitude of one or more cameras in the physical environment; And
In response to detecting a change in the attitude of one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object according to the detected change in the attitude of one or more cameras in the physical environment. Contains -;
Means enabled to detect that the calibration criteria are met while displaying the calibration user interface object moving on the display in response to a detected change in the attitude of one or more cameras in the physical environment; And
And in response to detecting that the calibration criteria are met, means enabled to stop displaying the calibration user interface object. An information processing apparatus for use in a computer system having a display generating component, one or more input devices, one or more cameras, and one or more attitude sensors, comprising:
Means for receiving a request to display an augmented reality view of a physical environment within a first user interface area comprising a representation of the field of view of the one or more cameras;
In response to receiving the request to display an augmented reality view of the physical environment, displaying a representation of the field of view of the one or more cameras, and upon determining that calibration criteria are not met for the augmented reality view of the physical environment, Means enabled to display a calibration user interface object that is dynamically animated according to the movement of one or more cameras in the physical environment-displaying the calibration user interface object,
While displaying the calibration user interface object, detecting, through the one or more attitude sensors, a change in the attitude of one or more cameras in the physical environment; And
In response to detecting a change in the attitude of one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object according to the detected change in the attitude of one or more cameras in the physical environment. Contains -;
Means enabled to detect that the calibration criteria are met while displaying the calibration user interface object moving on the display in response to a detected change in the attitude of one or more cameras in the physical environment; And
And, in response to detecting that the calibration criteria are met, means enabled to stop displaying the calibration user interface object. As a computer system,
Display creation component;
One or more input devices;
One or more cameras;
One or more attitude sensors;
One or more processors; And
A computer system comprising a memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of any one of claims 108-122. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display generating component, one or more input devices, one or more cameras, and one or more pose sensors, the computer A computer-readable storage medium comprising instructions that cause a system to perform the method of any one of claims 108-122. A graphical user interface on a computer system having a display generating component, one or more input devices, one or more cameras, one or more attitude sensors, a memory, and one or more processors for executing one or more programs stored in the memory, 108 A graphical user interface comprising user interfaces displayed according to the method of any one of claims 122. As a computer system,
Display creation component;
One or more input devices;
One or more cameras;
One or more attitude sensors; And
A computer system comprising means for performing the method of any of claims 108-122. An information processing apparatus for use in a computer system having a display generating component, one or more input devices, one or more cameras, and one or more attitude sensors, comprising:
An information processing apparatus comprising means for performing the method of any of claims 108-122. As a method,
In a device having a display generating component and one or more input devices comprising a touch-sensitive surface:
Displaying, by the display generating component, a representation of a first perspective of a virtual 3D object in a first user interface area;
To display a portion of the virtual 3D object that is not visible from the first viewpoint of the virtual 3D object while displaying the representation of the first viewpoint of the virtual 3D object in the first user interface area on the display Detecting a first input corresponding to a request to rotate the virtual 3D object with respect to a display; And
In response to detecting the first input:
According to a determination that the first input corresponds to a request to rotate the three-dimensional object about a first axis, determined based on the size of the first input and by a rotation above a threshold amount about the first axis Rotating the virtual 3D object about the first axis by an amount constrained by a limit on movement that limits the rotation of the virtual 3D object;
According to a determination that the first input corresponds to a request to rotate the 3D object about a second axis different from the first axis, the virtual 3D object by an amount determined based on the size of the first input Rotating about the second axis, in case of an input having a magnitude greater than each threshold, the device rotating the virtual 3D object about the second axis by a rotation exceeding the threshold amount. , Way. The method of claim 132,
In response to detecting the first input:
That the first input comprises a first movement of a contact across a touch-sensitive surface in a first direction and that the first movement of the contact in the first direction represents a representation of the virtual object with respect to the first axis. Depending on the determination that the first criteria for rotation are met-the first criteria include a requirement that the first input comprises a movement in the first direction above a first threshold amount in order for the first criteria to be satisfied -, determining that the first input corresponds to a request to rotate the 3D object about the first axis; And
That the first input comprises a second movement of the contact across the touch-sensitive surface in a second direction and that the second movement of the contact in the second direction is a representation of the virtual object with respect to the second axis. Depending on the determination of meeting second criteria for rotating-the second criteria include a requirement that the first input comprises a movement in excess of a second threshold amount in the second direction in order for the second criteria to be satisfied. And, determining that the first input corresponds to a request to rotate the three-dimensional object about the second axis, the first threshold is greater than the second threshold. The method of claim 132 or 133,
The rotation of the virtual 3D object with respect to the first axis is a first degree of correspondence between a characteristic value of the first input parameter of the first input and a rotation amount applied to the virtual 3D object centered on the first axis. occurs with a degree of correspondence;
Rotation of the virtual 3D object with respect to the second axis occurs as a second correspondence between the characteristic value of the first input parameter of the second input gesture and the amount of rotation applied to the virtual 3D object centered on the second axis. and;
Wherein the first degree of correspondence entails a smaller rotation of the virtual three-dimensional object with respect to the first input parameter than the second degree of correspondence. The method of any one of claims 132-134,
Detecting the end of the first input; And
After detecting the end of the first input, continuously rotating the 3D object based on the size of the first input before detecting the end of the input-continuously rotating the 3D object,
According to a determination that the 3D object is rotating about the first axis, the rotation of the object about the first axis is performed by a first amount proportional to the size of the rotation of the 3D object about the first axis. Slowing down; And
According to a determination that the three-dimensional object is rotating about the second axis, the second amount is proportional to the amount of rotation of the three-dimensional object with respect to the second axis, and a second amount different from the first amount is applied to the second axis. And slowing the rotation of the object with respect to. The method of any one of claims 132-135,
Detecting the end of the first input; And
After detecting the end of the first input:
Reversing at least a portion of the rotation of the three-dimensional object about the first axis according to a determination that the three-dimensional object has been rotated beyond a respective rotation threshold about the first axis;
In accordance with a determination that the three-dimensional object has not been rotated beyond the respective rotation threshold with respect to the first axis, withholding from reversing the rotation of the three-dimensional object with respect to the first axis. The method of any one of claims 132-134,
In accordance with a determination that the first input corresponds to a request to rotate the 3D object about a third axis different from the first axis and the second axis, rotating the virtual 3D object about the third axis How to withhold letting. The method of any one of claims 132-137,
Displaying a representation of a shadow cast by the virtual 3D object while displaying a representation of the first viewpoint of the virtual 3D object in the first user interface area; And
And varying the shape of the representation of the shadow according to the rotation of the virtual three-dimensional object about the first axis and/or the second axis. The method of claim 138,
While rotating the virtual 3D object within the first user interface area:
In accordance with a determination that the virtual 3D object is displayed as a second viewpoint exposing a predefined lower portion of the virtual 3D object, suspending displaying the expression of the shadow as an expression of the second viewpoint of the virtual 3D object A method comprising the steps of. The method of any one of claims 132-139,
Detecting a second input corresponding to a request to reset the virtual 3D object in the first user interface area after rotating the virtual 3D object in the first user interface area; And
In response to detecting the second input, displaying a representation of a predefined original viewpoint of the virtual three-dimensional object within the first user interface area. The method of any one of claims 132-140,
While displaying the virtual 3D object in the first user interface area, detecting a third input corresponding to a request to resize the virtual 3D object; And
In response to detecting the third input, adjusting a size of the representation of the virtual three-dimensional object in the first user interface area according to the size of the input. The method of claim 141,
While adjusting the size of the representation of the virtual 3D object in the first user interface area, detecting that the size of the virtual 3D object has reached a predefined default display size of the virtual 3D object; And
In response to detecting that the size of the virtual three-dimensional object has reached a predefined default display size of the virtual three-dimensional object, to indicate that the virtual three-dimensional object is displayed with the predefined default display size. Generating a tactile output. The method of any one of claims 132-142, wherein the device comprises one or more cameras, and the method comprises:
While displaying the representation of the third viewpoint of the virtual 3D object in the first user interface area, a first corresponding to a request to display the virtual 3D object in a second user interface area including the field of view of one or more cameras 4 detecting an input; And
In response to detecting the fourth input, displaying, via the display generating component, a representation of the virtual object over at least a portion of the field of view of the one or more cameras included in the second user interface area-the one or more The field of view of the cameras is a view of a physical environment in which the one or more cameras are located, and displaying a representation of the virtual object comprises:
Rotating the virtual 3D object at a predefined angle around the first axis; And
And maintaining a current angle of the virtual three-dimensional object with respect to the second axis. The method of any one of claims 132-143,
While displaying the representation of the fourth viewpoint of the virtual 3D object in the first user interface area, a fifth input corresponding to a request to return to the 2D user interface including the 2D representation of the virtual 3D object is provided. Detecting; And
In response to detecting the fifth input:
Rotating the virtual 3D object to show a representation of the viewpoint of the virtual 3D object corresponding to the 2D representation of the virtual 3D object;
And displaying a two-dimensional representation of the virtual three-dimensional object after the virtual three-dimensional object is rotated to show a representation of the respective viewpoint corresponding to the two-dimensional representation of the virtual three-dimensional object. The method of any one of claims 132-144,
Before displaying the representation of the first viewpoint of the virtual 3D object, displaying a user interface including a representation of the virtual 3D object including a representation of the view of the virtual 3D object from each viewpoint;
Detecting a request to display the virtual 3D object while displaying the representation of the virtual 3D object; And
In response to detecting the request to display the virtual 3D object, the display of the representation of the virtual 3D object is replaced with the virtual 3D object rotated to match each viewpoint of the representation of the virtual 3D object. A method comprising the step of: The method of any one of claims 132-145,
Before displaying the first user interface, displaying a 2D user interface including a 2D representation of the virtual 3D object;
While displaying the 2D user interface including the 2D representation of the virtual 3D object, a touch input that satisfies preview criteria at a location on the touch-sensitive surface corresponding to the 2D representation of the virtual 3D object Detecting a first portion of; And
In response to detecting the first portion of the touch input that satisfies the preview criteria, displaying a preview of the virtual three-dimensional object that is larger than a two-dimensional representation of the virtual three-dimensional object. The method of claim 146,
Detecting a second portion of the touch input while displaying the preview of the virtual 3D object; And
In response to detecting the second portion of the touch input:
In accordance with a determination that the second portion of the touch input meets menu display criteria, displaying a plurality of selectable options corresponding to a plurality of operations associated with the virtual object;
The first user interface including the virtual 3D object to display the 2D user interface including the 2D representation of the virtual 3D object according to a determination that the second portion of the touch input satisfies the staging criteria Replacing with. The method of any one of claims 132-147, wherein the first user interface includes a plurality of control units, and the method comprises:
Before displaying the first user interface, displaying a 2D user interface including a 2D representation of the virtual 3D object; And
In response to detecting a request to display the virtual three-dimensional object within the first user interface:
Displaying the virtual 3D object in the first user interface without displaying a set of one or more control units associated with the virtual 3D object;
And after displaying the virtual three-dimensional object in the first user interface, displaying the set of one or more control units. As a computer system,
Display creation component;
One or more input devices including a touch-sensitive surface;
One or more processors; And
And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs,
Displaying, by the display generating component, a representation of a first viewpoint of a virtual three-dimensional object in a first user interface area;
To display a portion of the virtual 3D object that is not visible from the first viewpoint of the virtual 3D object while displaying the representation of the first viewpoint of the virtual 3D object in the first user interface area on the display Detecting a first input corresponding to a request to rotate the virtual 3D object with respect to a display; And
In response to detecting the first input:
According to a determination that the first input corresponds to a request to rotate the three-dimensional object about a first axis, determined based on the size of the first input and by a rotation above a threshold amount about the first axis Rotating the virtual 3D object about the first axis by an amount constrained by a limit on movement that limits the rotation of the virtual 3D object;
According to a determination that the first input corresponds to a request to rotate the 3D object about a second axis different from the first axis, the virtual 3D object by an amount determined based on the size of the first input Instructions for rotating about the second axis-for inputs having a magnitude greater than each threshold, the device rotating the virtual 3D object about the second axis by a rotation greater than the threshold amount- That, the computer system. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having one or more input devices including a display generating component, and a touch-sensitive surface, cause the computer system to:
Causing, by the display generating component, to display a representation of a first viewpoint of a virtual 3D object in a first user interface area;
To display a portion of the virtual 3D object that is not visible from the first viewpoint of the virtual 3D object while displaying the representation of the first viewpoint of the virtual 3D object in the first user interface area on the display Detecting a first input corresponding to a request to rotate the virtual three-dimensional object with respect to a display; And
In response to detecting the first input:
According to a determination that the first input corresponds to a request to rotate the three-dimensional object about a first axis, determined based on the size of the first input and by a rotation above a threshold amount about the first axis Rotating the virtual 3D object about the first axis by an amount constrained by a limit on movement that limits the rotation of the virtual 3D object;
According to a determination that the first input corresponds to a request to rotate the 3D object about a second axis different from the first axis, the virtual 3D object by an amount determined based on the size of the first input -In the case of an input having a magnitude greater than each threshold, the device rotates the virtual 3D object about the second axis by a rotation greater than the threshold amount-to rotate about the second axis A computer-readable storage medium. As a computer system,
Display creation component;
One or more input devices including a touch-sensitive surface;
Means for displaying, by the display generating component, a representation of a first viewpoint of a virtual three-dimensional object in a first user interface area;
To display a portion of the virtual 3D object that is not visible from the first viewpoint of the virtual 3D object while displaying the representation of the first viewpoint of the virtual 3D object in the first user interface area on the display Means enabled to detect a first input corresponding to a request to rotate the virtual three-dimensional object relative to a display; And
In response to detecting the first input:
According to a determination that the first input corresponds to a request to rotate the three-dimensional object about a first axis, determined based on the size of the first input and by a rotation above a threshold amount about the first axis Rotating the virtual 3D object about the first axis by an amount constrained by a limit on movement that limits the rotation of the virtual 3D object;
According to a determination that the first input corresponds to a request to rotate the 3D object about a second axis different from the first axis, the virtual 3D object by an amount determined based on the size of the first input Means enabled to rotate about the second axis-for inputs having a magnitude above each threshold, the device rotates the virtual 3D object about the second axis by a rotation above the threshold amount. -Comprising, computer system. An information processing apparatus for use in a computer system having one or more input devices comprising a display generating component and a touch-sensitive surface, comprising:
Means for displaying, by the display generating component, a representation of a first viewpoint of a virtual three-dimensional object in a first user interface area;
To display a portion of the virtual 3D object that is not visible from the first viewpoint of the virtual 3D object while displaying the representation of the first viewpoint of the virtual 3D object in the first user interface area on the display Means enabled to detect a first input corresponding to a request to rotate the virtual three-dimensional object relative to a display; And
In response to detecting the first input:
According to a determination that the first input corresponds to a request to rotate the three-dimensional object about a first axis, determined based on the size of the first input and by a rotation above a threshold amount about the first axis Rotating the virtual 3D object about the first axis by an amount constrained by a limit on movement that limits the rotation of the virtual 3D object;
According to a determination that the first input corresponds to a request to rotate the 3D object about a second axis different from the first axis, the virtual 3D object by an amount determined based on the size of the first input Means enabled to rotate about the second axis-for inputs having a magnitude above each threshold, the device rotates the virtual 3D object about the second axis by a rotation above the threshold amount. -Including, information processing apparatus. As a computer system,
Display creation component;
One or more input devices including a touch-sensitive surface;
One or more processors; And
A computer system comprising a memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of any one of claims 132-148. A computer-readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having one or more input devices including a display generating component, and a touch-sensitive surface, cause the computer system to A computer readable storage medium comprising instructions for causing to perform the method of any of claims 132-148. A graphical user interface on a computer system having a display generating component, one or more input devices including a touch-sensitive surface, a memory, and one or more processors for executing one or more programs stored in the memory, comprising: A graphical user interface comprising user interfaces displayed according to the method of claim 1. As a computer system,
Display creation component;
One or more input devices including a touch-sensitive surface; And
A computer system comprising means for performing the method of any of claims 132-148. An information processing apparatus for use in a computer system having a display generating component, one or more input devices comprising a touch-sensitive surface, and one or more cameras, comprising:
An information processing apparatus comprising means for performing the method of any of claims 132-148. As a method,
In a device with a display generating component and a touch-sensitive surface:
A first object manipulation behavior performed in response to inputs satisfying first gesture recognition criteria and a second object manipulation behavior performed in response to inputs meeting second gesture recognition criteria, through the display generation component Displaying a first user interface area including a user interface object associated with a plurality of object manipulation behaviors;
Detecting the movement of one or more contacts across the touch-sensitive surface while displaying the first user interface area, detecting a first portion of the input to the user interface object, and detecting the one or more contacts Evaluating movement of the one or more contacts relative to both the first gesture recognition criteria and the second gesture recognition criteria while they are detected on the touch-sensitive surface; And
In response to detecting the first portion of the input, updating the appearance of the user interface object based on the first portion of the input-updating the appearance of the user interface object,
According to a determination that the first portion of the input meets the first gesture recognition criteria prior to meeting the second gesture recognition criteria:
Changing an appearance of the user interface object according to the first object manipulation behavior based on the first portion of the input;
Updating the second gesture recognition criteria by increasing a threshold for the second gesture recognition criteria; And
According to a determination that the input meets the second gesture recognition criteria prior to meeting the first gesture recognition criteria:
Changing an appearance of the user interface object according to the second object manipulation behavior based on the first portion of the input;
And updating the first gesture recognition criteria by increasing a threshold for the first gesture recognition criteria. The method of claim 158,
Updating an appearance of the user interface object based on the first portion of the input, and then detecting a second portion of the input; And
In response to detecting the second portion of the input, updating the appearance of the user interface object based on the second portion of the input-updating the appearance of the user interface object,
According to a determination that the first portion of the input meets the first gesture recognition criteria and the second portion of the input does not meet the updated second gesture recognition criteria, the user interface according to the second object manipulation behavior Changing the appearance of the user interface object according to the first object manipulation behavior based on a second portion of the input without changing the appearance of the object; And
According to a determination that the first portion of the input meets the second gesture recognition criteria and the second portion of the input does not meet the updated first gesture recognition criteria: the user interface according to the first object manipulation behavior And changing the appearance of the user interface object according to the second object manipulation behavior based on the second portion of the input without changing the appearance of the object. The method of claim 159, wherein after the first portion of the input satisfies the first gesture recognition criteria, while the appearance of the user interface object is changed according to the first object manipulation behavior based on the second portion of the input. And the second portion of the input includes an input that meets the second gesture recognition criteria before the second gesture recognition criteria are updated. The appearance of the user interface object according to claim 159 or 160, wherein after the first portion of the input satisfies the second gesture recognition criteria, based on the second portion of the input, the second object manipulation behavior During this change, the second portion of the input comprises an input that satisfies the first gesture recognition criteria before the first gesture recognition criteria are updated. The method of claim 159, wherein after the first portion of the input satisfies the first gesture recognition criteria, while the appearance of the user interface object is changed according to the first object manipulation behavior based on the second portion of the input. And the second portion of the input does not include an input that meets the first gesture recognition criteria. The user of any one of claims 159 or 162, wherein the user according to the second object manipulation behavior based on the second portion of the input after the first portion of the input meets the second gesture recognition criteria. While the appearance of an interface object is changed, the second portion of the input does not include an input that meets the second gesture recognition criteria. The method of claim 159, wherein updating the appearance of the user interface object based on the second portion of the input comprises:
According to a determination that the first portion of the input meets the second gesture recognition criteria and the second portion of the input meets the updated first gesture recognition criteria:
Changing the appearance of the user interface object according to the first object manipulation behavior based on the second portion of the input;
Changing an appearance of the user interface object according to the second object manipulation behavior based on the second portion of the input; And
According to a determination that the first portion of the input meets the first gesture recognition criteria and the second portion of the input meets the updated second gesture recognition criteria:
Changing the appearance of the user interface object according to the first object manipulation behavior based on the second portion of the input;
And changing the appearance of the user interface object according to the second object manipulation behavior based on the second portion of the input. The method of claim 164,
Detecting a third portion of the input after updating an appearance of the user interface object based on the second portion of the input; And
In response to detecting the third portion of the input, updating the appearance of the user interface object based on the third portion of the input-updating the appearance of the user interface object,
Changing an appearance of the user interface object according to the first object manipulation behavior based on the third portion of the input; And
And changing the appearance of the user interface object according to the second object manipulation behavior based on the third portion of the input. 165. The method of claim 165, wherein the third portion of the input does not include an input that satisfies the first gesture recognition criteria or an input that satisfies the second gesture recognition criteria. The method of any one of claims 164-166,
The plurality of object manipulation behaviors includes a third object manipulation behavior performed in response to inputs meeting third gesture recognition criteria;
Updating the appearance of the user interface object based on the first portion of the input,
According to a determination that the first portion of the input satisfies the first gesture recognition criteria before meeting the second gesture recognition criteria or before meeting the third gesture recognition criteria:
Changing an appearance of the user interface object according to the first object manipulation behavior based on the first portion of the input;
Updating the second gesture recognition criteria by increasing the threshold for the second gesture recognition criteria;
Updating the third gesture recognition criteria by increasing a threshold for the third gesture recognition criteria;
Upon determining that the input satisfies the second gesture recognition criteria prior to meeting the first gesture recognition criteria or the third gesture recognition criteria:
Changing an appearance of the user interface object according to the second object manipulation behavior based on the first portion of the input;
Updating the first gesture recognition criteria by increasing a threshold for the first gesture recognition criteria;
Updating the third gesture recognition criteria by increasing a threshold for the third gesture recognition criteria; And
According to a determination that the input satisfies the third gesture recognition criteria prior to meeting the first gesture recognition criteria or meeting the second gesture recognition criteria:
Changing an appearance of the user interface object according to the third object manipulation behavior based on the first portion of the input;
Updating the first gesture recognition criteria by increasing a threshold for the first gesture recognition criteria;
And updating the second gesture recognition criteria by increasing the threshold for the second gesture recognition criteria. The method of any one of claims 164-166,
The plurality of object manipulation behaviors include a third object manipulation behavior performed in response to inputs meeting third gesture recognition criteria,
The first portion of the input did not meet the third gesture recognition criteria before meeting the first gesture recognition criteria or the second gesture recognition criteria,
The device updated the third gesture recognition criteria by increasing a threshold for the third gesture recognition criteria after the first portion of the input meets the first gesture recognition criteria or the second gesture recognition criteria. ,
The second portion of the input did not meet the updated third gesture recognition criteria prior to meeting the updated first gesture recognition criteria or the updated second gesture recognition criteria;
The above method,
In response to detecting the third portion of the input:
According to a determination that the third portion of the input satisfies the updated third gesture recognition criteria, change the appearance of the user interface object according to the third object manipulation behavior based on the third portion of the input;
In accordance with a determination that the third portion of the input does not meet the updated third gesture recognition criteria, changing the appearance of the user interface object according to the third object manipulation behavior based on the third portion of the input A method comprising the step of withholding. The method of any one of claims 164-168, wherein the third portion of the input has satisfied the updated third gesture recognition criteria, and the method further comprises:
Detecting a fourth portion of the input after updating the appearance of the user interface object based on the third portion of the input; And
In response to detecting the fourth portion of the input, updating the appearance of the user interface object based on the fourth portion of the input-updating the appearance of the user interface object,
Changing an appearance of the user interface object according to the first object manipulation behavior based on a fourth portion of the input;
Changing an appearance of the user interface object according to the second object manipulation behavior based on the fourth portion of the input; And
And changing the appearance of the user interface object according to the third object manipulation behavior based on the fourth portion of the input. The method of claim 169, wherein the fourth portion of the input comprises:
An input that satisfies the first gesture recognition criteria,
An input that satisfies the second gesture recognition criteria, or
And does not include an input that meets the third gesture recognition criteria. 170. The method of any of claims 158-170, wherein both the first gesture recognition criteria and the second gesture recognition criteria require a first number of simultaneously detected contacts to be satisfied. The method according to any one of claims 158 to 171, wherein the first object manipulation behavior changes a zoom level or a displayed size of the user interface object, and the second object manipulation behavior is the user interface object. How to change the rotation angle of. The method according to any one of claims 158 to 171, wherein the first object manipulation behavior changes a zoom level or a displayed size of the user interface object, and the second object manipulation behavior is within the first user interface area. Repositioning the user interface object. The method according to any one of claims 158 to 171, wherein the first object manipulation behavior changes a position of the user interface object in the first user interface area, and the second object manipulation behavior is of the user interface object. How to change the rotation angle. The method of any one of claims 158-174, wherein the first portion of the input and the second portion of the input are provided by a plurality of continuously held contacts, the method comprising:
Re-establishing the first gesture recognition criteria and the second gesture recognition criteria to initiate additional first and second object manipulation behaviors after detecting the liftoff of the plurality of continuously held contacts. How to. The method of any one of claims 158 to 175, wherein the first gesture recognition criteria correspond to rotation about a first axis, and the second gesture recognition criteria are centered on a second axis orthogonal to the first axis. Corresponding to the rotation made by, how. As a computer system,
Display creation component;
Touch-sensitive surface;
One or more processors; And
And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs,
A first object manipulation behavior performed in response to inputs satisfying first gesture recognition criteria and a second object manipulation behavior performed in response to inputs meeting second gesture recognition criteria, through the display generation component Displaying a first user interface area including a user interface object associated with a plurality of object manipulation behaviors;
Detecting the movement of one or more contacts across the touch-sensitive surface while displaying the first user interface area, detecting a first portion of the input to the user interface object, and detecting the one or more contacts Evaluating the movement of the one or more contacts relative to both the first gesture recognition criteria and the second gesture recognition criteria while they are detected on the touch-sensitive surface; And
In response to detecting the first portion of the input, for updating the appearance of the user interface object based on the first portion of the input-updating the appearance of the user interface object,
According to a determination that the first portion of the input meets the first gesture recognition criteria prior to meeting the second gesture recognition criteria:
Changing an appearance of the user interface object according to the first object manipulation behavior based on the first portion of the input;
Updating the second gesture recognition criteria by increasing a threshold for the second gesture recognition criteria; And
According to a determination that the input meets the second gesture recognition criteria prior to meeting the first gesture recognition criteria:
Changing an appearance of the user interface object according to the second object manipulation behavior based on the first portion of the input;
Updating the first gesture recognition criteria by increasing a threshold for the first gesture recognition criteria-comprising instructions. A computer-readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display generating component and a touch-sensitive surface, cause the computer system to:
A first object manipulation behavior performed in response to inputs satisfying first gesture recognition criteria and a second object manipulation behavior performed in response to inputs meeting second gesture recognition criteria, through the display generation component Displaying a first user interface area including a user interface object associated with a plurality of object manipulation behaviors;
Detecting the movement of one or more contacts across the touch-sensitive surface while displaying the first user interface area, detecting a first portion of the input to the user interface object, and detecting the one or more contacts Evaluating movement of the one or more contacts relative to both the first gesture recognition criteria and the second gesture recognition criteria while they are detected on the touch-sensitive surface; And
In response to detecting the first portion of the input, causing updating the appearance of the user interface object based on the first portion of the input-updating the appearance of the user interface object,
According to a determination that the first portion of the input meets the first gesture recognition criteria prior to meeting the second gesture recognition criteria:
Changing an appearance of the user interface object according to the first object manipulation behavior based on the first portion of the input;
Updating the second gesture recognition criteria by increasing a threshold for the second gesture recognition criteria; And
According to a determination that the input meets the second gesture recognition criteria prior to meeting the first gesture recognition criteria:
Changing an appearance of the user interface object according to the second object manipulation behavior based on the first portion of the input;
Updating the first gesture recognition criteria by increasing a threshold for the first gesture recognition criteria, comprising instructions. As a computer system,
Display creation component;
Touch-sensitive surface;
A first object manipulation behavior performed in response to inputs satisfying first gesture recognition criteria and a second object manipulation behavior performed in response to inputs meeting second gesture recognition criteria, through the display generation component Means for displaying a first user interface area comprising a user interface object associated with a plurality of object manipulation behaviors;
Detecting the movement of one or more contacts across the touch-sensitive surface while displaying the first user interface area, detecting a first portion of the input to the user interface object, and detecting the one or more contacts Means enabled to evaluate movement of the one or more contacts relative to both the first gesture recognition criteria and the second gesture recognition criteria while they are detected on the touch-sensitive surface; And
In response to detecting the first portion of the input, means enabled to update the appearance of the user interface object based on the first portion of the input-updating the appearance of the user interface object,
According to a determination that the first portion of the input meets the first gesture recognition criteria prior to meeting the second gesture recognition criteria:
Changing an appearance of the user interface object according to the first object manipulation behavior based on the first portion of the input;
Updating the second gesture recognition criteria by increasing a threshold for the second gesture recognition criteria; And
According to a determination that the input meets the second gesture recognition criteria prior to meeting the first gesture recognition criteria:
Changing an appearance of the user interface object according to the second object manipulation behavior based on the first portion of the input;
And updating the first gesture recognition criteria by increasing a threshold for the first gesture recognition criteria. An information processing apparatus for use in a computer system having a display generating component and a touch-sensitive surface, comprising:
A first object manipulation behavior performed in response to inputs satisfying first gesture recognition criteria and a second object manipulation behavior performed in response to inputs meeting second gesture recognition criteria, through the display generation component Means for displaying a first user interface area comprising a user interface object associated with a plurality of object manipulation behaviors;
Detecting the movement of one or more contacts across the touch-sensitive surface while displaying the first user interface area, detecting a first portion of the input to the user interface object, and detecting the one or more contacts Means enabled to evaluate movement of the one or more contacts relative to both the first gesture recognition criteria and the second gesture recognition criteria while they are detected on the touch-sensitive surface; And
In response to detecting the first portion of the input, means enabled to update the appearance of the user interface object based on the first portion of the input-updating the appearance of the user interface object,
According to a determination that the first portion of the input meets the first gesture recognition criteria prior to meeting the second gesture recognition criteria:
Changing an appearance of the user interface object according to the first object manipulation behavior based on the first portion of the input;
Updating the second gesture recognition criteria by increasing a threshold for the second gesture recognition criteria; And
According to a determination that the input meets the second gesture recognition criteria prior to meeting the first gesture recognition criteria:
Changing an appearance of the user interface object according to the second object manipulation behavior based on the first portion of the input;
And updating the first gesture recognition criteria by increasing a threshold for the first gesture recognition criteria. As a computer system,
Display creation component;
Touch-sensitive surface;
One or more processors; And
A computer system comprising a memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of any one of claims 158-176. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display generating component and a touch-sensitive surface, cause the computer system to cause any of claims 158 to 176. A computer-readable storage medium containing instructions for causing to perform the method of claim. A graphical user interface on a computer system having a display generating component, one or more input devices, one or more cameras, a memory, and one or more processors for executing one or more programs stored in the memory, the graphical user interface of claims 158-176 A graphical user interface comprising user interfaces displayed according to the method of claim 1. As a computer system,
Display creation component;
Touch-sensitive surface; And
A computer system comprising means for performing the method of any of claims 158-176. An information processing apparatus for use in a computer system having a display generating component and a touch-sensitive surface, comprising:
An information processing apparatus comprising means for performing the method of any of claims 158-176. As a method,
In a device with a display generating component, one or more input devices, one or more audio output generators, and one or more cameras:
Displaying, via the display generating component, a representation of a virtual object in a first user interface area that includes a representation of the field of view of one or more cameras, wherein the displaying comprises in a physical environment captured within the field of view of the one or more cameras. Maintaining a first spatial relationship between the detected plane and the representation of the virtual object;
Detecting movement of the device to adjust the field of view of the one or more cameras; And
In response to detecting movement of the device to adjust the field of view of the one or more cameras:
Display of the representation of the virtual object in the first user interface area according to the first spatial relationship between the plane and the virtual object detected in the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted Adjust,
In accordance with a determination that movement of the device causes the virtual object above a threshold amount to move out of the displayed portion of the field of view of the one or more cameras, generating, through the one or more audio output generators, a first audio alert. Containing, method. 189. The method of claim 186, wherein outputting the first audio alert comprises generating an audio output indicative of an amount of the virtual object remaining visible on a displayed portion of the field of view of the one or more cameras. 187. The method of claim 186 or 187, wherein outputting the first audio alert comprises generating an audio output indicative of an amount of a displayed portion of the field of view obscured by the virtual object. The method of any one of claims 186 to 188,
Detecting an input by contact at a location on the touch-sensitive surface corresponding to a representation of the field of view of the one or more cameras; And
In response to detecting the input, and upon determining that the input is detected at a first location on the touch-sensitive surface corresponding to a first portion of the field of view of the one or more cameras not occupied by the virtual object, Generating a second audio alert. The method according to any one of claims 186 to 189, wherein the outputting of the first audio alert generates an audio output indicating an operation performed on the virtual object and a result state of the virtual object after the operation is performed. A method comprising the step of: The method of claim 190, wherein the resultant state of the virtual object after the operation is performed is described as the audio output in the first audio alert in relation to a reference frame corresponding to the physical environment captured within the field of view of the one or more cameras. Being, how. The method of any one of claims 186 to 191,
Detecting a further movement of the device to further adjust the field of view of the one or more cameras after generation of the first audio alert; And
In response to detecting further movement of the device to further adjust the field of view of the one or more cameras:
As the field of view of the one or more cameras is further adjusted, the representation of the virtual object in the first user interface area according to the first spatial relationship between the plane and the virtual object detected in the field of view of the one or more cameras Adjust the display,
In accordance with a determination that further movement of the device causes the virtual object above a second threshold amount to move into a displayed portion of the field of view of the one or more cameras, generating a third audio alert, via the one or more audio output generators. A method comprising the steps of. The method of any one of claims 186 to 192,
Display the representation of the virtual object in the first user interface area, and apply to the virtual object while a first object manipulation type among a plurality of object manipulation types applicable to the virtual object is currently selected for the virtual object Detecting a request to switch to another possible object manipulation type; And
In response to detecting the request to switch to another object manipulation type applicable to the virtual object, generating an audio output naming a second object manipulation type of a plurality of object manipulation types applicable to the virtual object- And the second object manipulation type is distinct from the first object manipulation type. The method of claim 193,
Generating the audio output naming the second object manipulation type among the plurality of object manipulation types applicable to the virtual object, and then detecting a request to execute an object manipulation behavior corresponding to the currently selected object manipulation type ; And
And in response to detecting the request to perform the object manipulation behavior corresponding to the currently selected object manipulation type, executing an object manipulation behavior corresponding to the second object manipulation type. The method of claim 193 or 194,
In response to detecting the request to switch to another object manipulation type applicable to the virtual object:
According to the determination that the second object manipulation type is a continuously adjustable manipulation type, to indicate that the second object manipulation type is a continuously adjustable manipulation type, the audio output naming the second object manipulation type and Creating an audio alert together;
And detecting a swipe input at a location on the touch-sensitive surface corresponding to a portion of the first user interface area displaying a representation of the field of view of the one or more cameras, the second object manipulation type Detecting a request to execute the corresponding object manipulation behavior; And
In response to detecting the request to execute the object manipulation behavior corresponding to the second object manipulation type, the object manipulation behavior corresponding to the second object manipulation type is determined by an amount corresponding to the size of the swipe input. A method comprising the step of executing. The method of any one of claims 186 to 195,
Before displaying the representation of the virtual object in the first user interface area, displaying the representation of the virtual object in a second user interface area, the second user interface area including a representation of the field of view of the one or more cameras Do not -;
Displaying the representation of the virtual object in the second user interface area, and while a first operation among a plurality of operations applicable to the virtual object is currently selected for the virtual object, another operation applicable to the virtual object is performed. Detecting a request to switch; And
In response to detecting the request to switch to another operation applicable to the virtual object in the second user interface area, generating an audio output naming a second operation of the plurality of operations applicable to the virtual object Step-the second operation is distinct from the first operation- The method of any one of claims 186 to 196,
Before displaying the representation of the virtual object in the first user interface area: While displaying the representation of the virtual object in a second user interface area that does not contain a representation of the field of view of the one or more cameras, Detecting a request to display a representation of the virtual object within the first user interface area including a representation of the field of view; And
In response to detecting the request to display a representation of the virtual object within the first user interface region comprising a representation of the field of view of the one or more cameras:
Displaying the representation of the virtual object in the first user interface area according to the first spatial relationship between the representation of the virtual object and the plane detected in the physical environment captured within the field of view of the one or more cameras;
Generating a fourth audio alert indicating that the virtual object is placed within the augmented reality view in relation to the physical environment being captured within the field of view of the one or more cameras. 197. The method of claim 197, wherein the third audio alert represents information regarding the appearance of the virtual object for a portion of the field of view of the one or more cameras. The method of claim 198,
And generating a tactile output with the placement of the virtual object in the augmented reality view in relation to the physical environment captured within the field of view of the one or more cameras. The method of any one of claims 186 to 198,
Simultaneously displaying a field of view of the one or more cameras, displaying a first control unit at a first position in the first user interface area;
In response to a determination that the controller fading criteria are satisfied, stopping displaying the first controller in the first user interface area while maintaining a display of the representation of the field of view of the one or more cameras in the first user interface area;
Each position on the touch-sensitive surface corresponding to the first position in the first user interface area while displaying the first user interface area without displaying the first control unit in the first user interface area Detecting a touch input at; And
And in response to detecting the touch input, generating a fifth audio alert including an audio output specifying an operation corresponding to the first control unit. As a computer system,
Display creation component;
One or more input devices;
One or more audio output generators;
One or more cameras;
One or more processors; And
And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs,
For displaying, via the display generation component, a representation of a virtual object within a first user interface area comprising a representation of the field of view of one or more cameras, the displaying being detected in a physical environment captured within the field of view of the one or more cameras Including maintaining a first spatial relationship between the plane being created and the representation of the virtual object;
For detecting movement of the device to adjust the field of view of the one or more cameras; And
In response to detecting movement of the device to adjust the field of view of the one or more cameras:
Display of the representation of the virtual object in the first user interface area according to the first spatial relationship between the plane and the virtual object detected in the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted Adjust,
In accordance with a determination that movement of the device causes the virtual object above a threshold amount to move out of the displayed portion of the field of view of the one or more cameras, through the one or more audio output generators, for generating a first audio alert A computer system comprising instructions. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display generating component, one or more input devices, one or more audio output generators, and one or more cameras, The computer system,
Causing, via the display generating component, to display a representation of a virtual object within a first user interface area containing a representation of the field of view of one or more cameras, wherein the displaying is detected in a physical environment captured within the field of view of the one or more cameras. Including maintaining a first spatial relationship between the plane being created and the representation of the virtual object;
To detect movement of the device to adjust the field of view of the one or more cameras; And
In response to detecting movement of the device to adjust the field of view of the one or more cameras:
Display of the representation of the virtual object in the first user interface area according to the first spatial relationship between the plane and the virtual object detected in the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted Make adjustments,
In accordance with a determination that movement of the device causes the virtual object above a threshold amount to move out of the displayed portion of the field of view of the one or more cameras, causing, through the one or more audio output generators, to generate a first audio alert. A computer-readable storage medium containing instructions. As a computer system,
Display creation component;
One or more input devices;
One or more audio output generators;
One or more cameras;
Means for displaying, via the display generating component, a representation of a virtual object within a first user interface area comprising a representation of the field of view of one or more cameras, wherein the displaying is in a physical environment captured within the field of view of the one or more cameras Including maintaining a first spatial relationship between the detected plane and the representation of the virtual object;
Means for detecting movement of the device for adjusting the field of view of the one or more cameras; And
In response to detecting movement of the device to adjust the field of view of the one or more cameras:
Display of the representation of the virtual object in the first user interface area according to the first spatial relationship between the plane and the virtual object detected in the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted Adjust,
In accordance with a determination that movement of the device causes the virtual object above a threshold amount to move out of the displayed portion of the field of view of the one or more cameras, through the one or more audio output generators, to generate a first audio alert A computer system comprising means to be enabled. An information processing apparatus for use in a computer system having a display generating component, one or more input devices, one or more audio output generators, and one or more cameras, comprising:
Means for displaying, via the display generating component, a representation of a virtual object within a first user interface area comprising a representation of the field of view of one or more cameras, wherein the displaying is in a physical environment captured within the field of view of the one or more cameras Including maintaining a first spatial relationship between the detected plane and the representation of the virtual object;
Means for detecting movement of the device for adjusting the field of view of the one or more cameras; And
In response to detecting movement of the device to adjust the field of view of the one or more cameras:
Display of the representation of the virtual object in the first user interface area according to the first spatial relationship between the plane and the virtual object detected in the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted Adjust,
In accordance with a determination that movement of the device causes the virtual object above a threshold amount to move out of the displayed portion of the field of view of the one or more cameras, through the one or more audio output generators, to generate a first audio alert An information processing apparatus comprising means to be enabled. As a computer system,
Display creation component;
One or more input devices;
One or more cameras;
One or more audio output generators;
One or more processors; And
A computer system comprising a memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of any one of claims 186 to 200. A computer readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system having a display generating component, one or more input devices, one or more audio output generators, and one or more cameras, A computer-readable storage medium comprising instructions that cause a computer system to perform the method of any of claims 186-200. A graphical user interface on a computer system having a display generating component, one or more input devices, one or more audio output generators, one or more cameras, a memory, and one or more processors for executing one or more programs stored in the memory, comprising: A graphical user interface comprising user interfaces displayed according to the method of any of claims 186-200. As a computer system,
Display creation component;
One or more input devices;
One or more audio output generators;
One or more cameras; And
A computer system comprising means for performing the method of any of claims 186-200. An information processing apparatus for use in a computer system having a display generating component, one or more input devices, one or more audio output generators, and one or more cameras, comprising:
An information processing apparatus comprising means for performing the method of any of claims 186-200.

Images