JP2020178341A - Electronic apparatus - Google Patents

Abstract

To provide an electronic apparatus improved in visibility of an image.SOLUTION: An electronic apparatus 1 comprises: an imaging unit 20 that can capture a first area at a first frame rate, and capture a second area at a second frame rate higher than the first frame rate; and a display control unit 70 that causes a first display unit 51 to display at least images captured in the first area and the second area, and causes a second display unit 53 to display the image captured in the second area.SELECTED DRAWING: Figure 5

Description

The present invention relates to an electronic device including an image sensor.

An electronic device including an image pickup device in which a back-illuminated image pickup chip and a signal processing chip are laminated (hereinafter referred to as a stacked image pickup device) has been proposed (see Patent Document 1). In the stacked image sensor, the back-illuminated image sensor and the signal processing chip are laminated so as to be connected to each predetermined region via micro bumps.

Japanese Unexamined Patent Publication No. 2006-49361

However, in an electronic device equipped with a conventional stacked image sensor, there are not many proposals to divide an image into blocks having one or more of the above regions and acquire an image captured for each block, and the stacked image sensor is used. The usability of the electronic devices provided was not sufficient.

The electronic device according to one aspect of the invention is connected to a plurality of pixels arranged in the row direction and the column direction, and the first pixel of the plurality of pixels, and is a first control for controlling the first pixel. A second control signal connected to a first control line from which a signal is output and a second pixel arranged on the row direction side from the first pixel among the plurality of pixels to control the second pixel. Is generated based on an image pickup element including a second control line from which is output, a first image generated based on the first signal from the first pixel, and a second signal from the second pixel. A second image and a control unit for displaying the second image on the display unit are provided.

According to the present invention, it is possible to realize an electronic device having improved image visibility.

It is sectional drawing of the laminated type image sensor. It is a figure explaining the pixel arrangement and the unit area of an image pickup chip. It is a circuit diagram corresponding to the unit area of an image pickup chip. It is a block diagram which shows the functional structure of an image sensor. It is a block diagram which illustrates the structure of the image pickup apparatus by one Embodiment. It is external drawing explaining the 1st display part and the 2nd display part provided in the image pickup apparatus. FIG. 7A is a diagram illustrating a live view image before determining the region of interest and the peripheral region. FIG. 7B is a diagram illustrating a region of interest determined by the control unit by the human body detection process based on the live view image. FIG. 8 (a) is a diagram showing an image acquired corresponding to the peripheral region, FIG. 8 (b) is a diagram showing an image acquired corresponding to the region of interest, and FIG. 8 (c) is a composite. It is a figure which illustrates the 1st live view image to be done. It is a flowchart explaining the flow of the photographing operation executed by the control part of an image pickup apparatus. 10 (a) and 10 (b) are diagrams illustrating a scene in which a plurality of main subjects exist. It is external drawing explaining the 1st display part and the 2nd display part provided in the image pickup apparatus.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
<Explanation of stacked image sensor>
First, a stacked image sensor 100 mounted on an electronic device (for example, an image pickup device 1) according to an embodiment of the present invention will be described. The stacked image sensor 100 is described in Japanese Patent Application No. 2012-139026 previously filed by the applicant of the present application. FIG. 1 is a cross-sectional view of the stacked image sensor 100. The image sensor 100 includes a back-illuminated image pickup chip 113 that outputs a pixel signal corresponding to incident light, a signal processing chip 111 that processes the pixel signal, and a memory chip 112 that stores the pixel signal. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are laminated and electrically connected to each other by a conductive bump 109 such as Cu.

As shown in the figure, the incident light is mainly incident in the Z-axis plus direction indicated by the white arrow. In the present embodiment, in the image pickup chip 113, the surface on the side where the incident light is incident is referred to as the back surface. Further, as shown in the coordinate axes, the left direction of the paper surface orthogonal to the Z axis is defined as the X-axis plus direction, and the Z-axis and the front direction of the paper surface orthogonal to the X-axis are defined as the Y-axis plus direction. In some subsequent figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes of FIG.

An example of the image pickup chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back surface side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and accumulating charges according to incident light, and a transistor 105 provided corresponding to the PD 104.

A color filter 102 is provided on the incident side of the incident light in the PD layer 106 via the passivation film 103. The color filter 102 has a plurality of types that transmit different wavelength regions from each other, and has a specific arrangement corresponding to each of the PD 104. The arrangement of the color filters 102 will be described later. A set of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

A microlens 101 is provided on the incident side of the incident light in the color filter 102 corresponding to each pixel. The microlens 101 collects incident light toward the corresponding PD 104.

The wiring layer 108 has a wiring 107 that transmits a pixel signal from the PD layer 106 to the signal processing chip 111. The wiring 107 may have multiple layers, and may be provided with passive elements and active elements.

A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on the facing surfaces of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are aligned by pressurization or the like. The bumps 109 are joined together and electrically connected.

Similarly, a plurality of bumps 109 are arranged on the surfaces of the signal processing chip 111 and the memory chip 112 facing each other. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressurized, so that the aligned bumps 109 are joined to each other and electrically connected.

The bonding between the bumps 109 is not limited to Cu bump bonding by solid phase diffusion, and micro bump bonding by solder melting may be adopted. Further, for example, about one bump 109 may be provided for one unit area described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD 104. Further, in the peripheral region other than the pixel region in which the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel region may be provided together.

The signal processing chip 111 has TSVs (Through Silicon Vias) 110 that connect circuits provided on the front and back surfaces to each other. The TSV 110 is preferably provided in the peripheral region. Further, the TSV 110 may also be provided in the peripheral area of the image pickup chip 113 and the memory chip 112.

FIG. 2 is a diagram illustrating a pixel arrangement of the image pickup chip 113 and a unit area 131. In particular, the state in which the image pickup chip 113 is observed from the back surface side is shown. For example, 20 million or more pixels are arranged in a matrix in the pixel area. In the present embodiment, for example, 16 pixels of adjacent 4 pixels × 4 pixels form one unit area 131. The grid lines in the figure show the concept that adjacent pixels are grouped to form a unit region 131. The number of pixels forming the unit region 131 is not limited to this, and may be about 1000, for example, 32 pixels × 64 pixels, or more or less (for example, 1 pixel).

As shown in the partially enlarged view of the pixel region, the unit region 131 includes four so-called Bayer arrays including four pixels of green pixels Gb, Gr, blue pixels B, and red pixels R in the vertical and horizontal directions. The green pixel is a pixel having a green filter as the color filter 102, and receives light in the green wavelength band among the incident light. Similarly, a blue pixel is a pixel having a blue filter as a color filter 102 and receives light in a blue wavelength band, and a red pixel is a pixel having a red filter as a color filter 102 and receives light in a red wavelength band. Receive light.

In the present embodiment, a plurality of blocks are defined so that each block includes at least one unit area 131, and each block can control the pixels included in each block with different control parameters. That is, it is possible to acquire an imaging signal having different imaging conditions between the pixel group included in a certain block and the pixel group included in another block. Examples of control parameters are frame rate, gain, thinning rate, number of rows or columns to be added to add pixel signals, charge accumulation time or number of times, digitization bits, and the like. Further, the control parameter may be a parameter in image processing after acquiring an image signal from a pixel.

FIG. 3 is a circuit diagram corresponding to the unit region 131 of the imaging chip 113. In FIG. 3, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. At least a part of each transistor described below corresponds to the transistor 105 of FIG.

As described above, the unit region 131 is formed from 16 pixels. The 16 PD 104s corresponding to the respective pixels are connected to the transfer transistor 302, and each gate of each transfer transistor 302 is connected to the TX wiring 307 to which the transfer pulse is supplied. In the present embodiment, the TX wiring 307 is commonly connected to the 16 transfer transistors 302.

The drain of each transfer transistor 302 is connected to the source of each corresponding reset transistor 303, and the so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to the Vdd wiring 310 to which the power supply voltage is supplied, and its gate is connected to the reset wiring 306 to which the reset pulse is supplied. In the present embodiment, the reset wiring 306 is commonly connected to the 16 reset transistors 303.

The drain of each amplification transistor 304 is connected to the Vdd wiring 310 to which the power supply voltage is supplied. Also, the source of each amplification transistor 304 is connected to the drain of each corresponding selection transistor 305. Each gate of the selection transistor is connected to a decoder wiring 308 to which a selection pulse is supplied. In the present embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. Then, the source of each selection transistor 305 is connected to the common output wiring 309. The load current source 311 supplies current to the output wiring 309. That is, the output wiring 309 for the selection transistor 305 is formed by the source follower. The load current source 311 may be provided on the imaging chip 113 side or the signal processing chip 111 side.

Here, the flow from the start of charge accumulation to the pixel output after the end of accumulation will be described. When a reset pulse is applied to the reset transistor 303 through the reset wiring 306 and at the same time a transfer pulse is applied to the transfer transistor 302 through the TX wiring 307, the potentials of the PD 104 and the floating diffusion FD are reset.

When the application of the transfer pulse is released, the PD 104 converts the received incident light into an electric charge and accumulates it. After that, when the transfer pulse is applied again without the reset pulse being applied, the accumulated charge is transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after charge accumulation. .. Then, when the selection pulse is applied to the selection transistor 305 through the decoder wiring 308, the fluctuation of the signal potential of the floating diffusion FD is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305. As a result, the pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

As shown in FIG. 3, in the present embodiment, the reset wiring 306 and the TX wiring 307 are common to the 16 pixels forming the unit region 131. That is, the reset pulse and the transfer pulse are applied to all 16 pixels at the same time. Therefore, all the pixels forming the unit region 131 start the charge accumulation at the same timing and end the charge accumulation at the same timing. However, the pixel signal corresponding to the accumulated charge is selectively output from the output wiring 309 by sequentially applying the selection pulse to each selection transistor 305. Further, the reset wiring 306, the TX wiring 307, and the output wiring 309 are separately provided for each unit area 131.

By configuring the circuit with the unit region 131 as a reference in this way, the charge accumulation time can be controlled for each unit region 131. In other words, pixel signals with different frame rates can be output between the unit regions 131. Furthermore, while the one unit region 131 is made to perform one charge accumulation, the other unit region 131 is made to repeat the charge accumulation many times to output a pixel signal each time. It is also possible to output each frame for moving images at different frame rates between the unit areas 131.

FIG. 4 is a block diagram showing a functional configuration of the image sensor 100. The analog multiplexer 411 sequentially selects 16 PD 104s forming the unit area 131, and outputs each pixel signal to the output wiring 309 provided corresponding to the unit area 131. The multiplexer 411 is formed on the imaging chip 113 together with the PD 104.

The pixel signal output via the multiplexer 411 is CDS and A / by a signal processing circuit 412 formed on the signal processing chip 111 that performs correlated double sampling (CDS) analog / digital (A / D) conversion. D conversion is performed. The A / D converted pixel signal is passed to the demultiplexer 413 and stored in the pixel memory 414 corresponding to each pixel. The demultiplexer 413 and the pixel memory 414 are formed on the memory chip 112.

The arithmetic circuit 415 processes the pixel signal stored in the pixel memory 414 and hands it over to the image processing unit in the subsequent stage. The arithmetic circuit 415 may be provided on the signal processing chip 111 or the memory chip 112. Although FIG. 4 shows the connections for one unit area 131, in reality, these exist for each unit area 131 and operate in parallel. However, the arithmetic circuit 415 does not have to exist for each unit area 131. For example, even if one arithmetic circuit 415 processes sequentially while referring to the value of the pixel memory 414 corresponding to each unit area 131 in order. Good.

As described above, the output wiring 309 is provided corresponding to each of the unit regions 131. Since the image sensor 100 has an image pickup chip 113, a signal processing chip 111, and a memory chip 112 stacked on top of each other, by using an electrical connection between the chips using bumps 109 for these output wirings 309, each chip is placed in the plane direction. Wiring can be routed without making it large.

<Explanation of imaging device>
FIG. 5 is a block diagram illustrating the configuration of the image pickup device 1 having the image pickup device 100 described above. In FIG. 5, the image pickup apparatus 1 includes an image pickup optical system 10, an image pickup unit 20, an image processing unit 30, a work memory 40, a display unit 50, a recording unit 60, and a control unit 70.

The imaging optical system 10 is composed of a plurality of lenses, and guides a light flux from the field of view to the imaging unit 20. The image pickup optical system 10 may be integrally configured with the image pickup device 1 or may be interchangeably configured with respect to the image pickup device 1. Further, the imaging optical system 10 may have a built-in focus lens or a built-in zoom lens.

The image pickup unit 20 includes the above-mentioned image pickup element 100 and a drive unit 21 that drives the image pickup element 100. By driving and controlling the image sensor 100 by the control signal output from the driving unit 21, the above-mentioned block unit can be independently accumulated and controlled. The control unit 70 gives instructions such as the position, shape, and range of the block to the drive unit 21.

The image processing unit 30 cooperates with the work memory 40 to perform image processing on the image data captured by the image capturing unit 20. In the present embodiment, the image processing unit 30 also performs detection processing of a main subject included in the image in addition to normal image processing (color signal processing, gamma correction, etc.). The main subject can be detected by the image processing unit 30 by using a known face detection function. Further, in addition to face detection, for example, as described in Japanese Patent Application Laid-Open No. 2010-16621 (US2010 / 0002940), the human body included in the image may be detected as the main subject.

The work memory 40 temporarily stores image data before and after JPEG compression and before and after MPEG compression. The display unit 50 is composed of, for example, a liquid crystal display panel, and displays images (still images and moving images) captured by the image pickup unit 20 and various information, and displays an operation input screen. The imaging device 1 according to the present embodiment has a first display unit 51 and a second display unit 53 as display units 50, the first display unit 51 has a first touch panel 52, and the second display unit 53 has a second touch panel 54. However, each is incorporated.

FIG. 6 is an external view illustrating the first display unit 51 and the second display unit 53 provided in the image pickup apparatus 1. In FIG. 6, the first display unit 51 is provided on the back surface of the image pickup apparatus 1, and the second display unit 53 is provided so as to be foldable via the hinge portion 55. The first touch panel 52 outputs a signal indicating the position where the user touches the first display unit 51, and the second touch panel 54 outputs a signal indicating the position where the user touches the second display unit 53.

In the present embodiment, the display unit 50 is configured by the first display unit 51 and the second display unit 53, but instead, the display unit 50 is configured by one display device, and the display devices are divided into two. The display area may be divided and displayed, the first display area may be treated as the display by the first display unit 51, and the second display area may be treated as the display by the second display unit 53. Further, the display unit 50 is composed of one display device, and the display device is displayed so as to display a display to be displayed on the first display unit 51 or a display to be displayed on the second display unit 53. It may be configured to switch the contents. For example, the display content is switched at the timing when the touch panel provided on the display unit 50 is operated.

Returning to the description of FIG. 5, the recording unit 60 stores various data such as image data in a storage medium such as a memory card. The control unit 70 has a CPU and controls the entire operation of the image pickup apparatus 1. In the present embodiment, the control unit 70 divides the image pickup surface of the image pickup device 100 into a plurality of blocks, and causes the blocks to acquire images at different frame rates. For this purpose, the control unit 70 instructs the drive unit 21 of the position, shape, range, and control parameters for each block.

<Area of interest and surrounding area>
In the present embodiment, the concept of the area of interest and the peripheral area is introduced in the screen to correspond to the plurality of blocks. The control unit 70 sets a region including a human body detected by the human body detection process as a region of interest based on, for example, a live view image. Then, a region other than the region of interest is set as a peripheral region. Here, the live view image is also called a preview image before the main imaging is performed, and refers to a monitor image acquired by the image sensor 100 at a predetermined frame rate (for example, 90 fps).

Instead of performing the human body detection process, a motion vector may be detected between a plurality of frames constituting the live view image, and a region including a moving subject detected based on the motion vector may be a region of interest.

Further, the main subject corresponding to the position where the user who visually recognizes the live view image touches the first touch panel 52 while the live view image is displayed on the first display unit 51 of the display unit 50 (displayed). The area of interest may be the area of interest.

The main subject (displayed) corresponding to the position where the user who visually recognizes the live view image touches the second touch panel 54 while the live view image is displayed on the second display unit 53 of the display unit 50. The area of interest may be the area of interest.

When the control unit 70 determines the region of interest and the peripheral region in this way, the control unit 70 instructs the drive unit 21 to divide the image sensor 100 into a block corresponding to the region of interest and a block corresponding to the peripheral region for storage control. send. Note that the control parameters common to the entire screen are applied to the live view image before the region of interest and the peripheral region are determined.

FIG. 7A is a diagram illustrating a live view image before determining the region of interest and the peripheral region. In FIG. 7A, the live view image is composed of, for example, a total of 48 unit areas 131 of 6 vertical × 8 horizontal. In FIG. 7A, the shape of the unit region 131 is vertically long, but the shape of the unit region 131 may be square or horizontally long.

Here, when displaying the live view image before determining the area of interest and the peripheral area on the display unit 50 (first display unit 51 or second display unit 53), the grid line 71 indicating the unit area is displayed in the live view. It may be displayed overlaid on the image.

FIG. 7B is a diagram illustrating a region of interest determined by the control unit 70 by the human body detection process based on the live view image. In FIG. 7B, a region including the detected human body (composed of a total of 20 unit regions 131 of 4 vertical × 5 horizontal surrounded by a frame 72) is designated as a region of interest. Further, a region other than the region of interest (composed of a total of 28 unit regions 131 outside the frame 72) is set as the peripheral region.

When the control unit 70 determines the region of interest, the frame 72 indicating the determined region of interest is superimposed on the live view image displayed on the display unit 50 (first display unit 51 or second display unit 53). It may be displayed.

In the present embodiment, after the attention region and the peripheral region are determined, different control parameters are applied to the block corresponding to the attention region and the block corresponding to the peripheral region of the image sensor 100. Then, the first display unit 51 and the second display unit 53 display different images based on the plurality of images obtained by applying different control parameters.

Specifically, the image corresponding to the region of interest and the image corresponding to the peripheral region are combined, and the combined image is displayed as the first live view image on the first display unit 51. Further, only the image corresponding to the region of interest is displayed as the second live view image on the second display unit 53.

8 (a) is an image 90 acquired corresponding to the peripheral region, FIG. 8 (b) is an image 80 acquired corresponding to the region of interest, and FIG. 8 (c) is a composite th. It is a figure which illustrates 1 live view image 90A. In FIG. 8A, the image 90 corresponding to the peripheral region is acquired with a frame rate of, for example, 90 fps. In FIG. 8B, the image 80 corresponding to the region of interest is acquired, for example, at a frame rate of 180 fps, which is twice that of the peripheral region.

According to FIGS. 8A and 8B, 8 frames of the image 80 corresponding to the region of interest can be acquired while 4 frames of the image 90 corresponding to the peripheral region are acquired. In this way, by acquiring the attention region at a higher frame rate than the peripheral region, the visibility of the second live view image 80A for the attention region displayed on the second display unit 53 (FIG. 6) is improved. Can be done.

Regarding the first live view image 90A displayed on the first display unit 51 (FIG. 6), when the image 80 corresponding to the region of interest is combined with the image 90 corresponding to the peripheral region, the peripheral region and the acquisition start timing are used. Is the same frame (the frame surrounded by the thick line in FIG. 8B), and the images are fitted and combined. In this way, the playback frame rate of the first live view image 90A is adjusted to match the frame rate of the peripheral region where the frame rate at the time of acquisition is low, so that the user observing the first live view image 90A feels uncomfortable. There is no. The frame of the alternate long and short dash line displayed on the screen of the first display unit 51 in FIG. 6 is described for convenience in order to make the range of the fitted and composited image 80 easy to understand, and is not necessarily displayed on the screen of the first display unit 51. You may. Here, when displaying an icon or the like overlaid on the first live view image 90A displayed on the first display unit 51, the image 80 corresponding to the region of interest is avoided and the image 80 is superimposed on the position corresponding to the peripheral region. And.

In the example of FIG. 6, the first live view image 90A including the region of interest and the peripheral region is displayed on the first display unit 51, and the second live view image 80A corresponding only to the region of interest is displayed on the second display unit 53. Although the display is performed, the display contents may be reversed between the first display unit 51 and the second display unit 53. For example, the display content is reversed at the timing when the first touch panel 52 or the second touch panel 54 is operated. Further, when the second live view image 80A corresponding to the region of interest is displayed, it may be enlarged and displayed by electronic zoom processing. The second live view image 80A displayed on the second display unit 53 in FIG. 6 is a second live view image displayed on the second display unit 53 rather than the image 80 displayed on the first display unit 51. An example shows a state in which 80A is enlarged by an electric enlargement process (so-called electronic zoom) so that it is displayed in a large size. The electronic zoom process in this case is performed by the image processing unit 30.

<Explanation of flowchart>
FIG. 9 is a flowchart illustrating a flow of a photographing operation executed by the control unit 70 of the imaging device 1. When the ON-OFF switch (not shown) is turned on and each part of the image pickup apparatus 1 is energized, the control unit 70 repeatedly activates the process according to FIG. In step S101 of FIG. 9, the control unit 70 starts imaging by the imaging unit 20. The acquisition of the live view image started in step S101 is performed by applying a control parameter common to all regions of the image sensor 100. That is, the entire region of the image sensor 100 is imaged at the same frame rate (for example, 90 fps).

The control unit 70 causes the image processing unit 30 to process the live view image data acquired by the imaging unit 20, and then displays the live view image data on the first display unit 51. In the present embodiment, the display of the second display unit 53 is turned off at this stage.

In step S102, the control unit 70 determines the region of interest based on the live view image as described above, and proceeds to step S103. The control unit 70 sets a region other than the region of interest as a peripheral region. In step S103, the control unit 70 controls the imaging unit 20 and the display unit 50 based on the attention region and the peripheral region determined in step S102.

The control unit 70 sends an instruction to the drive unit 21 to change the frame rate of the block corresponding to the region of interest of the image sensor 100 from 90 fps to 180 fps, which is doubled. In addition, you may change from 90 fps to 270 fps which is tripled. The control unit 70 turns on the display of the second display unit 53 in response to increasing the frame rate of the block corresponding to the region of interest, and displays the second live view image 80A corresponding to the region of interest only in the second display unit. It is displayed on 53 (Fig. 6). On the other hand, the first display unit 51 displays the first live view image 90A in which the image 80 corresponding to the region of interest is fitted and synthesized in the image 90 corresponding to the peripheral region (FIG. 6).

At this time, in order to improve the visibility of the region of interest, the image processing unit 30 is subjected to the above-mentioned electronic zoom processing, and the second live view image 80A in which the region of interest is enlarged is displayed on the second display unit 53. In this case, the enlargement ratio (electronic zoom magnification) may be set so that the display of the area of interest after enlargement does not extend beyond the display area of the second display unit 53. As a result, the user can visually recognize the second live view image 80A having a high frame rate in the region of interest from the second display unit 53 while visually recognizing the entire composition with the first live view image 90A of the first display unit 51.

In step S104, the control unit 70 determines whether or not there is an operation on the image pickup apparatus 1. When the operation is performed, the control unit 70 positively determines step S104 and proceeds to step S105, and when the operation is not performed, the control unit 70 negatively determines step S104 and repeats the determination process.

The operation of determining the presence or absence in step S104 includes an operation performed on the first touch panel 52 and the second touch panel 54 and an operation performed on an operation member (not shown). In the present embodiment, since the second live view image 80A corresponding to the region of interest is displayed on the second display unit 53, it is determined whether or not there is an operation on the second touch panel 54.

For example, the control unit 70 causes the second display unit 53 to display the front pin icon 56, the rear pin icon 57, the icon 58 for increasing the frame rate, and the icon 59 for decreasing the frame rate on the second display unit 53 as illustrated in FIG. The touch panel 54 accepts touch operations. In step S105, the control unit 70 executes a process associated with the icon displayed on the second display unit 53 at the touch position of the second touch panel 54 and proceeds to step S106. move on. For example, when the front pin icon 56 being displayed is touch-operated, the control unit 70 drives the focus lens of the imaging optical system 10 toward the front pin so that the main subject in the region of interest is the so-called front pin.

In step S106, the control unit 70 determines whether or not the release operation has been performed. The control unit 70 positively determines step S106 and proceeds to step S107 when the release button (not shown) is pressed (fully pressed), and negatively determines step S106 if the pressing operation is not performed. Then, the process returns to step S104. When returning to step S104, the above-described processing is repeated. The release icon may be displayed on the second display unit 53, and step S106 may be positively determined when the released release icon being displayed is touch-operated.

In step S107, the control unit 70 executes a shooting process, stores the acquired image data in a memory card or the like by the recording unit 60, and ends the process according to FIG. In the shooting process, a control parameter common to all areas of the image sensor 100 is applied to acquire and record a still image of one frame (main imaging), and an image of a plurality of frames is acquired even after the still image is acquired. .. Then, slow playback moving image data is generated and recorded based on the images of a plurality of frames acquired during the predetermined time before and after the release operation. The slow playback moving image data refers to moving image data to be reproduced at a frame rate (for example, 60 fps) slower than the frame rate acquired by the image sensor 100.

The control unit 70 generates slow playback moving image data as follows. That is, for displaying the first live view image 90A and the second live view image 80A described above from the first predetermined time (for example, 0.6 seconds before) to the time t1 from the release operation time (referred to as t1). A plurality of frame images temporarily stored in the work memory 40 (the frame rate is the frame rate set in the peripheral area. For example, 0.6 seconds when acquired at 90 fps is 54 frames). And a plurality of frame images stored in the work memory 40 from the time t1 to the second predetermined time after the time t1 (for example, 0.4 seconds later) (the frame rate after the release operation is the frame rate set in the region of interest). For example, 0.4 seconds when acquired at 180 fps is 72 frames), and slow playback moving image data is generated. As a result, playback is performed based on a plurality of frame images (126 in total) stored in the work memory 40 for 1 second (0.6 seconds before time t1 to 0.4 seconds after time t1) sandwiching time t1. Generates slow playback video data with a time of about 2 seconds.

When the slow playback moving image is generated in this way, slow playback video data based on the frame image acquired at a slow frame rate is obtained before the release operation, and slow playback video based on the frame image acquired at a fast frame rate after the release operation. Data is obtained. The slow image / moving image data is generated as MPEG data or JPEG data by the image processing unit 30.

According to the embodiment described above, the following effects can be obtained.
(1) The imaging device 1 includes an imaging unit 20 capable of imaging a peripheral region at a first frame rate of 90 fps and an image of a region of interest at a second frame rate of 180 fps, which is faster than the first frame rate, and a peripheral region and a region of interest. Since the control unit 70 for displaying the image captured in the first display unit 51 and the control unit 70 for displaying the image captured in the region of interest on the second display unit 53 are provided, the visibility of the image can be improved. Can be improved.

(2) In the image pickup apparatus 1 of the above (1), the control unit 70 further displays an icon related to the operation related to the image in the vicinity of the image displayed on the second display unit 53, so that the visibility is improved. The operability for images can also be improved.

(3) In the image pickup apparatus 1 of the above (1) and (2), the control unit 70 further performs additional display on the image captured in the region of interest displayed on the first display unit 51. Therefore, for example, the range of the area of interest can be displayed in an easy-to-understand manner.

(4) In the image pickup apparatus 1 of the above (1) to (3), the control unit 70 enlarges the image captured in the area of interest and displays it on the second display unit 53. The visibility of the image can be further improved.

(5) In the imaging devices 1 of (1) to (4) above, the control unit 70 includes a compositing unit that synthesizes an image captured in the peripheral region and an image captured in the region of interest, and the compositing unit includes the compositing unit. (1st live view image 90A) is displayed on the 1st display unit 51. As a result, the captured image can be displayed as one live view image.

(6) In the image pickup apparatus 1 of the above (5), the synthesis unit (control unit 70) has a first frame rate in the peripheral region from the time-series images 80 imaged at the second frame rate of 180 fps in the region of interest. Since an image corresponding to 90 fps is selected and the selected image is sequentially combined with the image 90 captured in the peripheral region, it is possible to suppress the discomfort given to the user observing the combined first live view image 90A. Can be done.

(7) Since the image pickup apparatus 1 of the above (1) to (5) is further provided with a control unit 70 that determines a part of the image captured as the peripheral region as the region of interest, for example, the image is captured as the peripheral region. The area including the face of a person in the imaged image and the area touch-operated on the screen displaying the image captured in the peripheral area can be determined as the area of interest.

(8) In the imaging device 1 of the above (7), the control unit 70 determines the region of interest based on face detection or human body detection for the image captured in the peripheral region, so that the region of interest is automatically focused even without a touch operation. You can decide the area.

(Modification example 1)
In the above-described embodiment, the case where the main subject is one has been described as an example. In the first modification, a case where the number of main subjects is a plurality is described. FIG. 10 is a diagram illustrating a scene in which a plurality of main subjects exist. In the case of photographing the scene illustrated in FIG. 10A, for example, the control unit 70 sets a boy behind the slide as the first main subject and a girl sliding on the slide as the second main subject. In this case, the area including the first main subject is set as the first attention area, and the area including the second main subject is set as the second attention area.

In the case of shooting the scene illustrated in FIG. 10B, for example, the control unit 70 sets a girl on a swing as the first main subject and a boy on the swing as the second main subject. Also in this case, the area including the first main subject is set as the first attention area, and the area including the second main subject is set as the second attention area.

In the first modification, the control unit 70 includes the first main subject based on the live view image when the first main subject and the second main subject having different moving speeds in the screen exist. The area of interest is determined, and the second area of interest that includes the second main subject is determined. The peripheral region in this case is a region other than the first attention region and the second attention region.

The control unit 70 sends an instruction to the drive unit 21, changes the frame rate of the block corresponding to the first attention region of the image sensor 100 from, for example, 90 fps to 150 fps, and changes the frame rate of the block corresponding to the second attention region. For example, change from 90 fps to 240 fps. As illustrated in FIG. 11, the control unit 70 further divides the second display unit 53 into two display areas and displays the live view image 81A corresponding to the first attention area on the right side of the screen, for example. The live view image 82A corresponding to the second attention region is displayed on the left side of the screen.

In the first modification, a plurality of live view images 81A and 81B corresponding to the plurality of areas of interest are arranged and displayed as a second live view image. The alternate long and short dash line drawn in the center of the screen of the second display unit 53 in FIG. 11 is drawn for convenience in order to make the boundaries of the plurality of live view images 81A and 81B easy to understand, and is not necessarily the screen of the second display unit 53. It is not necessary to display it in.

On the other hand, in the first display unit 51, a first live view image 90A obtained by fitting and synthesizing an image 81 corresponding to the first attention region and an image 82 corresponding to the second attention region in the image corresponding to the peripheral region is provided. Display it. The frame of the alternate long and short dash line displayed on the screen of the first display unit 51 in FIG. 11 is described for convenience in order to make the ranges of the plurality of images 81 and 82 easy to understand, and is not necessarily on the screen of the first display unit 51. It does not have to be displayed.

Further, in FIG. 11, the frame rate of the front pin icon 56A, the rear pin icon 57A, and the block corresponding to the first attention area is increased in the vicinity of the live view image 81A corresponding to the first attention area of the second display unit 53. The icon 58A and the icon 59A that lowers the frame rate of the block corresponding to the first attention area are displayed, and the touch operation is accepted by the second touch panel 54. The user can focus on the live view image 81A and perform a focus operation and a frame rate adjustment operation.

Furthermore, in FIG. 11, the frame rates of the front pin icon 56B, the rear pin icon 57B, and the block corresponding to the second attention area are set in the vicinity of the live view image 82A corresponding to the second attention area of the second display unit 53. The icon 58B for raising and the icon 59B for lowering the frame rate of the block corresponding to the second attention area are displayed, and the touch operation is accepted by the second touch panel 54. The user can focus on the live view image 82A and perform a focus operation and a frame rate adjustment operation.

(Modification 2)
In the above-described embodiment and the first modification, a camera having an optical viewfinder is illustrated as the image pickup device 1, but the image pickup device 1 may be configured by a camera having no optical viewfinder.

(Modification 3)
In the above-described embodiment and the first modification, the example in which the second display unit 53 is rotatably provided via the hinge has been described, but the second display unit 53 is attached to and detached from the housing of the image pickup device 1. It may be freely configured. In this case, the housing side of the imaging device 1 and the housing side of the second display unit 53 are each provided with wireless communication units, and the information to be displayed on the second display unit 53 by wireless communication via these wireless communication units is captured by the imaging device. Transmission is performed from the 1st side to the 2nd display unit 53 side. The wireless communication unit may be incorporated inside the housing or may be mounted outside the housing. As wireless communication, for example, infrared optical communication, Bluetooth (registered trademark), wireless LAN communication, or the like is used.

(Modification example 4)
In the above-described embodiment and the first modification, an example in which the second display unit 53 is configured as a dedicated display unit has been described. Instead, the liquid crystal display unit of the high-performance mobile phone may be operated as the second display unit, or the liquid crystal display unit of the tablet terminal may be operated as the second display unit. In this case, a wireless LAN communication unit is provided on the housing side of the image pickup device 1, and the liquid crystal of the high-performance mobile phone (or tablet terminal) is provided by wireless LAN communication between the image pickup device 1 and the high-performance mobile phone (or tablet terminal). Information to be displayed on the display unit is transmitted from the image pickup device 1 side to the high-performance mobile phone (or tablet terminal). The wireless LAN communication unit may be incorporated inside the housing of the image pickup apparatus 1 or may be mounted outside the housing.

(Modification 5)
In the above-described embodiment and the first modification, the camera is illustrated as the image pickup device 1, but the image pickup device 1 may be configured by a high-performance mobile phone (or tablet terminal). In this case, the camera unit mounted on the high-performance mobile phone (or tablet terminal) is configured by using the stacked image sensor 100.

In the fifth modification, in addition to the first high-performance mobile phone (or the first tablet terminal) that constitutes the image pickup apparatus 1, a second high-performance mobile phone (or the second tablet terminal) that operates as the second display unit is used. .. In this case, the second high-performance mobile phone (or the second tablet) is provided by wireless LAN communication between the first high-performance mobile phone (or the first tablet terminal) and the second high-performance mobile phone (or the second tablet terminal). Information to be displayed on the liquid crystal display unit of the terminal) is transmitted from the first high-performance mobile phone (or the first tablet terminal) to the second high-performance mobile phone (or the second tablet terminal).

The above description is merely an example, and is not limited to the configuration of the above embodiment. The above-described embodiment and the configurations of the respective modifications may be combined as appropriate.

1 ... Image sensor 10 ... Imaging optical system 20 ... Imaging unit 30 ... Image processing unit 40 ... Work memory 50 ... Display unit 51 ... First display unit 52 ... First touch panel 53 ... Second display unit 54 ... Second touch panel 60 ... Recording unit 70 ... Control unit 80A ... Second live view image 90A ... First live view image 100 ... Image sensor 109 ... Bump 111 ... Signal processing chip 112 ... Memory chip 113 ... Imaging chip 131 ... Unit area

Claims (19)

A first control line that is connected to a plurality of pixels arranged in the row direction and the column direction, and the first pixel of the plurality of pixels, and outputs a first control signal for controlling the first pixel. A second control line that is connected to a second pixel arranged on the row direction side from the first pixel among the plurality of pixels and outputs a second control signal for controlling the second pixel. With an image sensor including,
A control unit that displays a first image generated based on the first signal from the first pixel and a second image generated based on the second signal from the second pixel on the display unit.
Electronic equipment equipped with. In the electronic device according to claim 1,
The first pixel is connected to the first control line and has a first read circuit for reading the first signal.
The second pixel is an electronic device connected to the second control line and having a second read circuit for reading the second signal. In the electronic device according to claim 2.
The first read circuit has a first transfer unit that is connected to the first control line and transfers charge charged by photoelectric conversion.
The second read circuit is an electronic device connected to the second control line and having a second transfer unit that transfers charge charged by photoelectric conversion. In the electronic device according to claim 3,
The image sensor is connected to the first pixel and is connected to a third control line for outputting a third control signal for controlling the first pixel and the second pixel to control the second pixel. It has a fourth control line from which a fourth control signal is output, and
The first read circuit has a first reset unit which is connected to the third control line and resets the potential of the first floating diffusion to which the photoelectrically converted charge is transferred.
The second read circuit is an electronic device connected to the fourth control line and having a second reset unit that resets the potential of the second floating diffusion to which the photoelectrically converted charges are transferred. In the electronic device according to claim 2.
The first read circuit has a first reset unit which is connected to the first control line and resets the potential of the first floating diffusion to which the photoelectrically converted charge is transferred.
The second read circuit is an electronic device connected to the second control line and having a second reset unit that resets the potential of the second floating diffusion to which the photoelectrically converted charges are transferred. In the electronic device according to any one of claims 1 to 5.
The control unit is an electronic device that displays the first image on the first display monitor of the display unit and displays the second image on the second display monitor of the display unit. In the electronic device according to any one of claims 1 to 5.
The image pickup device is an electronic device having a plurality of the first pixels arranged side by side in the row direction and a plurality of the second pixels arranged side by side in the row direction. In the electronic device according to any one of claims 1 to 5.
The image pickup device is an electronic device having a plurality of the first pixels arranged side by side in the row direction and a plurality of the second pixels arranged side by side in the row direction. In the electronic device according to any one of claims 1 to 5.
The image pickup device has a plurality of the first pixels arranged side by side in the row direction and the column direction, and a plurality of the second pixels arranged side by side in the row direction and the column direction. Electronics. In the electronic device according to any one of claims 1 to 9.
The control unit controls the image sensor so that the frame rate for reading the first signal from the first pixel and the frame rate for reading the second signal from the second pixel are different frame rates. machine. In the electronic device according to any one of claims 1 to 10.
The first pixel has a first photoelectric conversion unit that converts light into electric charges.
The second pixel has a second photoelectric conversion unit that converts light into electric charges.
In the control unit, the storage time for accumulating the electric charge converted by the first photoelectric conversion unit and the accumulation time for accumulating the electric charge converted by the second photoelectric conversion unit are different. An electronic device that controls an image sensor. In the electronic device according to claim 10 or 11.
The control unit is an electronic device that controls a timing of outputting a first control signal to the first control line and a timing of outputting the second control signal to the second control line. In the electronic device according to any one of claims 1 to 12.
The image pickup device has a first conversion circuit for converting the first signal into a digital signal and a second conversion circuit for converting the second signal into a digital signal.
The control unit includes a first image generated based on the first signal converted into a digital signal by using the first conversion circuit, and the first image converted into a digital signal by using the second conversion circuit. An electronic device that displays a second image generated based on two signals on the display unit. In the electronic device according to claim 13,
The image sensor has an electron having a first semiconductor chip in which the first pixel and the second pixel are arranged, and a second semiconductor chip in which the first conversion circuit and the second conversion circuit are arranged. machine. In the electronic device according to claim 14,
The first semiconductor chip is an electronic device laminated by the second semiconductor chip. In the electronic device according to claim 13,
The imaging element includes a first storage circuit that stores the first signal converted into a digital signal using the first conversion circuit, and the second signal converted into a digital signal using the second conversion circuit. Has a second storage circuit that stores
The control unit has a first image generated based on the first signal stored in the first storage circuit and a second image generated based on the second signal stored in the second storage circuit. An electronic device that displays an image on the display unit. In the electronic device according to claim 16.
The image pickup element includes a first semiconductor chip in which the first pixel and the second pixel are arranged, a second semiconductor chip in which the first conversion circuit and the second conversion circuit are arranged, and the first. An electronic device having a storage circuit and a third semiconductor chip in which the second storage circuit is arranged. In the electronic device according to claim 17,
The first semiconductor chip is an electronic device laminated by the third semiconductor chip. In the electronic device according to claim 18.
The first semiconductor chip is an electronic device laminated by the second semiconductor chip.

Images