JP7483386B2 - Image Processing System - Google Patents

Description

The present invention relates to a technology that develops RAW video in accordance with playback information received from a client and transmits the video to the client.

A commonly used method for playing back videos recorded on a server is to stream the video from the server to the client. In this type of streaming delivery, the server stores in advance stream data encoded for multiple transmission speeds so that the video can be delivered with appropriate image quality in a variety of transmission environments.

The server described in Patent Document 1 calculates the transmission speed, selects stream data according to the transmission speed, and transmits it to the client.

JP 2003-169091 A

The configuration of Patent Document 1 is effective when handling general video image data encoded in H.264, HEVC, or the like. However, when a server device handles RAW video, there is a problem in that it is difficult to take advantage of the characteristics of the RAW video.

A RAW video is a video in which a RAW image is recorded as data for each frame of the video, and a RAW image is recorded RAW data obtained from a Bayer array image sensor. In order to obtain playable image data from a RAW image, the RAW image must be developed, and the hue, brightness, white balance, etc. of the image data are determined by the values of the parameters (development parameters) used in the development process of the RAW image.

The user adjusts the development parameter settings to achieve the hue and brightness they desire.

As a result, when the server handles RAW video, the image data itself to be displayed changes depending on the settings of the RAW video development parameters, so if a pre-encoded video is prepared, there is a problem that the video reflecting the user's RAW development settings cannot be confirmed. Also, when wanting to check the RAW development settings in high quality, there is a problem that the image quality is not satisfactory depending on the transmission speed.

In order to solve the above problem, a server of the present invention is a server capable of communicating with a client, and includes a receiving means for receiving development parameters from the client, an acquisition means for acquiring information regarding playback of a moving image from the client, a development processing means for performing a development process of a RAW moving image in accordance with the development parameters, and a transmission means for transmitting data of the RAW moving image developed by the development process to the client, wherein when the information acquired by the acquisition means is information regarding playback of a moving image, the development processing means executes a first development process that prioritizes development speed to develop the RAW moving image, and the transmission means transmits the data of the RAW moving image developed by the first development process by a first transmission method, and when the information acquired by the acquisition means is information regarding pausing of the moving image, the development processing means executes a second development process that prioritizes image quality to develop the RAW moving image, and the transmission means transmits the data of the RAW moving image developed by the second development process by a second transmission method .

This invention makes it possible to properly develop and transmit RAW video stored on a server.

2 is a block diagram showing the configuration of a client of the RAW video development system according to the present embodiment. FIG. 1 is a diagram showing an overview of the entire RAW video development system according to an embodiment of the present invention. FIG. 4 is a flowchart showing a RAW video processing procedure according to the present embodiment. FIG. 4 is a flowchart showing a RAW development processing procedure according to the present embodiment. FIG. 4 is a flowchart showing a playback method and a development processing decision process according to the present embodiment.

The following describes in detail an embodiment of the present invention with reference to the attached drawings.

The embodiment described below is one example of a means for realizing the present invention, and may be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions.

This embodiment is a RAW video development system (image processing system) in which a server that performs development processing of RAW videos and a client that sets parameters and operates playback of RAW videos are connected via a communications network.

The client operates according to an application program that allows the user to perform operations related to RAW video. The server operates according to a program that performs RAW development processing in response to instructions from the client and transmits the RAW development results to the client.

The RAW video used in this embodiment has a color depth of 12 bits, and is a RAW video in which RAW images are recorded uncompressed as data for each frame of the video. Note that the present invention can also be applied to other forms of RAW video, such as compressed RAW video.

(Overall system description)
FIG. 2 is a diagram showing an outline of the entire RAW video development system of this embodiment.
As shown in Fig. 2, the RAW video development system is made up of a client 210, the Internet 220 as a communication network, a server 230, and a storage 240. The client 210 and the server 230 are connected via the Internet 220. The server 230 is also connected to the storage 240. Note that the storage 240 may be a storage provided outside the server 230 as shown in Fig. 2, or a storage provided inside the server 230.

Client 210 stores a RAW video processing program that sets development parameters and plays RAW videos.

The server 230 stores a RAW development processing program that performs development processing of RAW video.

There are at least two types of development processes possible with RAW development processing programs: high-quality development processing and high-speed development processing. High-quality development processing performs development processing at 12 bits. High-speed development processing performs development processing at 8 bits. Also, high-speed development processing performs a simplified version of noise reduction processing, allowing for faster development compared to high-quality development processing. In other words, to put it the other way around, high-quality development processing is a slower development processing compared to high-speed development processing, and high-speed development processing is a lower-quality development processing compared to high-quality development processing.

RAW video files are stored in storage 240.

Details about each program will be provided below.

(Client description)
1 is a block diagram showing the configuration of a client 210. The client 210 is realized by a PC and a program that runs on the PC.

In FIG. 1, 190 is the client 210. 101 is a control unit that controls the entire client 210, for example a Central Processing Unit (CPU). 102 is a Read Only Memory (ROM) that stores programs and parameters that do not require modification. 103 is a Random Access Memory (RAM) that temporarily stores programs and data supplied from an external device, etc.

104 is an external storage device including a hard disk or memory card fixedly installed in the client 210, or an optical disk such as a floppy disk (FD) or compact disk (CD) that is removable from the computer device 100, a magnetic or optical card, an IC card, etc. The external storage device 104 can record digital data such as programs and image files. The external storage device 104 records a RAW video processing program, which will be described later, as program code that can be read by the control unit 101, and the control unit 101 is configured to execute this program code.

Reference numeral 105 denotes an interface with input devices such as a pointing device (mouse) and a keyboard 109 that receive user operations and input data. Reference numeral 106 denotes an interface with a display 110 that displays data held by the computer device 100 and data supplied thereto.

107 is a network interface for connecting to a network line 113 such as the Internet. It is connected to a server 230 via the network line.

108 is a system bus that connects each unit 101 to 107 so that they can communicate with each other.

In this embodiment, an example is shown in which the RAW video processing program is recorded in the external storage device 104, but the present invention is also applicable when the program is stored in the ROM 102 or on a network.

Next, we will explain the operation of the client in this embodiment.

Figure 3 is a flowchart showing the steps of the RAW video processing program executed by the control unit 101 in Figure 1.

In step S300, the control unit 101 displays the initial screen of the RAW video processing program. This is a process to display a window (not shown) in the same way as a general application.

The UI includes an Exit button, a file specification button, playback controls, a display size setting control, and a development parameter setting control. If no file is selected, buttons other than the Exit button and file specification button are inoperable. Note that no file is selected immediately after the program is launched.

In step S302, the control unit 101 measures the communication speed with the server 230. The measurement method may be a general method.

In step S305, the control unit 101 makes a file list request to the server 230 and obtains a list of readable RAW video files from the server 230 (file list request to server). At this time, the client 210 and the server 230 are connected via the Internet.

In step S310, the control unit 101 specifies the RAW video file to be developed. The control unit 101 displays a file specification control (not shown) on the display 110. The file specification controller displays the RAW video file list acquired in step S305. This allows the user to select a RAW video file in the storage 240. The file setting control is configured to use, for example, a file open dialog provided as standard in the OS (Operating System), and is configured to allow the selection of one image file.

In step S310, the control unit 101 also transmits the file name of the file selected by the user as the file name of the target RAW video file on the server (sets the file name on the server).

In addition, in step S310, the control unit 101 sets all buttons to a selectable state after the file name designation process.

In step S315, the control unit 101 sets development parameter values in response to user operations on the development parameter setting control (not shown). In this embodiment, the development parameters that can be set are white balance, brightness, contrast, and noise reduction. Of course, other parameters may also be set. Note that if the user does not perform any operations, this step does not process anything and moves to the next step.

In step S320, the control unit 101 sets the playback state in response to a user operation of a playback control (not shown). Playback operations that the user can operate include playing (playing) and pausing (stopping) the video. Note that an interface such as a playback slider can be used as the playback control.

When the user operates the playback control to perform a play operation, the playback state information is set to a play state. When the user operates the playback control to perform a pause operation, the playback state information is set to a pause state.

If the user does not perform any operation, this step does not process anything, maintains the playback state information, and moves to the next step.

In step S325, the control unit 101 sets the display size information in response to a user operation of a display size setting control (not shown). The user can select from three display sizes: 100% display, 50% display, and 25% display. Of course, other display sizes may also be selectable.

If the user does not perform any operation, this step will not process anything, will maintain the display size information, and will move to the next step.

In step S330, the control unit 101 performs a process of creating playback information. The playback information is composed of playback status information, transmitted video information, and communication speed information. Here, the playback status information is information indicating the playback status of the video at the client, the transmitted video information is information indicating the transmission form of the video to the client, and the communication speed information is information indicating the communication speed in the communication network. Note that the playback information may include other information.

In step S330, the control unit 101 sets the playback state information set in step S320 as the playback information. Also in step S330, the control unit 101 sets the display size information set in step S325 as the transmission video information. As the communication speed information, the control unit 101 sets the communication speed information measured in step S302.

In this embodiment, an example is shown in which display size information is set as the transmitted video information, but other configurations are also possible. For example, a configuration in which the bit rate of the video or the image quality of the video is set instead of the image size is also possible.

In step S335, the control unit 101 performs processing to send the development parameters and playback information to the server.

In step S340, the control unit 101 requests image data from the server and receives the developed image data from the server.

At this time, information (transmission method information) is also obtained indicating whether the image data is transmitted from the server in streaming format or as one frame of image data.

In step S345, the control unit 101 performs processing to display the image data acquired in step S340.

If the data is received via streaming, it will be processed and displayed. If one frame of image data is received, it will be processed to display the image data.

In step S350, the control unit 101 determines whether the file specification button has been pressed by the user. If the button has been pressed, the process proceeds to step S355. If the button has not been pressed, the process proceeds to step S360.

Processing moves to step S355 when the file specification button is pressed. In step S355, the control unit 101 sets the playback state information of the playback information to a paused state in order to pause video playback, and transmits the playback information to the server. Processing then moves to step S302, allowing the file specification process to be started anew.

In step S360, the control unit 101 determines whether the end button has been pressed by the user. If the button has been pressed, the program ends. If the button has not been pressed, the process proceeds to step S315, and the playback process is repeated.

By repeating steps S315 to S360, the image data reflecting the development parameter settings can be played back and displayed on the client side. Note that each processing step from step S315 to step S345 updates the image display for one frame, and by repeating this processing step multiple times, the image is updated and displayed as a video.

(Server description)
The RAW development server of this embodiment uses a workstation. The configuration of the server is the same as that of the client, so it will be described with reference to Fig. 1. The server 230 of this embodiment is realized by a workstation and a program that runs on the workstation.

In FIG. 1, 190 is the server 230. 101 is a control unit that controls the entire server 230, for example a Central Processing Unit (CPU). 102 is a Read Only Memory (ROM) that stores programs and parameters that do not require modification. 103 is a Random Access Memory (RAM) that temporarily stores programs and data supplied from an external device, etc.

104 is an external storage device including a hard disk or memory card fixedly installed in the server 230, or an optical disk such as a floppy disk (FD) or compact disk (CD) that is removable from the computer device 100, a magnetic or optical card, an IC card, etc. The external storage device 104 can record digital data such as programs and image files. In particular, in the external storage device 104 of this embodiment, a RAW development processing program, which will be described later, is recorded as program code that can be read by the control unit 101, and this program code is executed by the control unit 101.

Reference numeral 105 denotes an interface with input devices such as a pointing device (mouse) and a keyboard 109 that accept operations by a user (server administrator) to input data. Reference numeral 106 denotes an interface with a display 110 that displays data held by the computer device 100 and data supplied thereto.

107 is a network interface for connecting to a network line 113 such as the Internet. It is connected to a client 210 via the network line. The server 230 of this embodiment is also connected to a storage 240 via a network line. Note that in this embodiment, the server and storage are connected via a network line, but they may be connected by other methods. Of course, the RAW video file may also be recorded in an external storage device 104.

108 is a system bus that connects each unit 101 to 107 so that they can communicate with each other.

In this embodiment, an example is shown in which the RAW development processing program is recorded in the external storage device 104, but the present invention is also applicable when the program is stored in the ROM 102 or on a network.

Next, we will explain the operation of the server according to this embodiment.

Figure 4 is a flowchart showing the steps of the RAW development processing program executed by the control unit 101 in Figure 1.

In step S400, the control unit 101 determines whether the RAW development processing program has received any request from the client.

If there is a request from the client, the process proceeds to step S405. If there is no request, the process proceeds to step S485.

The process moves to step S405 when a request is received from the client.

In step S405, the control unit 101 determines whether the request from the client is a file list request. If it is a file list request, the process proceeds to step S410. If it is a different request, the process proceeds to step S420.

In step S410, the control unit 101 obtains a file list of RAW video files recorded in the storage 240.

In step S415, the control unit 101 sends a file list of the RAW video files recorded in the storage 240 to the client 210.

In step S420, the control unit 101 determines whether the request from the client is to set a file name. If it is, the process proceeds to step S425. If it is a different request, the process proceeds to step S430.

In step S425, the control unit 101 receives the file name of the RAW video file to be developed and performs processing to store it internally. If the target RAW video file is updated, the frame to be developed of the RAW video is set to the beginning.

In step S430, the control unit 101 determines whether the request from the client is for setting playback information. If it is for setting playback information, the process proceeds to step S435. If it is a different request, the process proceeds to step S445.

In step S435, the control unit 101 receives playback information from the client and performs processing to store it internally.

In step S445, the control unit 101 determines whether the request from the client is for setting development parameters. If it is for setting development parameters, the process proceeds to step S450. If it is a different request, the process proceeds to step S455.

In step S450, the control unit 101 receives development parameter information from the client and performs processing to store it internally.

In step S455, the control unit 101 determines whether the request from the client is for image data. If it is for image data, the process proceeds to step S460. If it is for another type of request, the process proceeds to step S485.

In step S460, the control unit 101 performs a process to determine the development process and transmission method of the image data to be transmitted to the client from the internally stored reproduction information. Details of step S460 will now be described in detail with reference to FIG. 5.

In step S600, the control unit 101 sets the development process to high-quality development. In step S605, the control unit 101 sets the transmission method to image data transmission.

Steps S610, S625, and S640 are decisions based on playback information.

In step S610, the control unit 101 determines the playback state information of the playback information. If the playback state information is in a play state, the process proceeds to step S615. Otherwise, the process proceeds to step S625.

In step S615, the control unit 101 sets the development process to high-speed development. In step S620, the control unit 101 sets the transmission method to streaming transmission.

In step S625, the control unit 101 makes a judgment based on the image size information in the playback information. If the image size information is less than 50%, the process proceeds to step S630. Otherwise, the process proceeds to step S640.

In step S630, the control unit 101 sets the development process to high-speed development. In step S635, the control unit 101 sets the transmission method to streaming transmission.

In step S640, the control unit 101 performs a judgment based on the communication speed information of the playback information. If the communication speed information is less than 100 Mbps, the process proceeds to step S645. Otherwise, the process ends and returns to step S465.

In step S645, the control unit 101 sets the development process to high-speed development. In step S650, the control unit 101 sets the transmission method to streaming transmission.

As described above, the development process and transmission method are determined by the processing of steps S600 to S650. When the video is paused, the image size is large, or the communication conditions are good, image data is sent to the client in high image quality even if the data volume is large. With this configuration, when it is desired to check the RAW development settings in a high-quality display, a high-quality display is possible by at least setting the playback state to a pause state. Of course, the development process and transmission process methods may be determined using judgments and thresholds other than those in this embodiment.

The playback information used in this embodiment is merely an example. For example, information on the image size, bit rate, and compression ratio of the transmitted video may be received from the client side, and the method of development processing and transmission processing may be determined depending on whether these values are above or below a threshold value.

Returning to the explanation of FIG.
In step S465, the control unit 101 performs development processing on the frame (one frame) of the RAW video to be developed in accordance with the development parameters received in step S450. The development processing method at this time is the development processing method determined in step S460. Furthermore, the RAW video file to be developed at this time is the RAW video file with the RAW video file name received in step S425. In the first development, the frame to be developed is the first frame.

In step S467, the control unit 101 performs an update process for the frame to be developed. If the playback state information is in the paused state, the frame to be developed is not changed. If the playback state information is in the play state, the frame to be developed is advanced to the next frame. However, even if the playback state information is in the play state, if the target frame is the final frame, the target frame remains the final frame.

In step S470, the control unit 101 determines whether the method of transmitting image data to the client is streaming transmission. If the transmission method determined in step S460 is streaming transmission, the process proceeds to step S480. If the image data is to be transmitted as a single image rather than streaming transmission, the process proceeds to step S475.

In step S475, the control unit 101 transmits the image data, which has been converted from the development results into a common image format, and the transmission method information to the client. In this embodiment, the development results are transmitted as 12-bit image data. Of course, the image data may be transmitted in other image formats. For example, the image data may be transmitted as compressed image data such as JPEG. Note that step S475 in this embodiment is a process for one frame. When transmitting image data for multiple frames, steps S400 to S480 are repeated for each frame.

In step S480, the control unit 101 transmits to the client streaming data in which the development results have been converted into a general streaming format, along with transmission method information. When converting to streaming in this embodiment, processing is performed so that the data is more highly compressed than when the image data is transmitted as is. Note that step S480 in this embodiment is processing for one frame. When transmitting streaming data for multiple frames, steps S400 to S480 are repeated for the number of frames. Of course, a method in which data for multiple frames is accumulated, and then video compression and transmission of the streaming data is also possible.

In step S485, the control unit 101 determines whether to terminate the RAW development processing program. Termination is expected to occur in typical cases, such as when the server administrator stops the RAW development processing program (stopping the provision of services to clients) or when the server is shut down.

If it is to end, the RAW development processing program is terminated. If it is not to end, the process returns to step S400 and the request processing from the client is repeated.

By repeating steps S400 to S485, it is possible to process requests from clients.

With the above configuration, images based on the settings of the RAW video development parameters can be displayed and checked on the client side.

In addition, if you want to check the RAW development settings in high quality, you can do so regardless of the transmission speed. For example, when a video is paused, a high quality display is possible. Therefore, this invention makes it possible to take advantage of the characteristics of RAW video when handling RAW video stored on a server.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various forms within the scope of the gist of the invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

The present invention can also be realized by executing the following process, that is, by supplying software (programs) that realize the functions of the above-described embodiments to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of the system or device read and execute the program.

Claims (6)

A server capable of communicating with a client,
A receiving means for receiving development parameters from the client;
An acquisition means for acquiring information regarding playback of a video from the client;
a development processing means for performing development processing of the RAW video in accordance with the development parameters;
a transmission means for transmitting data of the RAW video developed by the development process to the client,
When the information acquired by the acquisition means is information regarding playback of a moving image, the development processing means executes a first development process that prioritizes development speed to develop the RAW moving image, and the transmission means transmits data of the RAW moving image developed by the first development process by a first transmission method;
When the information acquired by the acquisition means is information regarding pausing the moving image, the development processing means executes a second development process that prioritizes image quality to develop the RAW moving image, and the transmission means transmits data of the RAW moving image developed by the second development process by a second transmission method.
A server comprising : The server of claim 1, characterized in that even if the information acquired by the acquisition means is information regarding pausing the video, if the communication speed with the client does not meet a predetermined speed, the development processing means executes a first development process that prioritizes development speed to develop the RAW video, and the transmission means transmits data of the RAW video developed by the first development process using a first transmission method . means for transmitting a file list to the client;
2. The server according to claim 1, wherein the development parameters received from the client are development parameters for one file in the file list . 2. The server according to claim 1, wherein the first transmission method is streaming transmission, and the second transmission method is a method of transmitting data as one image data . A method for controlling a server capable of communicating with a client, comprising the steps of:
a receiving step of receiving development parameters from the client;
An acquisition step of acquiring information regarding playback of a video from the client;
a development processing step for performing development processing of the RAW video in accordance with the development parameters;
a transmission step of transmitting data of the RAW video developed by the development process to the client,
When the information acquired in the acquisition step is information regarding playback of a moving image, the development step executes a first development process that prioritizes development speed to develop the RAW moving image, and the transmission step compresses data of the RAW moving image developed by the first development process into a first format and transmits the data by streaming;
If the information acquired in the acquisition step is information regarding pausing the moving image, the development step executes a second development process that prioritizes image quality to develop the RAW moving image, and the transmission step compresses a part of the data of the RAW moving image developed by the second development process into a second format and transmits the compressed file.
23. A method for controlling a server comprising : A computer readable program for causing a computer to function as each of the means of the server according to any one of claims 1 to 4 .

Images