WO2013187796A1

WO2013187796A1 - Method for automatically editing digital video files

Info

Publication number: WO2013187796A1
Application number: PCT/RU2013/000121
Authority: WO
Inventors: Александр Сергеевич ДИДЕНКО
Original assignee: Didenko Alexandr Sergeevich
Priority date: 2011-12-15
Filing date: 2013-02-14
Publication date: 2013-12-19

Abstract

The proposed invention relates to the field of color television signal processing in combination with recording, and specifically to the field of automatic and semi-automatic systems for editing digital movies from material filmed by a mobile phone camera. A method for automatically editing digital video files in which a digital editing smart-screenplay is created in the form of a digital data file and then, with the help thereof, user video fragments are processed and a video file is created.

Description

METHOD FOR AUTOMATIC INSTALLATION OF DIGITAL VIDEO FILES

Application area

I

The proposed invention relates to the field of color processing

i

television signals in combination with recording, in particular to the field of systems

one

automatic and semi-automatic editing of digital films from material, f

shot camera mobile phone.

BACKGROUND OF THE INVENTION

One of the consequences of the widespread use of smartphones with high-resolution video cameras as well as the ability to access high-speed Internet has been the emergence of a large number of so-called “mobile video”, which, among other things, consists of video footage shot by the user. According to statistics, in 201 1 the number of smartphones sold exceeded 1 billion (http://www.smartinsights.com/digital-marketing-strategy- alerts / mobile-marketing-statistics /), and every second smartphone user uses it to view , and approximately every fifth - for video recording (20.2% in the USA, 26.1% in Europe, 15.8% in Japan (http://mobithinking.com/mobile-marketing- tools / latest-mobile -stats)). Only on one of the main Internet services providing video hosting (hosting) services (both professional and amateur) (www.youtube.com) users upload about 48 hours of new videos per minute that are watched by three billion people a day - what made this site the third in the world (after Google.com and Facebook.com) by the number of visits one

(http: // searchenginewatchxom / article / 2073962 / New-YouTube-Statistics-48-Hours-of-Video-Uploaded-Per-Minute-3-Billion-Views-Per-Day). Due to portability, acceptable quality of video files and the possibility of distribution, the smartphone is increasingly becoming the preferred (compared to an amateur video camera) device for recording everyday events.

Nevertheless, despite the obvious advantages (more accurate image transmission, dynamics), there are several reasons why video shooting on mobile devices is less popular compared to photographing. In particular, the captured video takes up more space on the mobile device, it is not always high-quality (both in image and sound), it takes more time to watch, and not everyone has the patience to wait for an interesting moment. Therefore, a custom video needs to be edited. The specified editing should be carried out, if possible, on the mobile device itself, since the video transfer process takes time and requires certain efforts and user skills.

The purpose of editing should be to create less material.

duration than the source material, and with great potential

psychoemotional impact on the viewer in terms of perception of this new material

I

as a whole, consistent, characterized by a focus that sets it apart from other similar materials. This new quality is achieved through a process called “editing” among professional users, which consists in recombining the original sequence of frames in the video material with the aim of

length reduction, emphasis, rhythmization and structuring

sequence of frames. In 1917, the Russian director Lev Kuleshov wrote about the editing as follows: “In order to make a picture, the director must put together the individual pieces taken, erratic and incoherent, into a single whole and juxtapose individual moments in the most advantageous, integral and rhythmic sequence, just like a child makes up a whole word or phrase from separate, scattered cubes with letters. ” Key for the industry work Kuleshov (1907, 1929, in which

the so-called “Kuleshov effect” and “Kuleshov geographical experiment” are considered), Dzigi Vertov, Vsevolod Pudovkin and Sergei Eisenstein emphasized the importance of a meaningful combination in the installation of a system of metric, rhythmic, tonal, and associative-intellectual techniques.

The combination of individual fragments of the source material in a certain sequence is called interframe editing. It is important to note that

synchronization of the connection points of individual fragments of the video material with any arbitrarily chosen rhythm, including, for example, coinciding with the rhythm of the musical work superimposed on the video material, although it will improve the overall perception of the mounted material, but, as a rule, it will not allow achieving the desired result, that is, perceiving it as a whole, consistent, harmonious, with its own focus.

In addition to rhythmic interframe editing techniques, there are other interframe editing techniques, as well as intraframe editing, including such techniques as camera movement during shooting in axes perpendicular and / or parallel to the camera’s optical axis, changing the focal length in the camera’s lens, changing the angle between optical plane of the camera lens and the plane of the scene being shot, changing the relative position of the optical plane, the focal plane, and the focal point of the camera lens, changing the angular field of the object camera, changing the relative aperture (aperture) of the camera lens, as well as various combinations of the above techniques, carried out in order to capture certain objects or groups of objects in the space and time of the scene being shot.

Obviously, at the stage of interframe editing, intraframe editing is impossible, it is only possible to imitate some of its effects. The above factors of intraframe editing should be linked with each other and with factors of interframe editing in order to improve the perception of video material. For example, the rhythm may affect the perception of the video material.

alternating in the connected fragments of sizes, types of camera movement, special types of frames (for example, short shots) and effects (for example, freeze frame).

The indicated factors of interframe and intraframe editing should also be linked, on the one hand, with the content side of the scenes being mounted, and on the other, with the sound accompaniment of these scenes. The organization of the joint interaction of factors of visual and audio editing is called vertical editing. Thus, a necessary and sufficient condition for the perception of audiovisual

works as a whole is the coordinated work of many elements of visual and sound editing, provided that the effects of the remaining elements are not mutually contradictory, with the perception of the viewer in aspects of rhythmic, metric, tonal and associative-intellectual.

Editing an amateur mobile video has its own specificity that distinguishes it from editing material of other categories (for example, such as professional reportage video, professional art video, or amateur non-mobile video).

First of all, there are differences between the categories of professional and

amateur footage. Material related to the category of professional video is removed taking into account the fact that in the future it will be mounted. Thus, the intraframe editing techniques used at the shooting stage are pre-aligned with the interframe editing techniques, which will later be applied to the material. This is expressed, inter alia, in the absence of semantic voids in the material (when nothing significant for the viewer happens in the frame); immediately after turning on the camera and a few seconds before turning off a few seconds "for gluing" when the camera is not moves, and shooting is on, or dialing - when the optical plane shifts sharply with respect to the scene plane to achieve the effect of blurring the image, after which shooting stops immediately. In the conscious creation of material with the greatest variety in terms of intraframe editing techniques, when the interframe editing plan is uncertain (in the case of sequential shooting), or, on the contrary, in the creation of video material using strictly defined intraframe editing techniques when there is an intraframe editing plan (for example, in the format of pre-drawn storyboards that are often used when shooting

feature film).

Amateur material is more spontaneous, lack of pre-planned selection of video fragments with the necessary factors for intraframe editing, lack of gaps for connecting fragments, the presence of semantic voids, the presence of extremely complex spontaneous camera movements, hand shake, etc., which significantly complicates the installation task professional quality standards.

At the same time, amateur material shot on the camera of a mobile phone or smartphone (hereinafter - amateur mobile video) has its own

specificity compared to amateur videos shot on a stationary video camera or digital camera. Basically, these differences are caused by the fact that for the first time the video shooting function is not paramount for mobile devices, which affects their technical capabilities in this regard; secondly, mobile devices constantly accompany the modern user, always being at his disposal, in contrast to stationary devices specially designed for shooting.

From the point of view of content, amateur mobile video is more focused on any event, since the user always has a device for shooting at hand, and turns it on only when a suitable situation develops. If U2013 / 000121 the user specifically takes a separate camera, he will make the material less

saturated event content and will require additional installation efforts to cut off unnecessary material. The role is played by the fact that the disk space of a mobile device is, on average, less than the disk space of a video camera or camera, which the user has an idea about, so amateur material shot by him on a stationary camera will be on average longer than the material shot on a mobile phone.

From the point of view of the technology of shooting and storing material, stationary camera models are common, which when shooting do not always start a new file when you press a button, start shooting material, and add material to the end of an existing file that is currently open (if there is currently open file). In the future, this leads to the need to cut long files into shorter clips before editing, that is, an additional stage of processing the material is added, which, due to the peculiarities of the technology for shooting mobile video material, is not required when editing the latter.

In addition, the lenses of modern amateur stationary cameras are very complex and provide opportunities that are completely

corresponding to the capabilities of professional cameras. In the hands of a non-professional user, these capabilities are a drawback, since it is often the simultaneous use of all intraframe editing techniques that are inconsistent with each other, which subsequently significantly complicates the task of interframe editing. Mobile phone camera lenses currently provide only the most basic features, usually limited to "digital zoom", i.e. simulation at the camera software level of the effect of changing the focal length between the camera lenses, as well as software simulation of various filters. This leads to the fact that in the aspect of intraframe editing, filmed on a mobile Phone video is easier than video shot on a stationary camera. Individual files are more homogeneous in terms of the techniques used in them. This means that the task of classifying and recognizing such techniques, the initial sorting of the material, is simplified, since the sorting part, in fact, is performed by

user, adjusting to the limitations of his device.

The method of holding a mobile phone while shooting also imposes its specificity. All modern video formats were developed taking into account the fact that the scene being shot fits into a rectangle with a long side parallel to the horizon. The design of stationary cameras, as a rule, makes it difficult to shoot in a different orientation (with the short side parallel to the horizon). In turn, the design of phones usually involves shooting in an arbitrary orientation, both vertical and

horizontal. However, during installation, materials shot in different orientations must be aligned in orientation. This is a specific problem specific to mobile video.

However, the mobile phone has features not provided by the stationary camera. So, the phone can embed in the video material information about the geographical coordinates of the place where this video was shot. Some modern cameras that have the function of shooting video also have this feature. But the phone at the same time, in the presence of a built-in accelerometer, can embed information on the spatial orientation of the device when shooting in the video material, in addition to information about geographical coordinates. Currently, stationary cameras with similar functions are not manufactured.

Other are also the requirements for the format of the material obtained during the installation process. A mobile phone is not only the most common means of capturing a mobile video, but also the most common means of storing and viewing videos. Considering the disk limitations noted above space (memory size), this imposes its limitations not only on the duration of the source material, but also on the duration of the final installation result. In addition, a mobile phone, as a video viewer, does not provide sufficient viewing comfort, having a small screen and requiring the device to be kept on weight. In this regard, the viewer's ability to concentrate on the contents of the mobile video is limited, loss of interest in what is happening occurs in 2-3 minutes, due to which a rare mobile video exceeds the specified value by

duration. He-mobile video is usually much longer in time.

Thus, currently there are users on the market who need to install a large number of mobile video. Various methods of installation are used to convert the material. The use of installation methods spontaneously or fragmentarily does not allow to achieve the required quality

material conversion. To fully master the installation techniques, an unprofessional user needs to spend a lot of time and effort.

Traditionally, the user's need for editing was satisfied by using the skills of a professional editing director. Professional installation is carried out manually, using software, by personnel, the preparation of which combines the elements of training an engineer and an artist, and also includes long-term practice and development. Due to the complexity of training such specialists, it is impossible to manually edit all currently existing user videos. In addition, it cannot be carried out directly on a mobile device. A traditional alternative to manual installation is the use of automatic or semi-automatic installation methods.

In accordance with the foregoing, video editing solutions currently on the market can be divided into three groups. First one the largest group is professional digital non-linear editing tools (Avid Media Composer, Apple Final Cut Pro, Adobe Premiere Pro). The software products of this group use the latest available technologies, demanding on hardware and user skills. The solutions of this group are based on the interface concept of timeline, “time line” (publication US5999173, class IPC Gl 1B27 / 02, published 07.12.19,99), that is, a visual representation of the length of time corresponding to the mounted video film, consisting of several tracks containing audio or video information, as well as indications of various transformations (filters and effects) of this information. At the rendering stage, the timeline is converted by the editing program into a finished video.

The capabilities provided by software at this level are limited only by the user's qualifications, the requirements for which are extremely high.

The requirements for hardware at this level are so high that the implementation of such systems on mobile devices currently goes beyond the current technological advances.

The second group includes solutions that allow non-linear digital editing for an unprofessional user who has a certain level of training and has the hardware level corresponding to the average level of a home multimedia computer (Windows MovieMaker, Corel VideoStudio, Pinnacle Studio, Kino). The power of modern mobile phones is enough for mobile software to exist in this group (for example, there are programs for the Apple iPhone platform

ReelDirector, iMovie, Splice). Typically, a solution at this level provides the ability to edit video and overlay multiple audio tracks using the simplest timeline-based interface and a number of predefined filters and effects with limited settings. However, platforms of this type are often difficult for a person who does not have sufficient knowledge and skills.

Also on the market there is a third group of solutions that provide the possibility of automatic or semi-automatic installation with minimal involvement in the user process (desktop applications muvee reveal, muvee autoProducer, Apple iMovie, CineMagic Roxio Easy Media Creator). Often, some products from the second group provide such opportunities (for example, Pinnacle Studio, Corel VideoStudio, Windows MovieMaker desktop applications). The functioning of the software of this group is due to the fact that the user indicates which video clips and music from the sets should be used in the process of automatically creating a movie, determines the duration of the output video file, selects one of several proposed design styles, and if necessary assigns the simplest effects (frame change, color correction , etc.) The installation process itself is carried out automatically, that is, the software itself selects which parts of the material should be included the final product, and the choice is usually made

randomly, in some cases, the user can influence

the preference of a particular material, marking it in a certain way.

It should be noted that from the point of view of technology, the points of application of efforts of manufacturers of this software are concentrated mainly in two segments: recognition of transition points in video and audio. The transition points in the video correspond to the moments of the frame change or a significant change in the shooting conditions (illumination, color palette). Audio Transition Points

formalized as rhythmically selected fragments (strong / weak beats), or fragments in which the tonality of a musical work changes, or fragments marked by a change in other characteristics. In accordance with some editing features, combining transition points in audio and video can sometimes create positive artistic effect, which is embodied by the manufacturers of this software. ,

However, as shown above, simply comparing the transition points in audio and video is not enough to create an audiovisual work in compliance with the editing laws.

Mobile software of this category, for example, the HighLightCam and Flixlab programs, in fact, are client parts of the client-server software, because for editing video material is uploaded to the server where it is processed. Thus, the key to their functioning is the constant availability of a high-speed Internet channel.

The Videolicious program performs installation locally, but despite the prostate of its use, even in comparison with HighLightCam and Flixlab, the quality of the final product deteriorates. In addition, the main purpose of Videolicious is the installation of commercials for advertising a local business, which narrows the scope of its application. All the mobile applications discussed above are distinguished by a low speed of operation, which is reflected in the fact that the user is forced to wait for the installation process to complete, and only then look at the resulting product. This approach is incompatible with the requirement of instant customer satisfaction, which is expected by the consumer of mobile

applications.

State of the art

The following patent documents describe how to create video files using an editing template or script. Such a scenario contains a timeline, divided into virtual cells. A fragment of the video can be placed in each cell. Such an implementation of the method is quite accessible, since all professional and semi-professional installation programs allow you to save such installation structures in the computer’s permanent memory in xml format. However for automatic installation of one such structure is not enough, so various implementations offer their own additions.

Chronologically, one of the first implementations of the installation script idea was a thematic template, which is designed to help novice movie editors quickly perform work on creating a video that perpetuates memory in particular of significant occasions, such as a birthday. The template for each of these cases will be configured with predefined frames that are annotated with descriptions of events that may occur during the indicated cases (US7352952, class MP Gl 1B27 / 031, published on April 1, 2008). However, this method does not automatically create a movie. The user must upload the video to each cell and ensure that the selection requirements of the cells are respected. In addition, there is no synchronization with music. The absence of a randomization element leads to the fact that the generated video files are too uniform.

Pat. No. US2005 / 0217462 contains a description of a method that actually has many similarities with the mounting script. However, in the specified method

cell durations are set in percent, and for each duration

minimum and maximum values are set. This is due to the fact that the installation is done for the music selected by the user, and the algorithm must find in the specified timeline interval a moment for gluing frames combined with the rhythmic factor. Since the smart script proposed in this document is based on pre-selected musical material, there is no need for this kind of fuzziness. At the same time, fuzziness in this case can prevent the creation of an integral impression, for which the coordinated work of all elements is important:

rhythmic, tonal, etc. This drawback becomes even more significant given the fact that the material in the cells is randomly selected, not taking into account the essential features (size, pan direction, etc.)

A template based on an already recorded music video is devoid of many of the above disadvantages (publication US7716572B2, cl. MnKG06F 17/00, publ. 1 May 1, 2010). Since the original video is usually created by specialists, if the user observes the size and nature of the camera’s movement, the final video will most likely produce a positive effect. However, in this implementation, only information about the time between gluings is stored, and a high level of user involvement in the process is required in order to achieve such an effect. In addition, fragments of the original music video remain in the final product, which greatly reduces the applicability of such a solution.

Many implementations try to avoid the above drawbacks by deriving a set of descriptors either directly from the user video, or by inviting the user to set the descriptors on their own, and by taking this

information in the production process. In particular, the device and method for the production of audiovisual materials (publication US8006186, class IPC Gl 1B27 / 034, publ. 12/22/2000) proposes to extract data from user video and audio. However, the specified metadata extraction has been described extensively and, in fact, was implemented at the time of publication in several products. In addition, the algorithm does not imply synchronization of video and audio based on the extracted metadata.

The existing solutions are fraught with a number of compromises, and the choice of any alternative does not lead to a result that satisfies all the above requirements. Attempts to fully automate the installation lead either to excessive schematization of the process, or to excessive randomization, in both cases to a lack of flexibility, to simplification of the final result, which contradicts the task of improving the quality of the presentation. Rejecting automation requires user intervention, which, 0121, in turn, assumes that such a user has specific

professional skills. And this contradicts the task of ensuring mass decision. In addition, all existing solutions do not fully take into account the specifics of mobile video material.

SUMMARY OF THE INVENTION

The technical result of the claimed method for the automatic installation of digital video files is that the claimed method should be implemented locally on a mobile device using only its computing resources, while taking into account the specific features of mobile amateur video, and also allow viewing modified video files instantly, without delay in preliminary calculations . In this case, the claimed method must take into account

the patterns of perception of the footage used in professional editing allow mass reuse, do not require any qualifications and special knowledge from the user, and also allow the user to quickly and flexibly select the level of manual intervention in the material processing process taking into account desires and possibilities. In addition, the claimed method should ensure that the video files are more compact both in terms of byte size and in terms of duration than the original video materials. Additionally, the claimed method should ensure the receipt of video files that are perceived as whole, complete works with unique characteristics, comfortable for viewing, which are the best from the point of view of influencing the viewer “presentation” of the event, as well as allowing the viewer to view the event when viewing.

The technical result is achieved by the fact that in the method of automatically editing digital video files, they create a digital editing smart script in the form of a digital data file that serves as a virtual representation of the editing time a line that defines the sequence in which video clips of a certain duration with certain characteristics are connected (mounted) with each other, and with audio clips of a certain duration with certain

characteristics; include in the digital editing smart script information about the sequence, durations and relative start time of the mounted video and audio segments, information about the sequence, types, durations and relative start time of conversions of the potential content of the video and audio segments, filters and effects, as well as information about masks metadata imposed on the mounted video and audio segments, and thus narrow the range of video and audio materials that can be mounted in a sequence second digital mounting smart scenario; create video material by capturing images on the built-in camera of a mobile phone or portable digital camera; save video material on a mobile phone, portable digital camera, portable file transfer device, computer device or DVD-ROM; create a list of the mentioned video materials for editing, for

use as source materials for editing a digital video file; extract metadata from the aforementioned video material, including auxiliary metadata about the place and date of shooting, GPS coordinates, the direction of movement of the objects in relation to the plane of the camera lens or the optical axes of the camera, the direction of movement of the camera relative to the recorded scene, about the type

fixed objects, about the direction of the shooting vector, about the illumination and color saturation of the scene, about the sound background of the scene in them, while the metadata is not

limited to auxiliary metadata; introduce additional metadata about the video material; save metadata, auxiliary and additional metadata in the form of a set of digital data files, select video fragments so that the duration of the video fragment meets the requirements of the slot for U2013 / 000121 the duration of the fragment, while maximizing the assessment of the correspondence of the metadata of the selected fragment of the video material to the mask of the video tag of the smart script, assembly

a digital film from fragments of digital video files mounted in accordance with the metadata and installation instructions extracted from the digital editing script; placing said video fragments in each video tag of said smart script; save the received digital video file.

In addition, the technical result is achieved by the fact that in the specified method, a digital editing smart script is created in the form of an xml file, and this includes a set of special tags, each of which has its own parameters for the duration and relative start time, counted from the first millisecond smart -scenario, in addition, each tag has an indication of the type of tag, including video tag, audio tag, effect tag or meta tag, also each tag has tag-specific tag parameters and predefined tag values, or expression an event or algorithm by which the value of a tag parameter is calculated based on the values

parameters of other smartscript tags and input information specified at the installation stage. Moreover, the mentioned tags may include links to existing fragments of audio and video materials.

In addition, the technical result is achieved by the fact that when removing

metadata from the video material perform: automatic recognition of types of camera movement, object movement in the frame, plan, including, but not limited to, the middle plan, close-up and general plan, plan objects, including, but not limited to, the types of "man", "animal" , "Landscape", "interior", "exterior", and / or perform

automatic recognition of human speech and other sounds in the frame, convert the mentioned sounds into the corresponding text, and highlight keywords; and / or retrieving tags specific to the digital video file format, including, but not limited to geotags, the date the digital video file was created. one

In addition, the technical result is achieved by the fact that to enter additional metadata, at least one of the following is performed:

• presenting to the user a list of keywords for the user to select those keywords that best describe the contents of the video material or its individual fragments;

• presentation to the user of a hierarchy of thematic categories for placement by the user of each, several, or all video materials in one or more of the presented categories;

• presentation to the user of an input form for arbitrary keywords describing the contents of the video materials;

• providing the user with instructions and the recommended sequence of actions when creating video materials, in the form of text, sound or images, including text, sound and images transmitted to the user at the time of creation of the digital video material;

• registration of the position of the plane of the camera lens in three-dimensional space, carried out during the shooting of the digital video file by recording the output information of the position sensors in space, built-in device for shooting video files, including, but not limited to mobile phones or digital cameras.

In addition, the technical result is achieved in that the selection of a fragment of the video material for its placement in the video tag of the smart script is carried out by at least one of the following methods:

• random selection of one part of a digital video file from all parts of a digital video file that meet the above requirements of the slot in terms of its parameters;

• selection of the desired part of the digital video file; • the choice of a subset of fragments of video material such that

maximizes compliance with specified requirements using well-known procedures for finding a local maximum in the n-dimensional plane, including, but not limited to methods of genetic search, gradient descent, and simulated annealing.

In addition, the technical result is achieved by the fact that digital editing smart scripts, metadata, auxiliary and additional metadata, digital video files are stored on a computer device or portable file transfer device, or mobile phone, or digital camera, and then transferred via communication networks to another a computer device or portable file transfer device, or a mobile phone, or a digital camera.

Brief Description of the Drawings

The claimed invention is illustrated using the drawings shown in FIG.

1 - 12.

Moreover, in FIG. 1 and 2 show a diagram for creating an assembly smart script;

in FIG. 3 to 5 show a method for extracting metadata from video materials;

in FIG. 6 shows a method for adding user metadata;

in FIG. 7 shows a method for selecting material;

in FIG. 8 shows a method for defining timing;

in FIG. 9 shows a method for placing video fragments in video tags; in FIG. 10 and 1 1 show a method of mounting audio files into a video file;

in FIG. 12 shows a method of adding effects to a video file.

Disclosure of invention The main step of the claimed method is to create a smart script. Structurally, the smart script is a time line equal in time to the planned duration of the video file. The first step in creating a new smart script is to determine timing.

Timeline is a metaphor for a unit of video material in terms of time span, successive change of various audio fragments, successive change of various video fragments, with simultaneous audio and video component.

An infinite number of video files can be created on the basis of one timeline, provided that there is an infinite number of video fragments such that many video clips are larger than many video files.

Along the timeline, the director arranges various reserved words (hereinafter referred to as tags), assigning different values to their parameters. There are required and optional parameters. The duration of the impact of tags on the timeline is determined by a special duration parameter, its value is set in milliseconds. The beginning of the tag impact is determined by the offset parameter, which sets the start point of the tag as the offset in milliseconds relative to the start of the timeline. These two parameters are required for all tags. The tag duration cannot exceed the total timeline of the timeline. The base tag, and at the same time the structural unit of the timeline, is the video slot tag. The video slot tag determines the sequence and duration of video clips.

The total duration of the video slot tags cannot be less than the timing of the editing script. Similarly, the audio slot tag defines the sequence and duration of audio fragments of a material.

The video slot and audio slot tags are structural for the timeline, which means that the beginning and end of any other tag on the timeline must match the beginning and / or end of the action of at least one audio and / or video slot. At the same time, the beginning and end of the video or audio slot, in turn, do not have to coincide with the beginning and / or end of other tags, i.e. multiple audio and / or video slots can be grouped by any tag.

A tag parameter can contain a numeric or string value, as well as a fragment of the interpreted code, the interpretation of which results in a numeric or string value. Some string values are keywords for the interpreter, that is, they have a reserved meaning, understandable in the context of operations of transforming user material by the algorithm.

Tags can describe various image and sound parameters:

• view of the plan and the nature of the movement of the camera / objects: size of the plan, direction and speed of the camera, etc.

• characteristics of the plan on the content side - the keywords "people", "landscape", etc.

• physical characteristics of the image, which can be obtained by analyzing the image by well-known algorithms: illumination, spectrum, etc.

• additional characteristics generated by the shooting device at the time of shooting, if such capabilities are provided by the device and / or video format (such characteristics may include, for example, date, GPS coordinates, data of the built-in accelerometer, etc.)

• any rhythmic or altitude-dynamic characteristics of the audio track.

Each segment of the editing script can contain any number of any tags. In the future, when processing a custom video, the presence of two mutually exclusive tags in one segment (for example, tags "close-up" and “medium plan”) means that a plan corresponding to any of the tags may equally fall into the segment.

When creating a smart script, you can proceed from various premises. The task is to combine the factors of interframe, intraframe, and vertical editing into a single system that interacts with the audience in the rhythmic, metric, tonal, and associative-intellectual aspects, and creates a predetermined psycho-emotional effect.

Then the user data is processed and the metadata necessary for the smart script for editing is created. Metadata is extracted in the background, immediately after launching the application and at any time when the processor of the mobile device is not busy with the calculation of the material, thereby achieving the effect of "instant editing".

In addition to video materials (1 1) and audio materials (12), the user's material, as a rule, already contains auxiliary metadata, for example, the date - information about the shooting time (13), which the algorithm can also use. In addition, in each video clip, the definition of the place where the video was filmed is often used - geotagging (14). The user’s original video may also contain other data that can be used (for example, data characterizing the spatial position of the device at the time of video recording).

Using algorithms known from the prior art, their combinations and improvements, taking into account the capabilities of modern digital technology (for example, similar to those described in Determining Optical Flow, Berthold K.R. Horn, Brian G. Rhunck and An incremental network for on-line unsupervised classification and topology learning Shen Furaoa and Osamu Hasegawa) a program in a mobile communication device can recognize various types of objects in the frame, for example, faces, streets, and a number of other images that are present on the video (15,16,17,18). So one way to extract metadata is a video analysis algorithm, based on which tags can be assigned to user data.

In addition to recognizing faces, streets and other images that are in the frame, the program can determine the direction of movement in the frame. For example, in FIG. Figure 5 shows a panorama from left to right, and the program, analyzing the corresponding video fragment, assigns the “panorama from left to right” tag to this fragment.

By transmitting a video fragment to the input of the algorithm, the user can independently enter additional metadata characterizing it. In the original video, he can mark additional parameters in “out” and “in” out points, which will allow the algorithm to assemble a better quality video. For example, he marks a fragment from 0:00:00 to 0:01:15 as a “close-up” (23). From 0:01: 15 to 0:02:17 (25) he marks it as an “average plan”. From 0:02:17 to 0:03: 1 1 he marks it as a “general plan” (26). Some metadata may overlap with other metadata, for example, in the close up (23) and middle (25) metadata, in each case there are people on the user's video, and he can mark that these are people. Thus, this metadata intersects both middle (25) and close up (29). At the same time, in contrast to the editing script, mutually exclusive tags cannot be assigned to the user video, because they characterize what is already actually in the video, and not what could potentially be there, as is the case with the editing script.

If the user has a fragment of the video that he would like to receive in the clip with sound, and if the editing smart script provides such an opportunity, the user can mark this fragment as “use with sound” (27), (28). Accordingly, this video fragment along with the audio track will be taken into editing. You can use other methods of receiving metadata from the user: information about the persons depicted on the video, objects, meaningful and emotional characteristics in the form of ordinary words, etc.

Next, determine the video, shot in a vertical orientation, and crop it so that the remaining fragment is placed in a rectangle of standard shape. If desired, the user can independently specify which part of the vertically oriented video should be transmitted for editing. In addition, at this stage, the user can manually trim the source material, reducing the timing of the material transferred for installation.

As described earlier, each segment of the editing smart script has certain metadata formalized in the form of tags: size, additional parameters, etc. At the first stage, the algorithm summarizes the lengths of all metadata of the user's materials and checks the sufficiency of the material based on the requirements specified by the length tags of the smart script. For example, timing is considered separately for all close-ups, all general plans, all middle plans (see Fig. 8, in which (31) - editing script, (32) - clips, (34) - tags in clips, (33) - main audio track, (35) - list of tags, (36) - number of turns on each tag, (37) - number of seconds, how many clips with this tag are used in the editing script).

An example of automatic editing of digital video files. The algorithm refers to the first video tag from the tags (39) of the mounting smart script (38), and checks which other tags (39) affect the timeline during this video tag. From these, a set of required tags and a set of optional tags (39) are assembled, which is a subset of the set of all optional tags (39) of a segment and can be generated by random selection or more complex algorithms.

The user video also contains various metadata tags defined by the algorithm or entered by the user. Some tags may contain weight, or the probabilistic part, for example, is noted by the degree of confidence in the recognized image, or the specific ratio of images of this type in the plan.

The selection of a user video fragment for placement in the editing smart script segment is carried out stochastically, taking into account the maximization of the matching of the user video fragment tags to the script tags (40). If such a fragment is found, then the fragment (45) is marked in the source material of the user (44), which is set to the time line in the editing script (39) - in order to no longer use this fragment for editing and avoid repeating plans.

If in the initial editing script there is a sound effect tag with an intermediate noise parameter - use of sound - then the material (see Fig. 9) is selected in such a way as to capture the mixes at the beginning and end, that is, mix the sound at the beginning and end (51 - mix). The tag indicates how many seconds you need to mix the sound (52). Accordingly, the audio track changes: in addition to the main audio track (50), the algorithm also uses an audio track from the user's source material (51).

In order to avoid duplication of material, each selected fragment (53) is marked as already used. If the search engine shown in FIG. 9, finds the plan (54), which in the region (55) intersects with (53), then this plan is not valid. And the plan (56), which does not intersect with any other fragment, is valid, and it can be used to create a timeline from this material.

In an editing scenario, such inclusions as “processing of user material” are possible, for example, in FIG. 12 in the editing script (57) with an audio track (59) and with the user's audio (60), the tag “black and white” (61) was encountered. The program processes the user's material in accordance with this tag, in this case - makes it black and white. The parameter "speed = 0" means that during installation throughout the action of this tag there should be a freeze frame (62). In the editing scenario, it is possible to use captions (58), which the user can change as he wishes. If such a plan is encountered in the editing script, then the user always has the opportunity to change the text that is in this fragment of the editing script.

Although, taking into account the above description, only one embodiment of the claimed method can be made, one skilled in the art can implement various variants of the method based on the above description.

Thus, as follows from the above description, the claimed method is implemented locally on a mobile device using only its computing resources, and at the same time, all the specific features of mobile amateur video are taken into account, and it is also possible to view modified video files instantly, without delay in preliminary calculations. Moreover, the claimed method takes into account the patterns of perception of the footage used in professional editing, allows mass reuse, does not require any qualifications or special knowledge from the user, and also allows the user to quickly and flexibly select the level of manual intervention in the process of processing the material, taking into account the desire and of opportunities. In addition, the claimed method fully ensures the production of video files more compact both in terms of byte size and in terms of duration than the original video materials. In addition, the claimed method provides video files that are perceived as whole, complete works with unique characteristics, comfortable for viewing, which are the best from the point of view of influencing the viewer “presentation” of the event, as well as allowing the viewer to view the event when viewing.

Claims

CLAIM

1. A method for automatically editing digital video files, in which

create a digital montage smart script in the form of a digital data file that serves as a virtual representation of the montage time line defining

the sequence in which video clips of a certain duration with

by certain characteristics they are connected (mounted) among themselves, and with audio fragments of a certain duration with certain characteristics; include digital editing smart script information about

sequence, durations and relative start time of mounted video and audio segments, information about the sequence, types, durations and relative start time of transformations of potential contents of video and audio segments, filters and effects, as well as information about metadata masks superimposed on mounted video and audio segments , and thus narrow the range of video and audio materials that can be mounted in the sequence specified by the digital editing smart script;

create video material by capturing images on the built-in camera of a mobile phone or portable digital camera;

save video material on a mobile phone, portable digital camera, portable file transfer device, computer device or DVD;

create a list of the mentioned video materials for editing, for use as source materials for editing a digital video file;

extract metadata from the aforementioned video material, including

auxiliary metadata about the location and date of shooting, GPS coordinates, the direction of movement of the objects of the shooting relative to the plane of the camera lens or the optical axes of the camera, the direction of movement of the camera relative to the fixed scene, about the type fixed objects, about the direction of the shooting vector, about the illumination and color saturation of the scene, about the sound background of the scene in them, while the metadata is not

limited to auxiliary metadata;

introduce additional metadata about the video material;

save metadata, auxiliary and additional metadata in the form of a set of digital data files;

fragments of video material are selected in such a way that the duration of the video fragment matches the slot requirements for the fragment duration, while maximizing the compliance of the metadata of the selected fragment of the video material with the smart script video tag mask, assembling a digital film from fragments of digital video files mounted in accordance with the metadata and editing instructions extracted from the digital editing script;

place said fragments of video material in each video tag

the mentioned smart script;

save the received digital video file.

2. The method according to claim 1, in which they create a digital editing smart script in the form of an xml file, and include a set of special tags, each of which has its own parameters for the duration and relative start time, counted from the first millisecond of the smart script in addition, each tag has an indication of the type of tag, including video tag, audio tag, effect tag or meta tag, also each tag has tag specific parameters and preset tag values, or an expression or algorithm by which to calculate the value of the tag parameter based on the parameter values of other smart script tags and the input information specified at the installation stage.

3. The method according to claim 2, in which the mentioned tags include links to already

existing fragments of audio and video materials.

4. The method according to claim 1, in which when extracting metadata from the video material perform:

automatic recognition of types of camera movement, movement

the object in the frame, the plan, including, but not limited to, the average plan,

a close-up and general plan of objects of the plan, including, but not limited to, the types of “man”, “animal”, “landscape”, “interior”, “exterior”, and / or perform automatic recognition of human speech and other sounds that sound in frame, the mentioned sounds are transformed into the corresponding text, and keywords are highlighted; and / or

retrieve tags specific to the digital video file format, including, but not limited to geotags, the date the digital video file was created.

4. The method according to claim 1, in which at least one of the following steps is performed to enter additional metadata:

presenting to the user a list of keywords for the user to select those keywords that best describe the contents of the video material or its individual fragments;

presentation to the user of a hierarchy of thematic categories for the user to place each, several, or all video materials in one or more of the presented categories;

presentation to the user of an input form for arbitrary keywords describing the contents of the video materials;

providing the user with instructions and the recommended sequence of actions when creating video materials, in the form of text, sound or images, including text, sound and images transmitted to the user at the time of creation of the digital video material; registration of the position of the plane of the camera lens in three-dimensional space, carried out during the shooting of a digital video file by recording the output information of the position sensors in space, built-in device for shooting video files, including, but not limited to mobile phones or digital cameras.

5. The method according to claim 1, in which the selection of a fragment of the video material for its placement in the video tag of the smart script is carried out by at least one of the following methods:

a random selection of one part of the digital video file from all parts of the digital video file that meet the specified requirements of the slot in terms of their parameters; selecting the desired part of the digital video file;

selecting from a subset of fragments of video material such that maximizes compliance with the given requirements using well-known procedures for finding a local maximum in the n-dimensional plane, including, but not limited to methods of genetic search, gradient descent, and simulated annealing.

6. The method according to claim 1, in which digital editing smart scripts, metadata, auxiliary and additional metadata, digital video files are stored on a computer device or portable file transfer device, or mobile phone, or digital camera, and then transmitted over communication networks to another computer device or portable file transfer device, or mobile phone, or digital camera.