GB2446832A - A file server system - Google Patents

Abstract

A file server system 100 for providing user access to data stored in a storage means includes a database 400 and a high capacity store 300 (e.g. archived video image data). The system 100 includes means interacting with the database 400 to present to a user file identifiers indicative of virtual data files (e.g. video clips) apparently available from the system though not assembled into files, and responds to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database 400 and the high capacity store 300 a file containing the selected data. The server may be configured to offer virtual files in a plurality of forms, including at least one of: different file formats, different essence compressions, different essence sizes and different sample rates. In a preferred embodiment, the system detects when an ordered sequence of files is being requested by the user and predicts the opening of the next file in the sequence, thereby enabling the next file to be prepared for user access prior to the receipt by the system of a request therefor from the user.

Description

File Server System This invention relates to file server systems, and

it relates more especially, though not exclusively, to such systems for providing access for a plurality of client users to video image data archived in a high capacity store, such as a multi-disk store.

Massive amounts of such data are available from service providers and, in principle, such data is generally accessible to plural client users.

However, because of the overall magnitude of the stored data, difficulties arise in presenting it to the client users in a manner which is economical in terms of file storage and yet provides convenient, rapid and user-friendly file identification and selection facilities capable of affording rapid access to selected files.

The present invention aims to reduce or overcome such difficulties and, in accordance with the invention there is provided a file server system for providing user access to data stored in a storage means including a database and a high capacity store, the system comprising mapping means adapted to interact with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files, and means for responding to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data. The virtual files can alternatively be regarded as pseudo files.

Preferred embodiments of the invention permit the data stored in the storage means to be made available to commercial off the shelf (COTS) operating systems as files and folders, thereby enabling unmodified third party applications to access and modify the selected data.

This is achieved, in one preferred embodiment, by attaching an implementation of the Common Internet File System (CIFS) file sharing protocol to a storage layer associated with the storage means.

The server system is, in one embodiment, configured such that the virtual files advertised as available by the server do not, when advertised, exist as files on any block storage device anywhere in the system. The files are pseudo files, in the sense that they are offered by the server, but their contents are not constructed, from essence data (e.g. video data) archived on disk, and associated metadata information stored in a database, unless and until they are requested of the server.

This expedient allows the server to offer files in any form the service provider cares to deliver, including (but not limited to) different file formats, different essence compressions, different essence sizes and different sample rates. The server appears to have all of the various versions of the files available instantly, but in fact only constructs the file payload in RAM storage as client devices request the file contents.

The invention thus allows the service provider's bespoke hardware to assist with the construction of the file parts in RAM, thereby facilitating the exploitation, where the essence data includes video image data, of picture sizing, colour space conversion and bit packing facilities, in a way that is completely hidden from the third party applications or operating systems.

The invention thus permits the integration into a COTS operating system of virtual files that do not exist in the presented binary form anywhere on a block mode storage device and are only constructed on request.

Preferred embodiments of the invention allow arbitrary (even off machine) processing to occur during the file delivery process. For example, the file header may be generated from data stored in a database and the file payload assembled from elsewhere, including reading from block mode storage and arbitrary processing in hardware.

In some embodiments, the virtual files as advertised on the server may not even be fully deliverable when advertised. In a live recording scenario, for example, two hours of media might be represented (or advertised) as a large file, but there might be an internal high water mark, representing the current progress of the record head, which prevents full media delivery until the requested essence data, representing the as-yet unrecorded part of the file is received within the system.

Another way of looking at this mechanism is random access delivery of streaming media. A file offers random access to arbitrary byte ranges. If the essence offered within the file is of fixed I predictable size (as in uncompressed video pictures, or fixed size I frame compression), there is a simple relationship between the byte offset and the requested picture essence. Thus a file part can be built and delivered, without all the file contents being either available or formatted in RAM. This allows storage level mutation to be hidden from third party COTS operating systems and applications. The service provider can deal with recoding semantics privately, whilst offering simple, non mutating semantics to legacy applications.

Accordingly, large live records can be streamed in a transcoded state to any application, with very low latency between the record head and the receiving device, as indicated schematically in the following table.

Time Block Mode Storage Request Offering single frames I thu nks o transcoder as virtual files Path j Accepting transcoded frames / (hunks back into file system Delivering random access sub-parts of a large file transcoded in ________ arbitrary_ways.

This embodiment offers a file, the entire essence for which may or may not be present in the system. A third party application can ask for a part of the file. The system's server uses the byte range request to calculate the required frame range, and then offers the required media to be passed through a separate third party transcode engine (using scripting or IT automation). On receipt of the newly transcoded payload, the server formats the results, enabling the original byte range request to be honoured.

The transcoded media may be cached in a storage system by the data owner, or not, as the context dictates.

Some at least of the virtual files offered by the system may not have any part of their essence data within the service provider's storage system.

The server system can access other CIFS shares using standard operating system support services and any convenient operating protocol.

In some embodiments, for example by "mounting" files into the service provider's clips database, the system can appear to have within it alien media that is not present on the service provider's storage. This alien media can then be offered in all the various forms described hereinbefore.

Further, such embodiments allow the service provider's storage system to both cache and, if necessary, pre-cue essence media (i.e. to detect the opening of an ordered sequence of files and to predict from that sequence subsequent files likely to be opened; as described hereinafter) as it is requested.

In some embodiments, each version of a file does not necessarily require its own space on a block mode storage device. In fact, the server can create as many copies and versions of an essence file as the user requires, without increasing block mode disk space usage at all. In a practice used for many years by the applicants and known as "Frame Magic"; a recent development of which is described in our co-pending UK patent Jo application No. 0617526.9, the system reference counts the number of times a piece of essence data has been (potentially) used by the system, thus protecting the essence data from deletion until all references to the media have been finished with.

The system is capable of delivering each virtual file as advertised, and even allows files to be overwritten, using a copy -on -write based protection mechanism. Further, this embodiment of the invention allows for a number of different Network Attached Storage (NAS) heads (i.e. more than one server) to offer different file versions of media that is stored only once on a block storage device.

Embodiments of the invention have been configured to support the delivery of sequences of image files to third party applications. A typical example of the usage of such embodiments arises where a third party application attempts to play a sequence of video frames in order, to give their user the perception of a moving image. In such embodiments, as an application opens files, the server maintains a short history of the files that have been opened. If it appears that the application is opening an ordered sequence of files, the server intelligently predicts that the application is likely to open the next frame in the sequence, and will prepare the file in RAM ahead of that event; such advanced preparation of files being conveniently referred to as "heuristic pre-cuing".

Such embodiments can support massive deliverable bandwidths, as the read requests coming from the application can be responded to with very low latency. The network and disk systems are dc-coupled by using a RAM cache and the heuristic pre-cue to preload the frames into the cache.

If multiple servers are combined to offer aggregate bandwidth, using a round robin load balancing scheme, the heuristic pre-cue detects that the

S

sequence requested is in order but not adjacent. Therefore a series can be inferred, and the pre-cue of the correct predicted frame can be computed.

Such embodiments also permit a single server to accommodate a situation in which a third party player attempting to play faster than normal play speed, and therefore is skipping frames in the sequence in a regular pattern. This server side operation is quite independent of the client side approach to playing sequences of files as a video sequence. Thus, the approach chosen by the client side device to cache, pre-cue or read ahead is irrelevant to these embodiments of the invention.

This heuristic pre-cuing process is believed to be inventive of itself, but in any event it works well in combination with the file virtualisation mechanisms described hereinbefore.

Shared file systems normally have to impose a file level locking scheme to avoid changes to files adversely affecting the performance of other applications. Certain embodiments of the present invention, however, have been advantageously configured to exploit the fact that most essence does not change most of the time. In such embodiments, provided that the server system operates in an environment utilising non-destructive editing systems, all that can generally happen to a shared media file after it is created is that it can be deleted.

Thus, provided that the system is configured such that media that requires mutation can not be shared, a lock free file system can be utilised; thereby simplif'ing the shared file system semantics.

This has the following consequences: All shared media is immutable.

All immutable media is inherently suitable for caching by third party clients.

The file systems allow individual users to take copies of folders, so that they can guarantee the media will not be deleted whilst they require it.

Therefore the file system allows for lifetime control by appearing to allow users to take their own copies.

There are also some simple use cases where media does require mutation.

In these cases it is reasonable to restrict the media to private views, where only that user is able to view the mutating media. The server uses the user's identity supplied by the CIFS protocol to avoid exposing the private copy to any other users.

In order that the invention may be clearly understood and readily carried into effect, one embodiment thereof will now be described, by way of example only, with reference to the accompanying drawing, the single figure of which shows, schematically and in flow diagrammatic form, the operation of a system in accordance with one embodiment of the invention.

Referring now to Figure 1, a server system in accordance with one example of the invention is indicated schematically at 100.

In response to a connection request 101 submitted by a client user, the system 100 initiates at 201 the appropriate data structures allowing the user to interact with the system, and presenting an introductory screen display, from which the user can request at 102 a desired operational access to folders of files within the system. The system responds at 202 by providing a list of relevant folders held in a database 400; the folders comprising identifiers of video clips in this example, and the user responds at 103 by selecting a folder (i.e. a clip in this example) in response to which the system accesses the database 400 to derive metadata indicative of the file content of the selected folder for display at step 203.

The system, as previously mentioned, does not have these files as such already constructed and available for downloading to the user, but all of the relevant material needed to construct the requested files is to hand in a high capacity video disk store 300, which stores (in this example) video image data, and the database 400, which inter alia stores the metadata information associated with, and determinative of the content, size and other characteristics of, the video image data.

The user is thus, at 104, presented with a list of files which are advertised as available by the system 100; these being virtual, or pseudo, files, in the sense that they may not yet have been constructed and thus may not exist as such in any tangible form or in any storage location to which the system 100 has access. The system 100 notes at 204 the specific file selection made by the user and initialises internal data structures, as well as noting any request at 104 for particular operational requirements, such as sizing or formatting, and thereafter assembles at 205 the associated image data and metadata from the stores 300 and 400 into the requested files, which are presented in the requested size and format and read by the user at 105.

Once all requests of the user have been met, the communication with the system is closed at 106 by the user and internal data structures allocated at 104 are tidied.

The process of assembling files is implemented sufficiently rapidly that the user is unaware of the initial non-availability of the requested files as entities. Thus, the system presents effectively a listing of virtual files which are only constructed as and when requested by a client, thereby avoiding the need to construct and retain on local storage massive files which may be only rarely called for.

It will be appreciated that the system can effect reformatting or other reconfiguration of the source data held in the high density store 300 in accordance with client requests at a convenient point in the process of assembling the requested files. Thus, the source data can be advertised by the server system not only as being available as such, but also as being simultaneously available in plural formats; the particular source data and format being selected and engineered only after a specific client request for that material in that format has been received.

Sophisticated functionality, such as the pre-cuing facility described hereinbefore, can be implemented by suitable detection and responsive functions incorporated into the processes for recovering data from the stores 300 and 400.

Claims (18)

1. Claims: 1. A file server system for providing user access to data

   stored in a storage means including a database and a high capacity store, the system comprising mapping means adapted to interact with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files, and means for responding to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data.
2. 2. A system according to claim 1, comprising means for making the data stored in the storage means available to commercial off the shelf (COTS) operating systems as files and folders.
3. 3. A system according to claim 2, wherein an implementation of the Common Internet File System (CIFS) file sharing protocol is attached to a storage layer associated with the storage means.
4. 4. A system according to any preceding claim, configured such that the contents of selected files are constructed, from video image data archived in the high capacity store, and associated metadata information stored in the database, when requested of the server.
5. 5. A system according to claim 4, wherein the server is configured to offer virtual files in a plurality of forms, including at least one of: different file formats, different essence compressions, different essence sizes and different sample rates.
6. 6. A system according to claim 5, wherein a file header is generated from data stored in the database and the file payload assembled from the high capacity store and means are provided for subjecting the data to arbitrary processing in hardware.
7. 7. A system according to any preceding claim, wherein at least one virtual file advertised as available on the server is not fully deliverable when advertised.
8. 8. A system according to claim 7, wherein, in response to a user request for an incomplete virtual file, the server uses a byte range specified in the request to calculate the required frame range, arid then offers the required media to be passed through a separate third party transcoding engine.
9. 9. A system according to claim 8, wherein transcoded media is cached in a storage system.
10. 10. A system according to any preceding claim, wherein at least one virtual file offered by the system does not have any part of its essence data within the storage means of the server system.
11. 11. A system according to claim 10, configured to access other CIFS data.
12. 12. A system according to any preceding claim, further comprising means for managing the storage means such that each version of a file does not require its own space in the high capacity store.
13. 13. A system according to claim 12, wherein said means for managing comprises reference counter means for counting the number of times a piece of essence data has been requested by users of the system, and means for protecting the essence data from deletion until all usages thereof are concluded.
14. 14. A system according to any preceding claim, configured to support the delivery of sequences of image files to third party applications, including memory means for maintaining a history of files that have been opened; means for utilising said history to assess whether an ordered sequence of files is being opened; means for predicting the opening of the next file in the sequence, and means for preparing said file for delivery prior to the receipt by the system of a request therefor.
15. 15. A system according to claim 14, further comprising a RAM cache configured to store said predicted and prepared files.
16. 16. A system according to any preceding claim, wherein the server system operates in an environment utilising non-destructive editing systems, and wherein the system is configured as a lock free file system.
17. 17. A file server system for providing user access upon request to data files stored in a storage means; the system including detecting means for reviewing information associated with files currently open to a user and for producing a detection signal when an ordered sequence of files is being requested by the user; predictor means for responding to said detection signal to predict the opening of the next file in the sequence, and preparing means for preparing said next file for user access prior to the receipt by the system of a request therefor from the user.
18. 18. A file server system substantially as herein described with reference to andlor as shown in the accompanying drawing.