KR20160107158A - Apparatus and method for routing information in a non-volatile memory-based storage device - Google Patents

Abstract

Various systems, methods, devices, and computer-readable media for accessing a storage device are described. In certain exemplary implementations, an active / active fault tolerant storage device including two or more controllers may be implemented. In an aspect, each controller may have two or more processing entities that distribute processing of I / O requests. In one implementation, the components, modules, and configuration of the controller board can be arranged in a manner that increases heat dissipation, reduces power consumption, spreads power and workload, and reduces latency. In one implementation, each controller may be coupled to a non-volatile memory (NVM) blade that includes a non-volatile memory (NVM) storage medium. In one exemplary implementation, a standardization protocol, such as the Peripheral Component Interconnect Express protocol, may be used to communicate between the various components of the controller and also the NVM storage medium.

Description

Field of the Invention [0001] The present invention relates generally to a non-volatile memory based storage device, and more particularly,

Aspects of the present disclosure relate to computing and communication technologies. In particular, aspects of the present disclosure are directed to systems, methods, apparatus, and computer readable media for improving the performance of a storage device.

Storage devices for enterprise systems require mass storage capacity, low latency for reading and writing to storage devices, high bandwidth, low power consumption, and reliability. Traditionally, enterprise systems are implemented using media such as hard disk drives (HDDs) that hold data while power is turned off. A hard disk drive is a data storage device used to store and retrieve digital information using a fast spinning disk. The HDD consists of one or more rigid ("hard") rapidly rotating disks (platters) having magnetic heads arranged on the movable actuator arms for reading data and writing data to the disk surface. Due to the moving part, the HDDs are essentially prone to errors and failures and have a floor as to how low their access times and prices are.

Embodiments of the present invention solve these and other problems.

Various systems, methods, devices, and computer-readable media for accessing a storage medium are described. Techniques for optimally accessing the storage medium are described. In one implementation, the storage device may be implemented using nonvolatile memory (NVM).

In certain exemplary implementations, an active / active fault tolerant storage device including two or more controllers may be implemented. However, in other exemplary embodiments, an active / standby system may also be implemented. In some implementations, the controller may be implemented using an application specific integrated circuit (ASIC), field programmable gate array (FPGA), or any other technology that integrates the functionality of several distinct components onto a single die. In other implementations, the controller may also include a controller board having a plurality of discrete components. In an aspect, each controller board may have two or more processing entities that distribute processing of input / output (I / O) requests. In one implementation, the configuration of components, modules, and controller boards is arranged in a manner that increases heat dissipation to service I / O requests, reduces power consumption, spreads power and workload, and reduces latency .

In one implementation, each controller may be coupled to a non-volatile memory (NVM) blade that includes an NVM storage medium. Embodiments of the present invention may also provide additional enhancements to improve access time to NVM storage media. While some embodiments of the invention may be described herein using NVM storage media for illustrative purposes, in certain embodiments, the invention may not be limited to NVM storage media, and other suitable physical storage media may be used with the present invention Can be used without departing from the scope.

In one implementation, a standardization protocol such as the Peripheral Component Interconnect Express (PCIe) protocol may be used to communicate between various components of the controller board and also NVM storage media.

An exemplary storage device is a first routing entity from a plurality of routing entities coupled to a first blade from a plurality of blades, wherein the first blade can include an NVM storage medium, and the first blade coupled to the first routing entity The first processing entity may receive a first input / output (I / O) request, and the first data associated with the first I / O request may be coupled to the first routing entity, And to transmit the first data associated with the first I / O request to the first routing entity for storage of the first data on the first blade. The second processing entity may be coupled to the first routing entity, the second processing entity receiving the second I / O request, and the second data associated with the second I / O request being coupled to the first routing entity 1 blade and to transmit second data associated with the second I / O request to the first routing entity for storage of the second data on the first blade.

In some implementations, the storage device receives a third I / O request and the third I / O request is a read request for first data at a first location of the first blade coupled to the first routing entity And may further have a second processing entity configured to request the first data from the first location from the first routing entity and receive the first data from the first routing entity. In one implementation, the first processing entity and the second processing entity may be indirectly coupled to each other via the first routing entity. In one implementation, the controller board includes a first routing entity, a first processing entity, and a second processing entity. In one implementation, the first processing entity may be coupled to a first memory and the second processing entity may be coupled to a second memory. In one aspect, the transmission of data between the first processing entity and the first routing entity and the transmission of data between the second processing entity and the first routing entity are performed using a Peripheral Component Interconnect Express (PCIe) protocol.

In a particular implementation of the storage device, the storage device is a second routing entity from a plurality of routing entities coupled to a second blade from a plurality of blades, the second blade comprises an NVM storage medium, and the second routing entity The first processing entity may receive a third I / O request, and third data associated with the third I / O request may be stored on the second blade at a third location, And to transmit to the second routing entity third data associated with the third I / O request for storage of data associated with the third I / O request on the second blade. The storage device may further have a second processing entity coupled to a second routing entity, the second processing entity receiving a fourth I / O request, and wherein fourth data associated with the fourth I / To the second routing entity, fourth data associated with the fourth I / O request for storage of data associated with the fourth I / O request on the second blade .

In one implementation, the transmission of data between the first processing entity and the second routing entity and the transmission of data between the second processing entity and the second routing entity are performed using a Peripheral Component Interconnect Express (PCIe) protocol . In an aspect, a first I / O request received by a first processing entity may first be received at one or more interfacing entities and transmitted to the first processing entity via one of the plurality of routing entities.

An exemplary method of storing data includes receiving, at a first processing entity, a first I / O request, wherein at a first processing entity, first data associated with a first I / O request is coupled to a first routing entity Wherein the first blade includes an NVM storage medium, a second I / O module coupled to the first I / O module by a first processing entity for storage of data on the first blade, O request to a first routing entity, at a second processing entity, receiving a second I / O request, at a second processing entity, the first data associated with the second I / O request Determining that the second data is to be stored on the first blade at a second location coupled to the first routing entity, and determining that the second data associated with the second I / O request for storage of data on the first blade To the first routing entity System.

In some implementations, the exemplary method includes receiving, at a second processing entity, a third I / O request; at a second processing entity, a third I / O request is sent to the first routing entity Requesting first data from a first location from a first routing entity, and receiving first data from a first routing entity, .

In one implementation, the first processing entity and the second processing entity are indirectly coupled to each other through a routing entity. In one aspect, the controller board includes a first routing entity, a first processing entity, and a second processing entity. The first processing entity may be coupled to the first memory and the second processing entity may be coupled to the second memory. In some implementations of the method, the transmission of the first data between the first processing entity and the first routing entity and the transmission of the second data between the second processing entity and the first routing entity are performed by peripheral component interconnect (PCIe) Protocol. ≪ / RTI >

In a particular implementation of the method, the method further comprises: receiving, at the first processing entity, a third I / O request; at the first processing entity, the third data associated with the third I / Wherein the second blade includes a NVM storage medium, storing data associated with a third I / O request on a second blade in a second position Sending a third data associated with a third I / O request by the first processing entity to a second routing entity; receiving, at a second processing entity, a fourth I / O request; Determining that fourth data associated with a fourth I / O request is stored on a second blade at a fourth location coupled to a second routing entity, generating a fourth I / O request on the second blade, To the second processing entity for storage of the associated data. The fourth data are associated with 4 I / O request may include the step of transmitting to the second routing entity. In some implementations, the transmission of the third data between the first processing entity and the second routing entity and the transmission of the fourth data between the second processing entity and the second routing entity are performed using a Peripheral Component Interconnect Express (PCIe) protocol . The first blade may be one of a plurality of blades and the first routing entity may be one of a plurality of routing entities. The first packet received by the first processing entity may first be received at one or more interfacing entities and transmitted to the first processing entity via one of the plurality of routing entities.

The exemplary device comprises means for receiving a first I / O request, means for determining that first data associated with a first I / O request is stored on a first blade at a first location coupled to a first routing entity Wherein the first blade comprises means comprising an NVM storage medium, means for transmitting to the first routing entity first data associated with a first I / O request for storage of data on the first blade; Means for receiving a second I / O request; means for determining that second data associated with a second I / O request is stored on a second blade at a second location coupled to a second routing entity; And means for transmitting to the second routing entity second data associated with a second I / O request for storage of second data on the blade.

Various systems, methods, devices, and computer-readable media for accessing a storage medium are described. Techniques for optimally accessing the storage medium are described. In one implementation, the storage device may be implemented using nonvolatile memory (NVM) storage media.

In certain exemplary implementations, an active / active fault tolerant storage device including two or more controllers may be implemented. However, in other exemplary embodiments, an active / standby system may also be implemented. In some implementations, the controller may be implemented using an application specific integrated circuit (ASIC), field programmable gate array (FPGA), or any other technology that integrates the functionality of several distinct components onto a single die. In other implementations, the controller may also include a controller board having a plurality of discrete components. In an aspect, each controller board may have two or more processing entities that distribute processing of input / output (I / O) requests. In one implementation, the configuration of components, modules, and controller boards is arranged in a manner that increases heat dissipation to service I / O requests, reduces power consumption, spreads power and workload, and reduces latency .

In one implementation, each controller may be coupled to an NVM blade that includes an NVM storage medium. Embodiments of the present invention may also provide additional enhancements to improve access time to NVM storage media. Although some embodiments of the invention may be described herein using NVM storage media for illustrative purposes, in certain embodiments, the invention may not be limited to NVM storage media, and other suitable physical storage media may be used within the scope of the present invention Can be used without deviating from.

In one implementation, a standardization protocol such as the Peripheral Component Interconnect Express (PCIe) protocol may be used to communicate between various components of the controller board and also NVM storage media.

An exemplary storage device is a first controller configured to operate in an active mode, the first controller being configured to receive an input / output (I / O) request to store and retrieve data from an NVM storage medium, A second controller configured to receive I / O requests for storing and retrieving data from an NVM storage medium, and a plurality of NVM blades including an NVM storage medium, wherein the plurality of NVM blades At least one of which may include a storage device comprising a plurality of NVM blades coupled to a first controller and a second controller for storing and retrieving data from an NVM storage medium. In one implementation, at least one of the plurality of NVM blades includes a first routing interface in communication with the first controller and a second routing interface in communication with the second controller. In some implementations, the first routing interface can communicate with the first controller and the second routing interface can communicate with the second controller using the PCIe protocol.

In a particular implementation, for a read operation, the first controller receives the first I / O request and the first I / O request stores the first data associated with the first I / O request to the NVM storage medium Request and to transmit the command and the first data to at least one of the plurality of NVM blades to store the first data in the first location. In one implementation of the storage device, the first controller and the second controller may be configured to simultaneously decode I / O requests for read operations and request data from the NVM storage medium.

In a particular implementation, for a write operation, the second controller receives a second I / O request, and the second I / O request stores second data associated with the second I / O request on the NVM storage medium Request, and to transmit the command information associated with the second I / O request to the first controller. The first controller may be configured to receive the command information transmitted from the second controller and to transmit the storage command to at least one of the plurality of NVM blades. The second controller may be further configured to transmit the second data associated with the second I / O request to the one or more NVM blades.

In certain implementations, at least one of the plurality of NVM blades may include a first buffer coupled to the first routing interface for buffering instructions from the first controller. At least one of the plurality of NVM blades may be further configured to discard the command from the first controller if the first buffer is full beyond a predetermined threshold. In some implementations, at least one of the plurality of NVM blades may also include a command manager that mediates access to the NVM interface for commands from the first controller and the second controller. In the case where the command manager detects an error for the command, at least one NVM blade may send error information associated with the I / O request back to the commanded controller.

In some implementations, the first controller and the second controller may communicate fault tolerance information with each other. In an aspect, the first controller and the second controller may communicate fault tolerance information using non-PCIe bridges. In some instances, the fault tolerance information may include information about the failure of the first I / O request from the first controller to one of the plurality of NVM blades.

In one implementation, the first controller, the second controller, and the plurality of NVM blades may be coupled to a power rail, wherein the power rail is powered by a plurality of power sources. In one implementation, the first controller and the second controller may be a printed circuit board (PCB) that includes one or more processors for processing I / O requests and one or more routers for routing operations between a controller and a plurality of NVM blades have. In other implementations, the first controller and the second controller may be application specific integrated circuits (ASICs), each of which includes processing logic and routing logic.

An exemplary method for storing data on a storage device includes receiving a first I / O request at a slave controller, receiving a first data associated with a first I / O request with a first I / O request to an NVM storage medium Sending a command information associated with a first I / O request to a master controller, receiving command information transmitted from a slave controller at a master controller, 1 < / RTI > data from a slave controller to a first I / O request from a master controller and from a slave controller to at least one of a plurality of NVM blades including an NVM storage medium And transmitting the data.

The exemplary method includes receiving a second I / O request at the master controller, determining that the second I / O request is a request to store second data associated with the second I / O request to the NVM storage medium And transmitting the command and the second data to at least one of the plurality of NVM blades including the NVM storage medium to store the second data in the second location. The method includes receiving a second I / O request at a master controller, determining that a second I / O request is a request to read second data from a second location from an NVM storage medium, Retrieving second data associated with the second I / O request, receiving a third I / O request at the slave controller, receiving a third I / O request from the NVM storage medium from the third location, Reading from the NVM storage medium, and retrieving third data associated with the third I / O request from the NVM storage medium. In one implementation, the master and slave controllers can use the PCIe protocol to communicate with multiple NVM blades.

The foregoing has outlined rather broadly the exemplary features and technical advantages in order that the detailed description that follows may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be readily utilized as a basis for modifying or designing other structures to accomplish the same purpose of this disclosure. Such equivalent constructions do not depart from the spirit and scope of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS The features believed to be characteristic of the concepts disclosed herein, both as to organization and method of operation, as well as their associated advantages, will be better understood from the following description when considered in conjunction with the accompanying drawings. Each of which is provided for the purposes of illustration and description only and is not provided as a definition of the limitations of the claims.

The foregoing has outlined rather broadly the exemplary features and technical advantages in order that the detailed description that follows may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be readily utilized as a basis for modifying or designing other structures to accomplish the same purpose of this disclosure. Such equivalent constructions do not depart from the spirit and scope of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS The features believed to be characteristic of the concepts disclosed herein, both as to organization and method of operation, as well as their associated advantages, will be better understood from the following description when considered in conjunction with the accompanying drawings. Each of which is provided for the purposes of illustration and description only and is not provided as a definition of the limitations of the claims.

An aspect of the disclosure is illustrated by way of example. The following description is provided with reference to the drawings, in which like reference numerals are used to refer to like elements throughout. While various details of one or more of the techniques are set forth herein, other techniques are also possible. In some instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the various techniques.
A further understanding of the nature and advantages of the examples provided by this disclosure may be realized by reference to the remaining portions of the specification and the drawings, wherein like reference numerals are used throughout the several figures to refer to like elements. In some instances, a sub-label is associated with a reference number to represent one of a number of similar components. Where reference numerals are referenced in the existing sublabels without the specification, reference numerals refer to all such similar components.
Figure 1 illustrates an exemplary high-level block diagram of a storage device in accordance with an embodiment of the invention.
Figure 2 illustrates another exemplary block diagram of a storage device according to an embodiment of the invention.
Figure 3 illustrates another exemplary block diagram of a storage device according to an embodiment of the invention.
Figure 4 illustrates an exemplary block diagram of a storage device according to another embodiment of the present invention.
5 is a flow chart illustrating a method of performing an embodiment of the present invention in accordance with an embodiment of the present invention.
Figure 6 is a flow chart illustrating another method of performing an embodiment of the present invention in accordance with another embodiment of the present invention.
Figure 7 illustrates an exemplary block diagram of a controller board in accordance with an embodiment of the present invention.
Figure 8 illustrates an exemplary block diagram of address space for various components as seen by respective components on a controller board in accordance with at least one implementation of the present invention.
9 illustrates another exemplary high-level block diagram of a storage device according to an embodiment of the invention.
Figure 10 illustrates an exemplary block diagram of an NVM blade in accordance with an embodiment of the present invention.
11 illustrates an exemplary block diagram of a blade controller in accordance with an embodiment of the invention.
12 illustrates another exemplary block diagram of a blade controller in accordance with an embodiment of the present invention.
Figure 13 illustrates a computer system that performs an embodiment of the present invention.

Several exemplary implementations will now be described with reference to the accompanying drawings, which form a part thereof. Although specific embodiments in which one or more aspects of the disclosure may be implemented are described below, other implementations may be used and various modifications may be made without departing from the scope of the present disclosure or the scope of the appended claims.

Before discussing embodiments of the present invention, the description of some terms may be helpful in understanding embodiments of the present invention.

In some implementations, a "storage device" may include a computer system configured to store and retrieve data from a storage medium, as discussed herein. The computer system may be implemented using some or all of the components described with reference to FIG. In some implementations, the storage device may be used in a corporate environment or other similar environment having a need for accessing data over the network using a low latency and highly available link to the storage device. Lower power consumption, lower cost, and better heat dissipation may also be desirable from a storage device. In some implementations, the storage device may be a rackmountable device, and multiple storage devices may be deployed and maintained collectively. In other implementations, the storage device may be a stand-alone device. Although the storage device may have other peripherals and devices similar to conventional computer systems, in some implementations the storage device may be stripped down with a modular design that is optimized to minimize the use of physical space and energy, Server computer. The storage device also stores on the storage medium within the storage device to receive I / O requests, decode their I / O requests, and convert I / O requests into read, write, and configure instructions for the underlying physical medium, Lt; RTI ID = 0.0 > a < / RTI > file system software stack.

In some embodiments of the present invention, "flash storage media" may include non-volatile memory (NVM), as discussed herein. In some instances, an implementation of a storage device using an NVM may also be referred to as a solid state device. An exemplary implementation of an NVM-based device may include using NOR, NAND, Magnetoresistive RAM (MRAM), Ferroelectric RAM (FRAM), Resistive RAM (RRAM)), phase change memory or any other suitable technique, But is not limited thereto. NOR flash provides high-speed random access and allows data to be read and written to a specific memory location, e.g., to a single byte. The NAND flash can be read randomly, but typically processes data in small blocks that are sequentially written at high speed, called pages. The NAND flash can read faster than it records, thus quickly transferring the entire page of data. NOR Flash can act in the same way except that readings can be faster than NAND flash and the write can be slower. In general, NAND technology, which is less expensive than high density NOR flash, can provide higher capacities for the same size silicon.

In some implementations, embodiments of the present invention may utilize single level cell (SLC) NAND flash technology. In other implementations, embodiments of the invention may utilize multi-level cell (MLC) NAND flash storage media. MLC NAND is a flash memory technology that uses multiple levels per cell to allow more bits to be stored using the same number of transistors. In SLC NAND flash technology, each cell may be in one of two states, storing one bit of information per cell. Most MLC NAND flash memory technologies have four possible states per cell, so it can store two bits of information per cell. Using MLC NAND may be advantageous to reduce the cost per storage unit due to the higher data density.

As described herein, a "blade", "flash blade", or "NVM blade" may, in some implementations, refer to a grouping of one or more NVM chips together to provide storage, . An NVM blade may have a blade controller that mediates access to NVM storage media. The NVM blade controller may be responsible for receiving commands to access / store data on NVM storage media, processing instructions, and storing or retrieving data from NVM storage media. In one implementation, the NVM blade controller may be implemented using an application specific integrated circuit (ASIC). In other implementations, the NVM blade controller may be implemented using a field programmable gate array (FPGA).

As defined herein, a "controller board" includes a variety of hardware, firmware, and software components that receive I / O requests and convert their I / O requests into instructions to read, write, or configure NVM storage media . In one implementation, the controller board may be implemented using a printed circuit board (PCB), wherein the various components of the controller board are coupled to the board and can communicate with each other using a bus. In other implementations, other communication means, such as wireless, may be used to communicate between the components. Figure 7 is an exemplary implementation of a controller board. While embodiments of the invention may be described with respect to several discrete components, in some embodiments, the functionality of several discrete components may be performed by a single silicon die. For example, the functionality of a number of distinct components, such as processing and routing, may be implemented within an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a multi-chip module (MCM) Can be performed by a controller implemented in silicon technology. As described herein, in one implementation, a "controller board" may refer to several distinct components that implement a set of functions using a printed circuit board, while a "controller" For example, a PCB board with separate components) and a controller (e.g., functionality of several distinct components implemented in an ASIC, FPGA, etc.).

As discussed herein, a "processing entity" may refer to one or more physical or logical processors. The term " processing entity "or" processing complex "may be used interchangeably throughout this specification without departing from the scope of the present invention. For example, processing entities may include dual-core, quad-core, or multicore processors from vendors such as Intel, Qualcomm, and Tyler. The processing entity may execute the file system software stack to access the storage medium and may decode the I / O request from the network. In one implementation, the processing entity may include a root complex for a PCIe protocol or similar protocol. In one implementation, the processing entity may be implemented as processing logic within an ASIC, FPGA, or MCM.

As described herein, a "routing entity" may refer to one or more routers that route data between an interfacing entity, a processing entity, an NVM blade, and a routing entity itself. In one implementation, the routing entity may represent a PCIe node or endpoint for the PCIe protocol.

As described herein, an "interfacing entity" may refer to one or more host interface chips that interface with a storage device. In one implementation, the interfacing entity may send an I / O request to the routing entity using the PCIe protocol. I / O requests at the interface chip may be received using any suitable protocol, such as Gigabit Ethernet, Fiber Channel, dial-in, or even the PCIe protocol.

As described herein, an "I / O request" may refer to an input / output request from a network to a storage device for storing or retrieving data from a storage medium.

As described herein, "Peripheral Component Interconnect Express (PCIe)" includes higher maximum system bus throughput, lower I / O pin count and smaller physical footprint, better performance scaling for bus devices, A high-speed serial computer expansion bus standard designed for detailed error detection and reporting mechanisms and unique hot plug functionality. In a conventional PCIe system, a PCIe root complex enumerates all endpoint devices coupled to the processor and creates a tree-like structure.

Storage devices for enterprise systems require mass storage capacity, low latency for reading and writing to storage devices, high bandwidth, low power consumption, and reliability. Traditionally, an enterprise system is implemented using a storage medium, such as a hard disk drive (HDD), that holds data while power is turned off. An HDD is a data storage device used to store and retrieve digital information using a fast spinning disk. An HDD is composed of one or more rigid ("hard") rapidly rotating disks (platters) having magnetic heads arranged on a movable actuator arm for reading data and writing it to a surface.

Due to the moving parts involved in reading and writing data, the HDD is inherently prone to errors and failures and has a floor for improving the seek time for the data. Additionally, since the HDD has a spinning platter, there is also a limit on how small the part can be manufactured and the power consumption of the part.

In certain implementations, the techniques described herein propose to implement a storage device using an NVM storage medium. In some implementations, it may generally be advantageous to use an NVM storage medium because the NVM storage medium has a lower seek time, does not have a moving part, and may be generally more reliable than an HDD.

In one implementation, the components, modules, and configuration of the controller board can be arranged in a manner that increases heat dissipation, reduces power consumption, spreads power and workload, and reduces latency.

Conventional storage devices may provide one or more controller boards with respective controller boards containing a unit processing complex to receive I / O requests, process requests, and send storage requests to appropriate storage media. In accordance with the ever increasing demands for increasing network speed and increasing storage device size, a unitary point of access to the physical medium for storage is a bottleneck for a system that causes high latency for I / O requests . Increasing the processing load in a unit processing complex causes higher heat concentration in smaller areas, thus challenging proper heat dissipation. Moreover, a single processing unit may not be able to process transactions fast enough to follow an I / O request. In conventional systems, the unitary system design may not be a problem since the transaction bottleneck is often more than the seek time for reading and writing to the HDD and not the processing path for the HDD.

In some implementations, the storage device may be implemented using an NVM storage medium. In general, NVM storage media may have a lower seek time than conventional HDDs. In the case of the lower search period provided by the NVM storage medium, a conventional controller board design using a single processing complex may cause a lane configuration. Embodiments of the present invention may also provide additional enhancements to improve access time to NVM storage media. While some embodiments of the invention may be described herein using NVM storage media for illustrative purposes, the invention is not limited to NVM storage media and other suitable physical storage media may be used without departing from the scope of the present invention. Can be used.

Moreover, conventional storage devices can implement a fault tolerant system by maintaining mirrored storage of data. In other words, for each write operation, the data may be stored in at least two separate storage subsystems using independent processing paths. In the event of an unexpected failure in the first storage subsystem, such as a power failure, a failure of the storage medium, or an error in the processing path, the second storage system with the mirrored data is used to retrieve and store data during the recovery of the first system Can be used as an active backup. For HDDs, maintaining mirrored data is essential due to the low reliability of the medium and may be feasible due to the lower cost associated with the medium.

In some implementations, the NVM storage medium may be used to implement a fault tolerant system. In comparison, NVM storage media may be more reliable and less prone to error than conventional storage media. In some implementations, the reliability of the data stored on the NVM storage medium may be assured using techniques such as multiple array independent disks (RAID) or other suitable error recovery and correction techniques. Thus, instead of mirroring the entire system including the storage medium, as described in more detail in the embodiments discussed herein with reference to the figures, a number of paths for the same read or write operation may be stored on the same physical location of the NVM storage medium To reduce the overall cost of the system may be advantageous in implementations implemented using NVM storage media.

Figure 1 illustrates an exemplary high-level block diagram of a storage device in accordance with an embodiment of the invention. Block 102 illustrates a storage device having an array of blades 120a-n coupled to two controller boards 104 and 106 and a controller board. In one implementation, the storage device from Figure 1 may represent an active / active storage system. The active / active configuration allows the processing module for both controller boards to process I / O and provide standby capability for others. In one simple example, if a read or write command for a particular blade fails from the controller board 104, the same read or write may be attempted through the controller board 106. The communication protocol may be implemented to communicate status information between the controller boards 104 and 106. It may be advantageous to implement active / active storage devices to improve performance, since the processing modules associated with both controller boards can handle I / Os simultaneously or nearly simultaneously. However, the storage device from Figure 1 is not limited to active / active storage devices and can also be used in an active / passive configuration, and the processing module for one controller board is active to handle I / O requests, The other is idle in idle mode where the active primary controller board is ready to replace I / O activity that may have failed or taken offline.

As shown in Figure 1, each NVM blade may be coupled to both controller boards. Each controller board has routing modules 108 and 110 for routing, processing modules 112 and 114 for processing I / O requests, and host interfaces 116 and 118 for receiving I / O requests. In one implementation, routing modules 108 and 110 may be responsible for routing I / O requests from interface modules 116 and 118 to processing modules 112 and 114 for further processing of I / O requests . Processing modules 112 and 114 may use the file system software stack (not shown) to process I / O requests. Routing modules 108 and 110 also route access and storage requests from processing modules 112 and 114 to NVM blades 120a-n. In one implementation, the NVM blades are coupled to routing modules 108 and 110 using a PCIe protocol or any other suitable protocol.

In one implementation, each NVM blade may be coupled to both controller boards 104 and 106 to allow each physical address of the NVM storage medium to be accessible by either one of the controller boards. Such an arrangement may be advantageous in avoiding duplication of the underlying storage medium and mirroring of data, and the reliability of the data on the physical medium may be enhanced by a more reliable storage medium such as RAID, or any combination thereof, and / or a sophisticated data recovery technique As shown in FIG.

Figure 2 illustrates another exemplary block diagram of a storage device according to an embodiment of the invention. Figure 2 shows an image of two controller boards, each controller board comprising two processors, a memory, a router, and an interface chip. Figure 2 also shows a 42 NVM blade with a center channel for air flow. Although not shown, the storage device may also include two bridge boards with power management functionality and onboard NVM. The onboard NVM may be used to store dynamic metadata such as pointers, updated activities, cache backups, and read / write buffers. In some implementations, an NVM, such as a byte recordable magnetic RAM, may be used to implement the onboard NVM. Additionally, the storage device may include 12 fans, 8 fans used to cool the NVM memory, and 4 fans used to cool the controller board. The components can be placed in the exemplary configuration of FIG. 2 to optimize air flow, process load, and heat dissipation. The storage device may also include multiple power sources. The power supply is generally susceptible to failure and can fail due to a failure of the fan or other power components. Having multiple power supplies to power the storage device can avoid failures of the storage device due to component failure for one of the power supplies. In one implementation, the controller or controller board can be powered through a power rail, which can supply power from multiple power sources. In the event of a fault in one of the power sources connected to the power rail, the power rail continues to supply power from the functional power source. In some implementations, a failed power source may be hot-swappable (i.e., can be replaced without power cycling the storage device) with an appropriate functional power source.

NVM blades and controller / controller boards can have individually implemented digital circuit breakers that prevent shorts if any of the boards fails. Moreover, the power supply can also be implemented in a manner that only allows it to supply power to the power rail, but not to draw power from the power rail in the event of a power failure. In one embodiment, the diode may be used to prevent power from being discharged from a failed power source.

The number of components described with reference to FIG. 2, such as the controller board, the power supply, the NVM blade, the bridge board, and the fan and its associated configuration, is not limited and is provided as an example illustrating a particular configuration of the storage device.

Figure 3 illustrates another exemplary block diagram of a storage device according to an embodiment of the invention. As shown in Figure 3, the components of the storage device may be configured to fit into a rectangular shaped box. In one exemplary configuration, the airflow can be from front to back, and the fan is disposed on the back side of the storage device. Such a configuration can be advantageous for grouping together a number of storage devices into a rack configuration from an enterprise data storage facility. However, the shape of the storage device is not limited to the rectangular shaped box shown in Fig.

Figure 4 illustrates an exemplary block diagram of a storage device according to an embodiment of the invention. The system 402 of FIG. 4 illustrates a storage device having a first controller board 404 and a second controller board 406. For illustrative purposes, FIG. 4 shows a single NVM blade 420 from a plurality of NVM blades.

The first controller board 404 includes a first processing entity 412, a memory coupled to the first processing entity 416, a second processing entity 414, a memory coupled to the second processing entity 432, (408), and a routing entity (410).

The second controller board 406 includes a third processing entity 424, a memory coupled to the third processing entity 428, a fourth processing entity 418, a memory coupled to the fourth processing entity 430, A routing entity 422, and a routing entity 426.

In one implementation, routing entities 410 and 426 send I / O requests from interfacing entities 408 and 422 to one of the processing entities 412, 416, 428, and 430 for further processing of I / You may be responsible for routing. The processing entity may process the I / O request using a file system software stack (not shown). Routing entities 410 and 426 also route data requests from processing entities 412, 416, 428, and 430 to NVM blade 420.

In some implementations, the routing entity 410 from the first controller board 404 and the routing entity 426 from the second controller board 406 are connected to the NVM blade 420 to store and retrieve data from the NVM blade 420. [ (Not shown). In one implementation, the NVM blade 420 is coupled to the routing entity using the PCIe protocol. Such an arrangement may be advantageous in avoiding duplication of the underlying storage medium and mirroring of data, and the reliability of the data on the physical medium may be enhanced by a more reliable storage medium such as RAID, or any combination thereof, and / or a sophisticated data recovery technique As shown in FIG.

4, in one exemplary configuration, the first processing entity 412 receives one or more I / O requests and the data associated with the I / O requests is for a storage operation and the first routing entity 410 And to send data associated with the I / O request to the first routing entity for storage of the first data on the first blade 420, have. In one implementation, the file system software stack executing on the first processing entity 412 can determine the location and NVM blade operation associated with the I / O request. For example, in one implementation, the first processing entity 412 may perform one or more address translations from the file identifier for the data on the physical storage medium to the physical location. In one aspect, the I / O request received by the first processing entity 412 may first be received at the interfacing entity 408 and transmitted to the first processing entity 412 via one of the plurality of routing entities.

Similarly, the second processing entity 414 receives another I / O request and stores the data associated with the I / O request in another location on the first blade 420 that is coupled to the first routing entity 410 And to transmit data associated with the I / O request to the first routing entity 410 for storage of data on the first blade 420. [ The second processing entity 414 may also execute a file system software stack that determines the location and storage operation associated with the I / O request.

The example illustrates an exemplary configuration for performing load balancing to access the same NVM blade 420 between two processing entities from the same controller board and for spreading multiple I / O requests between processing entities 412 and 414 and ≪ / RTI > Although two processing entities are shown, a number of processing entities may be used. This may be advantageous in spreading loads handling I / O requests and also avoiding bottlenecks while performing multiple storage operations at the same time as the same physical medium at very high speeds.

The stored data may also be retrieved from the physical medium using similar techniques. For example, the second processing entity 414 may be configured to receive an I / O request to read data stored by the first processing entity 412 or any other processing entity for this point . The second processing entity 414 determines that the I / O request is a read request for data at the location of the first blade 420 coupled to the first routing entity 410, and the first routing entity 410 determines that the I / And may receive the first data from the first routing entity 410. [

In one exemplary configuration, the first processing entity 412 and the second processing entity 414 may not be directly coupled, but may be coupled to each other through the first routing entity 410. The transmission of data between the first processing entity 412 and the first routing entity 410 and the transmission of data between the second processing entity 414 and the first routing entity 410 is accomplished using the PCIe protocol or any other suitable protocol . ≪ / RTI >

For illustrative purposes, Figure 4 shows one NVM blade and two controller boards, but each controller board has two processing entities, two memory and routing entities and an interfacing entity, But is not limited to the number of entities shown in FIG. For example, other exemplary configurations may include multiple NVM blades, multiple routing entities, and multiple interfacing entities, without departing from the scope of the present invention. Figure 7 is an example of such a configuration with multiple routers (routing entities) and multiple interface chips (interfacing entities).

In another exemplary configuration, the first processing entity 412 and the second processing entity 414 may be coupled to another (second) routing entity (not shown) on the first controller board 404. Similar to the routing entity 410, the second routing entity can also be coupled to another NVM blade and can handle storage access commands received from both the first processing entity 412 and the second processing entity 414 have. The transmission of data between the first processing entity 412 and the second routing entity (not shown) and the transmission of data between the second processing entity 414 and the second routing entity (not shown) Lt; RTI ID = 0.0 > protocol. ≪ / RTI > Similarly, the components on the second controller board 406 may be configured and operated in a manner similar to the first controller board 404 described above.

The NVM blade 420 may include a plurality of routing interfaces to communicate with a plurality of controller boards. In one exemplary implementation of the storage device 402 a first controller board 404 including the routing entity 410 and a second controller board 406 including the routing entity 426 are coupled to the NVM blade 420 ). The NVM blade 420 may be coupled to the first controller board 404 via the routing entity 410 and the NVM blade may be coupled to the second controller board 406 via the routing entity 426. [ In one implementation, the NVM blade 420 communicates with the routing entities 410 and 426 on the controller board using the PCIe protocol or any other suitable protocol. In one implementation, the NVM blade includes an NVM storage medium. In other implementations, the storage device may comprise a plurality of NVM blades and the controller board may comprise a plurality of routing entities.

In some implementations, the routing entity 410 from the first controller board 404 and the routing entity 426 from the second controller board 406 may be coupled together. In some implementations, the two routing entities may be coupled to each other using a non-PCIe-compliant transparent bridge. In one implementation, the two routing entities 410 and 426 may communicate fault tolerance information, system state information, transaction completion information, and other information regarding the state of the controller board.

In one implementation, the storage device 402 from FIG. 4 may represent an active / active storage system. The active / active configuration allows the processing module for both controller boards to handle I / O reads and provide standby capability for others. In one simple example, if a read or write command for a particular blade fails from the controller board 404, the same read or write may be attempted through the controller board 406. As described above, the communication protocol may be implemented to communicate status information between the controller boards 404 and 406 via routing entities 410 and 426. [ Since the processing modules associated with both controller boards can handle I / O concurrently, it may be advantageous to implement active / active storage devices to improve performance. However, the storage device from Figure 4 is not limited to active / active storage devices and can also be used in an active / passive configuration, and the processing module for one controller board is active to handle I / O requests, The other is the idle mode in which the active primary controller board is ready to replace I / O activity that may have failed or taken offline.

In one implementation of the active / active system, one or more controller boards may assume the role of a master board and one or more other boards may assume the role of a slave board. The master controller board can perform all data writing to the NVM blade, while the master or slave board can perform the reading.

In one exemplary implementation, the I / O write operation reaching the slave controller board can be partially performed by the master controller board. For example, information associated with a write command or a write command may be transmitted from the slave controller board to the master controller board. In one implementation, the NT PCIe bridge may be used to transfer information associated with the write operation from the slave controller board to the master controller board. In one implementation, data for a write operation reaching the slave controller board may still be provided to the NVM blade by the slave controller board.

The master and slave controller boards can maintain a mapping table that maps read and write operations to the NVM blades. In one implementation, the read and write tables are stored in one of the NVM blades. In one implementation, the read and write tables may be shared by two controller boards. In addition, in other implementations, the read and write tables can be maintained separately by the controller board. In the case where each controller board has its own table, the master controller board can update the table for the master and slave controller boards.

If the slave controller board fails, the master controller board continues to operate. On the other hand, if the master controller board fails, the storage device resolves to the slave controller board. The slave controller board becomes the new master controller board and can begin to handle all I / O write operations.

The system described above may allow for distributing workloads for read transactions across two or more controller boards as the read operation requires time and processing power to decode I / O requests.

5 is a flow chart illustrating a method of performing an embodiment of the present invention in accordance with an embodiment of the present invention. Signaling in method 500 may be performed on processing logic that includes hardware (circuitry, dedicated logic, etc.), software (e.g., executed on a general purpose computing system or a dedicated machine), firmware (embedded software) Lt; / RTI > In one implementation, method 500 is performed by one or more computer systems 1300 as described in FIG.

The flowchart of FIG. 5 shows a first processing entity 502 and a second processing 504 that process I / O requests. Although Figure 5 shows only two processing entities, a number of processing entities may be implemented to perform the embodiments of the present invention as described with reference to Figure 5. For example, an embodiment of the present invention performs similar steps of the present invention performed by a first processing entity or a second processing entity using a third, fourth, fifth, or any number of processing entities can do. Moreover, although only one I / O request is shown for each processing entity between the start and end indicators in FIG. 6, any number of I / O requests can be performed.

At step 506, the first processing entity coupled to the plurality of NVM blades receives the first I / O request through the routing entity.

In step 508, the first processing entity determines whether the first I / O request is a write or read request. If it is determined in step 508 that the first I / O request is a read request, then in step 510, the first processing entity determines whether the target NVM blade from the plurality of NVM blades and the location Can be determined. In one implementation, the first processing entity may determine the location in the target NVM blade and the target NVM blade by performing one or more address translations using the file system software stack executing on the first processing entity.

At step 512, the first processing entity requests data associated with the first I / O request. At step 514, the first processing entity receives data via a routing entity for a read I / O request.

If it is determined in step 508 that the first I / O request is a write request, then in step 516, the first processing entity determines whether the target NVM blade from the plurality of NVM blades and the location in the target NVM blade Can be determined. In one implementation, the first processing entity may determine the location in the target NVM blade and the target NVM blade by performing one or more address translations using the file system software stack executing on the first processing entity. At step 518, the first processing entity sends data to the target NVM blade via the routing entity to store data in the target NVM blade for the write I / O request.

Similarly, for a second processing entity 504, at step 520, a second processing entity coupled to a plurality of NVM blades may receive a second I / O request via the routing entity. The second processing entity 504 may receive a second I / O request before / after the first I / O request received at the first processing entity or simultaneously with the first I / O request. Moreover, the first processing entity 502 and the second processing entity 504 may perform steps identified in FIG. 5 independently of each other.

At step 522, the second processing entity determines whether the second I / O request is a write or read request. If it is determined in step 522 that the second I / O request is a read request, then in step 524, the second processing entity determines whether the target NVM blade from the plurality of NVM blades and the location within the target NVM blade Can be determined. In one implementation, the second processing entity may determine the location within the target NVM blade and the target NVM blade by performing one or more address translations using the file system software stack executing on the second processing entity. At step 526, the second processing entity requests data associated with the second I / O request. At step 528, the second processing entity receives data via a routing entity for a read I / O request.

Alternatively, in step 522, if it is determined in step 522 that the second I / O request is a write request, then in step 530, the second processing entity determines whether the target NVM blade from the plurality of NVM blades, The position within the NVM blade can be determined. In one implementation, the second processing entity may determine the location within the target NVM blade and the target NVM blade by performing one or more address translations using the file system software stack executing on the second processing entity. At step 532, the second processing entity sends data to the target NVM blade via the routing entity to store data in the target NVM blade for a write I / O request.

As discussed above, similar to the first processing entity 502, the second processing entity 504 may process the I / O request. In some implementations, the first processing entity and the second processing entity may process I / O requests in any sequence with respect to each other and also process I / O requests simultaneously. Moreover, the first processing entity and the second processing entity may concurrently process transactions targeted to one of the plurality of NVM blades.

Referring again to Fig. 4, an example of two processing entities in the system may be illustrated by any of the processing entities shown in Fig. For example, the two processing entities may be 412 and 414 on the same controller board 404, or the processing entity 412 and processing entity 428 residing on different controller boards.

Communication between one or more components discussed with reference to FIG. 5 may be performed using a PCIe protocol or any other suitable protocol. Although the method of FIG. 5 may be advantageous to spread I / O requests among a plurality of processing entities, I / O requests may allow for faster processing, avoid bottlenecks and facilitate better heat dissipation Causing memory behavior on the same NVM blade.

It should be appreciated that the particular steps illustrated in Figure 5 provide a particular way of switching between operating modes according to an embodiment of the present invention. Other sequences of steps may also be suitably performed in alternative embodiments. For example, alternative embodiments of the present invention may perform the outlined steps in a different order. To illustrate, a user may choose to change from a third operating mode to a first operating mode, from a fourth mode to a second mode, or any combination thereof. Moreover, the individual steps illustrated in FIG. 5 may include a number of sub-steps that may be performed in various sequences as appropriate for the individual steps. Moreover, the additional steps may be added or removed depending on the particular application. One of ordinary skill in the art will recognize and appreciate many of the variations, modifications, and alternatives of method 500.

Figure 6 is a flow chart illustrating another method of performing an embodiment of the present invention in accordance with an embodiment of the present invention. Signaling in method 600 may be performed on processing logic that includes hardware (circuitry, dedicated logic, etc.), software (e.g., executed on a general purpose computing system or a dedicated machine), firmware (embedded software) Lt; / RTI > In one implementation, the method 600 is performed by one or more computer systems 1300 as described in FIG.

The flow chart of FIG. 6 shows a first processing entity 602 and a second process 604 that process an I / O request. Although Figure 6 shows only two processing entities, a number of processing entities may be implemented to perform the embodiments of the present invention as described with reference to Figure 6. [ For example, an embodiment of the present invention performs similar steps of the present invention performed by a first processing entity or a second processing entity using a third, fourth, fifth, or any number of processing entities can do. Moreover, although only one I / O request is shown for each processing entity between the start and end indicators in FIG. 6, any number of I / O requests can be performed. Figure 6 illustrates one implementation of the implementation illustrated in Figure 5.

In step 606, the first processing entity coupled to the plurality of NVM blades receives the first I / O request via the first routing entity.

In step 608, the first processing entity determines whether the first I / O request is a write or read request. If it is determined in step 608 that the first I / O request is a read request, then in step 610, the first processing entity receives the first NVM from the plurality of NVM blades coupled to the first routing entity, It can be determined that the data is read from the first position of the blade. In one implementation, the first processing entity may determine a first location on the first NVM blade and the first NVM blade by performing one or more address translations using a file system software stack executing on the first processing entity.

At step 612, the first processing entity requests data associated with the first I / O request via the first routing entity. In step 614, the first processing entity receives the data via the first routing entity and completes the read I / O request.

If it is determined in step 608 that the first I / O request is a write request, then in step 616, the first processing entity is configured to receive a first NVM blade from a plurality of NVM blades and a first NVM blade Lt; RTI ID = 0.0 > position. ≪ / RTI > In one implementation, the first processing entity may determine a first location on the first NVM blade and the first NVM blade by performing one or more address translations using a file system software stack executing on the first processing entity. In step 618, the first processing entity transmits data to the first NVM blade through the first routing entity to store data at a first location on the first NVM blade.

Similarly, for the second processing entity 604, at step 620, the second processing entity coupled to the plurality of NVM blades may receive a second I / O request via the first routing entity. The second processing entity 604 may receive a second I / O request before / after the first I / O request received at the first processing entity or concurrently with the first I / O request.

At step 622, the second processing entity determines whether the second I / O request is a write or read request. If it is determined in step 622 that the second I / O request is a read request, then in step 624, the second processing entity receives the first NVM from the plurality of NVM blades coupled to the first routing entity, It can be determined that the data is read from the first position of the blade. In one implementation, the second processing entity may determine a first location on the first NVM blade and the first NVM blade by performing one or more address translations using a file system software stack executing on the second processing entity. At step 626, the second processing entity requests data associated with the second I / O request via the first routing entity. At step 628, the second processing entity receives the data via the first routing entity and completes the read I / O request.

Alternatively, at step 622, if it is determined that the second I / O request is a write request, then at step 630, the second processing entity receives a write request from a plurality of NVM blades coupled to the first routing entity Lt; RTI ID = 0.0 > NVM < / RTI > In one implementation, the first processing entity may determine a first location on the first NVM blade and the first NVM blade by performing one or more address translations using a file system software stack executing on the second processing entity. At step 632, the second processing entity sends data to the target NVM blade via the first routing entity to store data in the target NVM for the write I / O request.

As discussed above, similar to the first processing entity 602, the second processing entity 604 may process the I / O request. In some implementations, the first processing entity and the second processing entity may process I / O requests in any sequence with respect to each other and also process I / O requests simultaneously. Moreover, the first processing entity and the second processing entity may concurrently process transactions targeted to one of the plurality of NVM blades.

Referring again to FIG. 4, an example of two processing entities residing on the same controller board and accessing the same NVM blades through the same routing entity is shown in the processing entities 412 and 414 resident on the same controller board 404 ≪ / RTI > The steps described in Figure 6 allow two processing entities residing on the same controller board to simultaneously process and service I / O requests targeted to the same location on the same NVM blade or even NVM blades. As shown in FIG. 6, although I / O requests can be decoded and processed at the individual processing entities, it can use the same routing entity to access the NVM blades, thus saving cost by avoiding duplication of hardware .

Communication between one or more components discussed with reference to FIG. 6 may be performed using a PCIe protocol or any other suitable protocol. While the method of Figure 6 may be advantageous for spreading I / O requests among a plurality of processing entities, I / O requests may allow for faster processing, avoid bottlenecks and facilitate better heat dissipation Causing memory behavior on the same NVM blade.

It should be understood that the particular steps illustrated in FIG. 6 provide a particular way of switching between operating modes according to an embodiment of the present invention. Other sequences of steps may also be suitably performed in alternative embodiments. For example, alternative embodiments of the present invention may perform the outlined steps in a different order. To illustrate, a user may choose to change from a third operating mode to a first operating mode, from a fourth mode to a second mode, or any combination thereof. Moreover, the individual steps illustrated in FIG. 6 may include a number of sub-steps that may be performed in various sequences as appropriate for the individual steps. Moreover, the additional steps may be added or removed depending on the particular application. One of ordinary skill in the art will recognize and appreciate many of the variations, modifications, and alternatives of method 600.

Figure 7 illustrates an exemplary block diagram of a controller board in accordance with an embodiment of the present invention. In one implementation, the controller board 702 may represent the controller board 104 or 106 of FIG. 7, the controller board has two processors 704 and 708, four routers 712, 714, 716 and 718 and four interface chips 720, 722, 724 and 726. Processor 0 704 may have a memory controller that controls access to its local memory 706a-d. Similarly, processor 1 708 may also have a memory controller that controls access to its local memory 710a-d. In one implementation, the interface chips and routers may communicate with each other using a PCIe protocol or any other suitable protocol. PCIe can also be used as a routing protocol for communication between a processor and a router. I / O requests at the interface chip may be received using any protocol, such as Gigabit Ethernet, Fiber Channel, dial-in, or even PCIe protocol.

As shown in FIG. 7, in one implementation, each interface chip may communicate data to either processor 704 and 708 through a router. Each interface chip may be coupled to at least one router via a PCIe protocol or any other suitable protocol. An I / O request can reach one of the interface chips. The interface chip can send I / O requests to the router using the PCIe protocol. Each router is connected to both processors on the controller board 702. The router receives the I / O request and determines the processor to send the I / O request for further processing. Once the processor has decoded the I / O request and confirmed the operation of storing or retrieving data from the NVM storage medium, the processor sends a memory operation command to one of the routers. Each router is coupled to a subset of NVM storage media via NVM blades. For example, in FIG. 7, each router connects to approximately one quarter of the total number of NVM blades. The decision to send the NVM storage media request to the router may be based on the address of the storage / access request in the NVM storage address space. For example, if the processor 704 determines to cause the I / O request to cause storage on the NVM blade that combined the router (R2) 716, then the processor uses the PCIe protocol to send the request to the router (R2) 716 As shown in Fig. The router (R2) 716 sends a storage request to each NVM blade for storage.

In certain implementations, the configuration described with respect to FIG. 7 reduces the load associated with various electrical components, increases the throughput of operations on NVM storage media, and dissipates heat from various components within the storage device Can be advantageous.

In conventional PCIe systems, the central processing unit may include a root complex for the entire system. A PCIe root complex enumerates all endpoint devices coupled to the processor and creates a tree-like structure. All requests originating at the endpoints are handled by one or more processors coupled to the PCIe root complex. In a storage device having multiple requests originating at the same endpoint as the interface chip, the root complex and the processor become bottlenecks for processing transactions in the system. In one implementation, a more powerful processor can be used to expedite I / O requests and assist the bottleneck. This approach can temporarily assist the bottle neck, but it can increase the power load associated with the processor. Moreover, the processor can also generate more heat over a small area on the controller board due to the increased number of I / O requests processed by the processor. Increased heat on a single processor or closely clustered processor can challenge maintaining a stricter thermal envelope for the storage device as a whole, at an acceptable level. Additional power loads and heat can generate more failures at both component and device levels.

Embodiments of the present invention propose to spread processing and routing functionality to access NVM storage across controller boards for a plurality of processing entities. In one implementation, multiple processing entities may be spread across a controller board that processes I / O requests. In one implementation, one of the processing entities may serve as a PCIe root complex and the second processing entity may serve as an endpoint. For example, in FIG. 7, processor 0 (707) may be configured as a PCIe root complex and processor 1 (708) may be configured as an endpoint. In one implementation, the memory space for processor 1 708 is listed four times as an endpoint for each of the routers (router 0 712, router 2 714, router 3 716, and router 4 718) . In instances where the receiving router for an I / O request does not have the proper mapping for an I / O request, the router may send an I / O request to a processing entity configured as a PCIe root complex to determine the mapping. Also, the interface chip may be configured with routing information at the time of configuration.

In the case where routing is already established in the interface chips and routers, I / O requests arriving at the interface chip and sent to the router may be sent to either of the processing entities 704 and 708 that spread processing functionality. Besides processing, the described architecture can also propagate the connectivity of the link. For example, multiple interface chips may be implemented to receive I / O requests at the same time and send their I / O requests to the router. Moreover, the NVM blades are distributed among the routers to allow access to the NVM blades to be distributed among multiple routers, to avoid buses, or to route backlogs. Such a configuration can also be advantageous to allow simultaneous access to multiple blades, as described in FIG. 7, which significantly improves read and write performance when accessing NVM blades accessible via different routers. In an alternative embodiment, an implementation of the present invention proposes a plurality of processing entities each having its own root complex to spread processing and routing functionality to access NVM storage across the controller board. Each endpoint (i. E., A router) may be connected to more than one root complex. Thus, an I / O request arriving at the interface chip and forwarded to the router may be sent to any of the processing entities 704 and 708 that spread processing functionality. Besides processing, the described architecture can also propagate the connectivity of the link. For example, multiple interface chips may be implemented to receive I / O requests at the same time and send their I / O requests to the router. Moreover, the NVM blades are distributed among the routers to allow access to the NVM blades to be distributed among multiple routers, to avoid buses, or to route backlogs. Since each processor is connected to all the routers on the controller board, each processor can address any NVM storage address separately. Such a configuration can also be beneficial to allow simultaneous access to multiple blades, as shown in Figure 7, which significantly improves read and write performance when accessing NVM blades accessible via different routers.

Processor 0 704 may be booted from boot ROM 728 and processor 1 708 may be booted from boot ROM 734. [ In one implementation, the boot ROM image running on the processor 704 may also include initialization information for the storage file system stack. In one implementation, the storage file system operating system (OS) may be loaded from onboard NVM. In other implementations, the storage file system OS may be loaded from one of the NVM blades. In one implementation, the image for the OS running on processor 0 704 and processor 1 708 may be different. The file system OS may be responsible for converting I / O requests into hardware reads and writes.

In certain implementations, onboard NVM 736 may be used to store dynamic metadata such as pointers, updated activities, cache backups, and read / write buffers. In some implementations, an NVM, such as a byte recordable magnetic RAM (MRAM), may be used to implement the onboard NVM. The controller board may also have a debug port 740 coupled to the processor 704 and the processor 708. The debug port may support one or more separate interfaces such as USB, PCIe, Gigabit Ethernet, and the like.

Figure 8 illustrates an exemplary block diagram of address space for various components as seen by respective components on a controller board in accordance with at least one implementation of the present invention. In one implementation, the address space may be defined as a PCIe address space.

P0 810 represents a view of the PCIe address space from processor 0 704 in FIG. P1 830 represents a view of the PCIe address space as seen from processor 1 708 of FIG. R0 850, R1 860, R2 870 and R3 880 are the PCIe addresses from router 0 712, router 1 714, router 2 716, and router 3 718, respectively. Represents a view of space. In one implementation, a PCIe root complex such as processor 0 (704) may find all endpoints and configure the PCIe address space for each endpoint.

In some implementations, access to any of the various PCIe ranges seen from any of the components of the controller board may result in access to different PCIe address ranges and different types of responses. For example, according to one embodiment of the invention, accessing one range of PCIe address space from the processor may cause a configuration change in one of the routers. In another example, accessing another range of PCIe address space may cause read / write access to one of the NVM blades coupled to one of the routers. Some access to the PCIe address space may also be mapped to memory for one of the local processors for the processor or one of the contiguous processors on the controller board. In another example, some access to the PCIe address space may cause read / write to components on the adjacent controller board via non-transparent (NT) PCIe bridges.

Through the PCIe address space, several entities have at least partial access to the address space of other entities on the controller board. For example, at P0 810, the processor (P0) 704 has access to its own memory and has partial access to the memory of the processor (P1) 708 and the address space of the router, respectively. In one implementation, the NVM blades are grouped into four separate groups of NVM blades, and each group of NVM blades can be coupled to one of the routers. Any one of the NVM blades belonging to a particular group of NVM blades is accessible through a router to which a group of NVM blades can be coupled.

In FIG. 8, from the PCIe address space for P0 810, B-G0 808 represents the address space for the first group of NVM blades accessible via router (R0) 712. The router (R0) 712 may be coupled to the first group of NVM blades and may be configurable from the processor (P0) 704 via an address space designated by host bus adapter 0 (HBA0) 806 . Similarly, processor (P0) 704 includes a second group of NVM blades through address space (B-G1) 814 and a second router (Rl) 714 via HBA1 812, an address space B -G2) 818 and the third router (R2) 716 through the HBA2 816 and the fourth group of NVM blades through the address space (B-G3) 822, And the fourth router (R3) 718 via the HBA 3 820. In some implementations, a portion of the address space 824 may be reserved. In certain implementations, onboard NVM, such as MRAM 828, may be used to store dynamic metadata such as pointers, updated activities, cache backups, and read / write buffers. Furthermore, the processor (P0) 704 is connected to the local processor 706a-d via its own local memory 706a-d via the PCIe address space DRAM (P0) 802 and the adjacent processor P1 0.0 > 708 < / RTI > In some implementations, processor (P0) 704 may also send a message to an element of the adjacent controller board via NT port 826. [

Similar to P0 810, views of the PCIe address space from each of the components can provide each component with the ability to interact with each other using the PCIe address space. For example, processor (P1) 708 may communicate with each of the routers (HBA0 840, HBA1 838, HBA2 836, and HBA3 833) via its PCIe address space (P1) ), The PCIe address space for the NVM blades (B-G0 841, B-G1 839, B-G2 837 and B-G3 834) (P0) 832, MRAM 842, and NT port 838 of its own local memory 710a-d via the processor (P0) 704 and the adjacent processor (P0)

A router is also similar to a view of the PCIe address space, but can have a more limited view. For example, the router (Ro) 712 may have a PCIe address space view R0 850 of the system. The router R0 may be capable of communicating with the processor P0 704 and the processor P1 708 via the DRAM (P0) 851 and the DRAM (P1) 853, respectively. In certain implementations, onboard NVM, such as MRAM 854, may be used to store dynamic metadata such as pointers, updated activity, cache backups, and read / write buffers. Access to the PCIe address space (HBA0) 858 by other components on the controller board can be interpreted as an instruction to the router (R0) 712. [ Access to the B-G0 856 may be interpreted as a read and write request to the NVM blade coupled to the router (R0) 712. The routers (R0) 712 may not have a PCIe address space reserved for other routers or NVM blades since there is no direct coupling between their components, as shown in Fig. The router (R3) 718 also includes a processor (P0) 704, a DRAM (P0) 881, a processor (P1) 708, a DRAM (P1) 883, MRAN 885, its own configuration space, and NVM blades.

Routers R1 714 and R2 716 also access processors P0 704 and P1 708 via DRAMs P0 861 and 871 and DRAM P1 863 and 873, . The configuration space for routers R1 714 and R2 716 is accessed through HBA1 866 and HBA2 877 and its associated NVM blades B-G1 867 and B-G2 878 . In addition, the routers R1 714 and R2 716 may be capable of transmitting messages to the routers on the adjacent controller boards via each of the NT ports 865 and 875. [

In some implementations, some of the address ranges in the PCIe address space for each component may be used for future use (843, 852, 857, 862, 864, 868, 872, 874, 876, 879, 882, 884 and 888) And can be reserved for use.

As discussed previously, the PCIe address space configuration shown in FIG. 8 is for illustrative purposes only and is not limited to other implementations of address space.

9 illustrates another exemplary high-level block diagram of a storage device according to an embodiment of the invention. Block 902 illustrates a storage device having an array of NVM blades 920a-n coupled to two controllers 904 and 906 and a controller. In one implementation, controllers 904 and 906 may be coupled together using a communication protocol to communicate status information between controllers 904 and 906 for read and write transactions using bridge 908. [

In one implementation, the first controller 904 and the second controller 906 include one or more processors for processing I / O requests, one or more routers for routing operations between a controller and a plurality of NVM blades, and one or more interfacing chips (PCB) including a printed circuit board (PCB). An example of such a controller board has been discussed previously in Figures 1-8. In other implementations, the functionality of a number of discrete components may be performed by a controller implemented in an ASIC, FGPA, MCM, or any other suitable solution. In one implementation, the first controller 904 and the second controller 906 may be implemented in an ASIC that includes processing logic and routing logic, respectively. In one implementation, the controller may also include interfacing logic. 9, the first controller 904 may be coupled to the host interface 916 to receive an I / O request and respond to an I / O request, and the second controller 906 May be coupled to another host interface 918. [

In certain implementations, the storage device from Figure 9 may represent an active / active storage device. An active / active configuration allows the processing logic on the controller to handle I / O and provide standby capabilities for others. Since the processing logic associated with both controllers can handle I / O concurrently or nearly simultaneously, it may be advantageous to implement an active / active storage device to improve performance. However, the storage device from Figure 9 is not limited to active / active storage devices and can also be used for active / passive configuration, and the processing logic for one controller is active to handle I / O requests, Is the idle mode in which the active primary controller board is ready to replace the I / O activity that may have failed or taken offline.

In one implementation, in the active / active system shown in FIG. 9, the first controller 904 may be configured to operate in an active mode and receive an I / O request to store and retrieve data from the NVM storage medium. Similarly, the second controller 906 may also be configured to operate in an active mode and receive an I / O request to store and retrieve data from the NVM storage medium. 9 shows only two controllers, but a plurality of controllers can operate in an active mode.

Additionally, the storage device may include a plurality of NVM blades 920a-n that include NVM storage media. In one implementation, each NVM blade can be coupled to both controllers 904 and 906, allowing each physical address of the NVM storage medium to be made accessible by either one of the controllers. Such a configuration may be advantageous to avoid duplication of basic storage media and mirroring of data, and the reliability of data on the physical medium may be more reliable storage media such as RAID, or any combination thereof, and / or sophisticated data recovery technology As shown in FIG. Each NVM blade may include a first routing interface in communication with the first controller 904 and a second routing interface in communication with the second controller 906. In one implementation, the first routing interface communicates with the first controller and the second routing interface uses the PCIe protocol or any other suitable protocol to communicate with the second controller.

In one implementation of the active / active system, one or more controllers assume the role of a master controller and the other one or more controllers assume the role of a slave controller. In one implementation, the master controller may perform or initiate all data writes to the NVM blade, while either the master or the slave board may perform the read.

In general, a storage device can service more read operations than a storage or write operation to a storage medium. Also, read operations in general can be completed faster than write or write operations. Thus, the rate at which the read operation can be serviced can be constrained by the rate at which the I / O request can be decoded and processed by the processing logic of the controller. Thus, it may be advantageous to load balance the I / O read operation between the two or more controllers in an active / active system for processing and decoding. Thus, both the master and slave controllers can handle I / O read operations. Thus, in FIG. 9, both the first controller 904 and the second controller 906 can be configured to decode I / O requests simultaneously or nearly simultaneously for read operations and to request data from the NVM storage medium.

In one exemplary implementation, the write operation to reach the slave controller board can be performed in part by the master controller. For example, information associated with a write command or a write command may be transmitted from the slave controller to the master controller. In one implementation, bridge 908 (e.g., a PCIe NT bridge) may be used to transfer information associated with a write operation from the slave controller to the master controller. In one implementation, data for a write operation reaching the slave controller may still be provided to the NVM blade by the slave controller.

For purposes of illustration, at a given point in time, the first controller 904 may be a master controller and the second controller 906 may be a slave controller. In one example, the I / O request can reach the first controller 904, which can act as a master controller. The first controller 904 may determine that the I / O request is a write operation that stores data associated with the I / O request on the NVM storage medium. The master controller can process I / O requests, determine NVM blades, send write commands, and send commands and data to the NVM blades that store the data.

In another example, the I / O request may arrive at a second controller 906 that can act as a slave controller. The second controller 906 may determine that the I / O request is a write operation that stores data associated with the I / O request on the NVM storage medium. The second controller 906 may send command information associated with the second I / O request to the first controller 904, which may act as a master controller. The master / first controller 904 may receive command information sent from the second controller 906, determine an NVM blade on which data may be stored, and send a write command to the NVM blade. A write command may be sent by the master controller, but a second controller 906, acting as a slave controller, may send data associated with the I / O request to the NVM blade. Managing all write operations from the master can help to maintain write consistency in the system. On the other hand, sending data from the slave controller to the NVM blade for an I / O write request received at the slave controller requires a bridge between the first controller 904 and the second controller 906 to transmit data between the two. (E.g., a NT PCIe bridge) that requires a significant increase in the bandwidth of the bridge 908 (e.g., NT PCIe bridge).

The master and slave controllers can maintain a mapping table that maps read and write operations to the NVM blades. In one implementation, the read and write tables are stored in one of the NVM blades. In one implementation, the read and write tables may be shared by the two controllers. Furthermore, in other implementations, the read and write tables may be maintained separately by the controller. In the case where each controller has its own table, the master controller can update the table for both the master and slave controllers.

If the slave controller fails, the master controller continues to operate as before. On the other hand, if the master controller fails, the storage device is resolved as a slave controller. In other words, the slave controller becomes a new master controller and can begin to process the write operation. For example, if the first controller 904 acting as the master controller encounters a non-recoverable error, the system can be resolved and the second controller 906 can become the master controller.

In some implementations, the storage device may also include multiple power sources. The power supply is generally susceptible to failure and can fail due to a failure of the fan or other power components. Having multiple power supplies to power the storage device can avoid failures of the storage device due to component failure for one of the power supplies. In one implementation, the controller board can be powered via a power rail, and the power rail can supply power from multiple power sources. In the event of a fault in one of the power sources connected to the power rail, the power rail continues to supply power from the functional power source. In some implementations, the failed power supply may be hot-swappable (i.e., without power cycling the storage device) to an appropriate functional power source. Figure 10 illustrates an exemplary block diagram of an NVM blade in accordance with an embodiment of the present invention. In some implementations, the NVM blade 1002 may represent one implementation of one of the NVM blades 420 of FIG. 4 or the NVM blades 920a-n from FIG. The exemplary NVM blade 1002 may include one or more NVM chips 1006 and 1008 and a blade controller 1004. The NVM chip may include an NVM storage medium. The NVM chip may be coupled to the blade controller 1004 via shared buses 912 and 1014 or a dedicated bus (not shown). The blade controller 1004 may be responsible for receiving instructions to access / store data on the NVM chip, to process the instructions, and to store or retrieve data from the NVM chips and other configuration instructions. Although not shown, the NVM chip may also reside on the opposite side of the NVM blade. In one implementation, the blade controller 1004 may be implemented using an application specific integrated circuit (ASIC). In other implementations, the NVM blade controller may be implemented using a field programmable gate array (FPGA).

11 illustrates an exemplary block diagram of a blade controller in accordance with an embodiment of the invention. In one implementation, blade controller 1004 may have two or more PCIe interfaces 1014 and 1116 connecting to a routing entity on a controller (or controller board). For example, the PCIe interface 1114 may be coupled to one of the PCIe interfaces on the routing entity from the first controller and the PCIe interface 1116 may be coupled to one of the PCIe interfaces on the routing entity from the second controller . Each PCIe interface may maintain command queues 1010 and 1112 associated with commands arriving from each controller to which the PCIe interface is coupled. In one implementation, the data path for the data associated with the controller can be maintained separately. For example, data associated with each controller may be decompressed after appropriately compressing in blocks 1106 and 1108 and retrieving data from the NVM storage medium, prior to storage of data on the NVM storage medium. Maintaining the individual data paths allows higher throughput of the data and can reduce errors associated with the data path. In one implementation, error detection and / or correction may be performed in blocks 1106 and 1108 using an error correction code (ECC). For example, data may be coded and compressed before storing data on the NVM storage medium, and decompressed and checked for errors when retrieving data. If an error is detected, in some scenarios, the data may be recoverable. If the error is irrecoverable, the NVM blade may discard the read request or respond to an error condition for the controller board.

Command manager 1104 arbitrates commands on multiple PCIe interfaces. Command manager 1104 decodes the instruction and accesses the appropriate NVM storage medium from the array of chips for storage / access of data. By arbitrating the command, in some implementations, the command manager 1104 may allow only one active command to access / store data via the NVM interface 1102 in any particular time period. In some implementations, the PCIe interface, command queue, and ECC compression / decompression logic may be implemented separately to interface with each controller board. Such a separation between the read / write path, the queue and the logic may be advantageous to avoid a failure on one interface of the NVM blade that adversely affects the second interface of the NVM blade. For example, if the command queue 1110 begins to back up due to an error anywhere from the first controller board to the NVM interface 1102, the read / write data path from the second controller board to the NVM storage medium will normally You can continue to function. Therefore, in the case where the storage operation for the NVM storage medium fails from one first controller board, the storage operation for the same memory location on the nonvolatile memory is completed using the second controller board .

12 illustrates another exemplary block diagram of a blade controller in accordance with an embodiment of the present invention. This alternative implementation of the blade controller 1004 also includes two or more PCIe interfaces 1214 and 1216 connecting to the routing logic on the controller and a command queue 1210 And 1212). In one implementation, the instruction queue can be implemented using buffers. In one implementation, the instruction queue may be configured to discard instructions from the first controller if the instruction queue buffer is full beyond a predetermined threshold.

In one implementation, the unified data path and the unified command path may be implemented as shown in FIG. In some implementations, data from the data path may be compressed in block 1206 and decompressed after retrieval from the NVM storage medium before the data is stored in the NVM storage medium. In one implementation, error detection and / or correction may be performed at block 1206 using an error correction code (ECC). For example, the data may be coded and compressed before the data is stored on the NVM storage medium, and decompressed and checked for errors when retrieving the data. If an error is detected, in some scenarios, the data may be recoverable. If the error is irrecoverable, the NVM blade may discard the read request or respond to an error condition for the controller.

Command manager 1204 can arbitrate commands from multiple PCIe interfaces. Command manager 1204 decodes the commands and accesses the appropriate NVM storage medium from the array of chips for storage / access of data. By arbitrating the command, the command manager 1204 can allow only one active command to access / store data via the NVM interface 1202 in any particular time period. As shown in Figure 12, consolidated data and command paths can result in cost and design efficiency.

Although not shown in the figures, in one implementation, individual instructions and / or data queues may be maintained for each NVM chip from a plurality of NVM chips including NVM storage media for NVM blades. Furthermore, a separate set of commands and / or data queues may be maintained for each controller. For example, in one implementation of an NVM blade with 32 NVM chips, 32 commands and / or data queues may be maintained for requests originating from the first controller and 32 commands and / 2 controller. ≪ / RTI > Such an arrangement may allow a number of outstanding commands to be initiated, processed and / or completed while other commands are initiated, processed and completed on the NVM blades, unless the operation is targeted to the same NVM chip. The command manager 1004 can arbitrate commands issued from the two controllers.

Having described many aspects of a vertically integrated architecture, an example of a computing system in which various aspects of the present disclosure may be implemented can now be described with respect to FIG. According to one or more aspects, a computer system as illustrated in FIG. 13 may be implemented in a computing device that can implement, perform, and / or execute any and / or all of the features, methods, and / May be included as part thereof. For example, computer system 1300 may represent a portion of a device and / or an element of an access point device. The device may be any computing device having a wireless unit such as an RF receiver. In one implementation, the system 1300 is configured to implement any of the methods described herein. FIG. 13 illustrates an example of a system that can perform the methods provided by various other implementations, and / or may be implemented in a host computer system, a remote kiosk / terminal, a sales device, a mobile device, a set top box, and / Which provides a schematic illustration of one implementation of a computer system 1300 capable of functioning as a system. Figure 13 is intended to provide only a general illustration of various components, any and / or all of which may be suitably utilized. Thus, Figure 13 extensively illustrates how individual system components can be implemented in relatively separate or relatively more integrated ways.

A computer system 1300 is shown that includes hardware components that can be electrically coupled (or otherwise communicate in any other manner) via bus 1305. [ The hardware components may include one or more processors 1310 that include, without limitation, one or more general purpose processors and / or one or more special purpose processors (e.g., digital signal processing chips, graphics acceleration processors, etc.); One or more input devices 1315 that may include, without limitation, a camera, a mouse, a keyboard, and the like; And one or more output devices 1320 that may include, without limitation, display units, printers, and the like. The computing device 1300 may also include a sensor (s), such as a temperature sensor, a power sensor, etc., to monitor the health of the system.

The computer system 1300 may further include (and / or communicate with) one or more non-volatile storage devices 1325, one or more non-volatile storage devices may store local and / or network accessible storage And may include without limitation, disk drives, drive arrays, optical storage devices, solid state storage devices such as random access memory ("RAM") and / or read only memory ("ROM" The read-only memory may be programmable, NVM-updatable, and so on. Such a storage device may be configured to implement any suitable data storage including, without limitation, various file systems, database structures, and the like.

The computer system 1300 may also include a communication subsystem 1330 that may be a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a chipset 802.11 devices, WiFi devices, WiMax devices, cellular communication facilities, etc.). Communication subsystem 1330 may allow data to be exchanged with a network (e.g., a network, such as described below), another computer system, and / or any other device as described herein. In many implementations, the computer system 1300 may further include a non-volatile memory 1335, which may include RAM or ROM devices, as described above. Computer system 1300 may also include a transceiver 1350 that facilitates communication with external entities by communication subsystem 1330.

The computer system 1300 also includes software that is presently located in a working memory 1335 that includes an operating system 1340, a device driver, an executable library, and / or other code such as one or more application programs 1345 Elements, and one or more application programs may include computer programs provided by various implementations, as described herein, and / or implement methods provided by other implementations, and / RTI > and / or < / RTI > By way of example only, the one or more procedures described for the discussed method (s) discussed above may be implemented with code and / or instructions executable by a computer (and / or a processor within the computer); In an aspect, such code and / or instructions may then be used to configure and / or adapt a general purpose computer (or other device) to perform one or more operations in accordance with the described methods.

A set of these instructions and / or code may be stored on a computer readable storage medium, such as the storage device (s) 1325 described above. In some cases, the storage medium may be contained within a computer system, such as computer system 1300. In other implementations, the storage medium may be separate from and / or be provided in an installation package, such as a computer system (e.g., a removable medium such as a compact disc) May be used to program, configure, and / or adapt. These instructions may include executable code and / or computer system 1300 (e.g., a variety of commonly available compilers, installers, compressors, etc.) that may be executed by computer system 1300 and / / Decompression utility), and / or install code that takes the form of executable code upon installation.

Substantial changes can be made according to specific requirements. For example, customized hardware may also be used, and / or certain elements may be implemented in hardware, software (including portable software such as an applet, etc.), or both. In addition, a connection to another computing device such as a network input / output device may be utilized.

Some implementations may utilize a computer system (e.g., computer system 1300) to perform the method in accordance with the present disclosure. For example, some or all of the procedures of the described method may be performed by one or more instructions (including operating system 1340 and / or other code, e.g., application programs 1345) included in work memory 1335 May be performed by the computer system 1300 in response to executing one or more sequences of instructions (e. Such instructions may be read into the working memory 1335 from another computer readable medium, such as one or more of the storage device (s) 1325. [ By way of example only, execution of a sequence of instructions contained in working memory 1335 may cause processor (s) 1310 to perform one or more procedures of the methods described herein.

The terms "machine readable medium" and "computer readable medium ", as used herein, refer to any medium that participates in providing data that allows the machine to operate in a particular manner. In one implementation, which is implemented using computer system 1300, various computer readable media may be involved in providing instructions / code to processor (s) 1310 for execution and / May be used to store and / or store code (e.g., as a signal). In many implementations, the computer-readable medium is a physical and / or typed storage medium. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical and / or magnetic disks, such as storage device (s) 1325. Volatile media include, without limitation, dynamic memory, such as work memory 1335. [ The transmission medium may include a coaxial cable (not shown) including wires including a bus 1305, as well as various components of the communication subsystem 1330 (and / or a medium in which the communication subsystem 1330 provides communication with other devices) , Copper wire, and optical fiber. Therefore, the transmission medium may also take the form of waves (including, without limitation, waves, acoustic waves and / or light waves, such as those generated during radio wave and infrared data communication).

Some implementations may use a computer system (e.g., processor 1310) to perform the method in accordance with the present disclosure. For example, some or all of the procedures of the described method may be performed in response to executing one or more sequences of one or more instructions (which may be included in an operating system and / or other code, e.g., an application program) And can be performed by a viewing apparatus. Such instructions may be read into the working memory from another computer readable medium, such as one or more of the storage device (s). By way of example only, execution of a sequence of instructions contained in a work memory may cause the processor (s) to perform one or more procedures of the methods described herein.

Also, implementations using the computer system described herein are not limited to being physically connected to a viewing device. The processing may occur in a wired or other device connected to the viewing device wirelessly. For example, instructions that execute instructions by a processor or phone or tablet within a phone may be included in these descriptions. Similarly, a network within a remote location may receive the processor and send data to the viewing device.

The terms "machine readable medium" and "computer readable medium ", as used herein, refer to any medium that participates in providing data that allows the machine to operate in a particular manner. In one implementation, which is implemented using processor 1310, various computer readable media may be involved in providing instructions / code to processor (s) 1310 for execution and / or may be associated with such instructions / (E. G., As a signal). ≪ / RTI > In many implementations, the computer-readable medium is a physical and / or typed storage medium. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical and / or magnetic disks. Volatile media include, without limitation, dynamic memory such as NVM memory or DDR3 RAM. The transmission medium includes, without limitation, coaxial cables, copper wires, and optical fibers, as well as the various components of the communication subsystem (and / or the medium over which the communication subsystem provides communication with other devices). Therefore, the transmission medium may also take the form of waves (including, without limitation, waves, acoustic waves and / or light waves, such as those generated during radio wave and infrared data communication).

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored or transmitted as one or more instructions or code on a computer readable medium. The computer readable medium may comprise a computer data storage medium. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementations of the techniques described in this disclosure. "Data storage medium" refers to a product as used herein and does not refer to a transient propagation signal. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM, or other optical disk, which may be used to store the desired program code in the form of an instruction or data structure, Storage, magnetic disk storage, or other magnetic storage device, NVM memory, or any other medium. As used herein, a disk includes a compact disk (CD), a laser disk, an optical disk, a digital versatile disk (DVD), a floppy disk and a Blu-ray disk, and a disk typically reproduces data magnetically On the other hand, a disc optically reproduces data with a laser. Combinations of the above should also be included within the scope of computer readable media.

The code may be executed by one or more processors, e.g., one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any previous structure or any other structure that is appropriate for the implementation of the techniques described herein. Further, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or included in a combined codec. Further, the techniques may be fully implemented with one or more circuits or logic elements.

The techniques of the present disclosure may be implemented in a wide variety of devices or devices, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various elements, modules, or units are described in this disclosure to emphasize the functional aspects of a device configured to perform the disclosed technique, but need not necessarily be realized by different hardware units, rather, Likewise, various units may be combined with a codec hardware unit, or may be provided by a set of interactive hardware units, including one or more processors as described above, along with appropriate software and / or firmware stored on a computer readable medium .

Various examples have been described. These and other examples are within the scope of the following claims.

Claims (40)

17. A plurality of nonvolatile memory (NVM) blades, each of the nvm blades from the plurality of nvm blades comprising: a plurality of nonvolatile memory (NVM) blades comprising nonvolatile memory (NVM);
A first processing entity coupled to the plurality of NVM blades,
Receiving a first input / output (I / O) request;
Determine if the I / O request is a read or write request;
In response to determining that the I / O request is a read request,
Determine a location in the target NVM blade from which the target NVM blades and data from the plurality of NVM blades are read;
Request and receive data associated with the first I / O request from the target NVM blade;
In response to determining that the I / O request is a write request;
Determining a location in the target NVM blade from which the target NVM blades and data from the plurality of NVM blades are stored;
A first processing entity configured to transmit data to the target NVM blade to store the data in the target NVM blade; And
A second processing entity coupled to the plurality of NVM blades,
Receiving a second input / output (I / O) request;
Determine if the I / O request is a read or write request;
In response to determining that the I / O request is a read request,
Determine a location in the target NVM blade from which the target NVM blades and data from the plurality of NVM blades are read;
Requesting and receiving data associated with the second I / O request from the target NVM blade;
In response to determining that the I / O request is a write request;
Determining a location in the target NVM blade from which the target NVM blades and data from the plurality of NVM blades are stored;
And a second processing entity configured to transmit data to the target NVM blade to store the data in the target NVM blade. 2. The storage device of claim 1, wherein the plurality of NVM blades are coupled to the first processing entity and the second processing entity using one or more routing entities. 2. The storage device of claim 1, wherein the first processing entity accesses the same target NVM blade servicing the first I / O request as a second processing entity serving the second I / O request. 2. The storage device of claim 1, wherein the first processing entity accesses the same location on the same target NVM blade servicing the first I / O request as a second processing entity serving the second I / O request. The storage device of claim 1, wherein the controller board comprises a first processing entity, a second processing entity, and a routing entity that routes data between the processing entity and the plurality of NVM blades. 6. The storage device of claim 5, wherein the first processing entity and the second processing entity access the target NVM blade through the same routing entity. 2. The system of claim 1 wherein a first controller board comprises the first processing entity and is coupled to the plurality of NVM blades using a first routing entity and the second controller board includes the second processing entity, Wherein the storage device is coupled to the plurality of NVM blades using an entity. 2. The method of claim 1, wherein the transmission of data between the first processing entity and the target NVM blade and the transmission of data between the second processing entity and the target NVM blade use a Peripheral Component Interconnect Express (PCIe) protocol ≪ / RTI > 2. The storage device of claim 1, wherein a first I / O request received by the first processing entity is received prior to one or more interfacing entities and is transmitted to the first processing entity via one of the plurality of routing entities. The storage device of claim 1, wherein the NVM storage medium is NAND flash. CLAIMS 1. A method of storing data,
Receiving a first input / output (I / O) request at a first processing entity, the first processing entity being coupled to a plurality of non-volatile memory (NVM) blades and having a plurality of NVM blades Each comprising a non-volatile memory (NVM);
Determining, at the first processing entity, whether the I / O request is a read or write request;
In response to determining that the I / O request is a read request,
Determining, by the first processing entity, a target NVM blade from the plurality of NVM blades and a location within a target NVM blade from which data is read; and
Requesting and receiving, by the first processing entity, data associated with the first I / O request from the target NVM blade; And
In response to determining that the I / O request is a write request;
In response to determining that the I / O request is a write request,
Determining, by the first processing entity, a target NVM blade from the plurality of NVM blades and a location in a target NVM blade where data is stored; and
Transmitting data from the first processing entity to the target NVM blade to store the data in the target NVM blade; And
Receiving, at a second processing entity, a second input / output (I / O) request, said second processing entity being coupled to said plurality of NVM blades;
Determining, at the second processing entity, whether the I / O request is a read or write request;
In response to determining that the I / O request is a read request,
Determining, at the second processing entity, a target NVM blade from the plurality of NVM blades and a location within a target NVM blade from which data is read;
Requesting and receiving, at the second processing entity, data associated with the second I / O request from the target NVM blade; And
In response to determining that the I / O request is a write request;
Determining, at the second processing entity, a location within a target NVM blade from which the target NVM blades and data from the plurality of NVM blades are stored; And
And transmitting data from the second processing entity to the target NVM blade to store the data in the target NVM blade. 12. The method of claim 11, wherein the plurality of NVM blades are coupled to the first processing entity and the second processing entity using one or more routing entities. 12. The method of claim 11, wherein the first processing entity accesses the same target NVM blade servicing the first I / O request as a second processing entity servicing the second I / O request. 12. The method of claim 11, wherein the first processing entity accesses the same location on the same target NVM blade servicing the first I / O request as a second processing entity serving the second I / O request. 12. The method of claim 11, wherein the controller board comprises a routing entity that routes data between the first processing entity, the second processing entity, and the processing entity and the plurality of NVM blades. 16. The method of claim 15, wherein the first processing entity and the second processing entity access the target NVM blade through the same routing entity. 12. The system of claim 11, wherein the first controller board comprises the first processing entity and is coupled to the plurality of NVM blades using a first routing entity and the second controller board includes the second processing entity and the second routing entity Wherein the plurality of NVM blades are coupled to the plurality of NVM blades using an entity. 12. The method of claim 11, wherein the transmission of data between the first processing entity and the target NVM blade and the transmission of data between the second processing entity and the target NVM blade use a Peripheral Component Interconnect Express (PCIe) protocol Lt; / RTI > 12. The method of claim 11 wherein a first I / O request received by the first processing entity is received prior to one or more interfacing entities and is transmitted to the first processing entity via one of the plurality of routing entities. 12. The method of claim 11, wherein the NVM storage medium is NAND flash. A first controller configured to operate in an active mode, the first controller configured to receive an input / output (I / O) request to store and retrieve data from a non-volatile memory (NVM) storage medium;
A second controller configured to operate in an active mode, the second controller being further configured to receive an I / O request to store and retrieve data from the NVM storage medium; And
A plurality of non-volatile memory (NVM) blades comprising an NVM storage medium, wherein at least one of the plurality of NVM blades is coupled to the first controller and the second controller for storing and retrieving data from the NVM storage medium A storage device comprising a plurality of non-volatile memory (NVM) blades. 22. The storage device of claim 21, wherein at least one of the plurality of NVM blades comprises a first routing interface in communication with the first controller and a second routing interface in communication with the second controller. 23. The storage device of claim 22, wherein the first routing interface communicates with the first controller using the PCIe protocol and the second routing interface communicates with the second controller. 25. The apparatus of claim 24, wherein the first controller comprises:
Receive a first I / O request;
Determining that the first I / O request is a request to store first data associated with the first I / O request on the NVM storage medium;
And to transmit the command and the first data to at least one of the plurality of NVM blades to store the first data in a first location. 25. The method of claim 24,
The second controller,
Receive a second I / O request;
Determine that the second I / O request is a request to store second data associated with the second I / O request on the NVM storage medium;
And send command information associated with the second I / O request to the first controller;
The first controller,
Receive the transmitted command information from the second controller;
And to send the store command to at least one of the plurality of NVM blades;
And wherein the second controller is further configured to transmit second data associated with the second I / O request to the one or more NVM blades. 22. The storage device of claim 21 wherein the first controller and the second controller are configured to simultaneously decode I / O requests for read operations and request data from the NVM storage medium. 22. The storage device of claim 21, wherein the first controller, the second controller, and the plurality of NVM blades are coupled to a power rail, wherein the power rail is powered by a plurality of power sources. 22. The storage device of claim 21, wherein at least one of the plurality of NVM blades comprises a first buffer coupled to a first routing interface for buffering instructions from the first controller. 29. The storage device of claim 28, wherein at least one of the plurality of NVM blades is further configured to discard the command from the first controller if the first buffer is full beyond a predetermined threshold. 22. The storage device of claim 21, wherein at least one of the plurality of NVM blades includes a command manager for arbitrating access to an NVM interface for commands from the first controller and the second controller. 32. The storage device of claim 30, wherein the command manager, upon detecting an error for the command, sends error information associated with the I / O request back to the controller from which the command originated. 32. The storage device of claim 31, wherein the first controller and the second controller communicate fault tolerance information with each other. 33. The storage device of claim 32, wherein the first controller and the second controller communicate with each other using fail-over information using a non-PCIe bridge. 33. The storage device of claim 32, wherein the failure tolerance information includes information about failure of a first I / O request from the first controller to one of the plurality of NVM blades. 32. The system of claim 31, wherein the first controller and the second controller include a printed circuit board including one or more processors for processing I / O requests and one or more routers for routing operations between the controller and the plurality of NVM blades (PCB). 32. The storage device of claim 31, wherein the first controller and the second controller are application specific integrated circuits (ASICs) each comprising processing logic and routing logic. A method of storing data on a storage device,
Receiving a first I / O request from a slave controller;
Determining that the first I / O request is a request to store first data associated with the first I / O request in a nonvolatile memory (NVM) storage medium;
Sending command information associated with the first I / O request to a master controller;
Receiving, at the master controller, the transmitted command information from the slave controller; And
And an NVM storage medium for storing instructions that use transmitted command information for the first data from the slave controller and a first I / O request from the master controller to store the first data in a first location To at least one of a plurality of non-volatile memory (NVM) blades. 39. The method of claim 37,
Receiving a second I / O request from the master controller;
Determining that the second I / O request is a request to store second data associated with the second I / O request on an NVM storage medium;
Sending an instruction to store the second data at a second location and transmitting the second data to at least one of a plurality of NVM blades comprising an NVM storage medium. 38. The method of claim 37, wherein the master and slave controllers use the PCIe protocol to communicate with the plurality of NVM blades. 39. The method of claim 37,
Receiving a second I / O request from the master controller;
Determining that the second I / O request is a request to read second data from a second location from the NVM storage medium;
Retrieving second data associated with the second I / O request from the NVM storage medium;
Receiving a third I / O request from the slave controller;
Determining that the third I / O request is a request to read third data from a third location from the NVM storage medium; And
Retrieving third data associated with the third I / O request from the NVM storage medium.

Images