CN102971727B

CN102971727B - In software DSM device system, record dirty information

Info

Publication number: CN102971727B
Application number: CN201080068022.1A
Authority: CN
Inventors: S.严; Y.高; X.周; H.陈; B.萨哈; S.罗
Original assignee: Intel Corp
Current assignee: Intel Corp
Priority date: 2010-05-11
Filing date: 2010-05-11
Publication date: 2016-05-11
Anticipated expiration: 2030-05-11
Also published as: EP2569718B1; CN102971727A; US8516220B2; US20120023296A1; TWI443515B; WO2011140673A1; EP2569718A4; TW201214120A; EP2569718A1

Abstract

Can utilize the dirty position of page table entry system to come for the dirty information of software DSM device system log (SYSLOG). In certain embodiments, because this dirty position register system can obtain in some processor, so this can improve performance and substantially not increase expense. By extra order is provided, can be about every other the making for obtaining coherence of the dirty position of existing page table entry.

Description

In software DSM device system, record dirty information

Technical field

The present invention relates generally to software DSM device system, and more specifically, relate in software DSM device system and record dirty information.

Background technology

In distributed shared memory (DSM) system, the each node in trooping, except accessing the unshared privately owned memory of each node, is also accessed shared storage. Software DSM (SDSM) can be implemented or be implemented as programming library in operating system. If be implemented in programming library, it is addressable for developer, if but its in operating system, be implemented, it is inaccessible for developer.

Some SDSM systems depend on the page fault for multiple things, comprise dirty (dirty) record. Which data dirty record for accurately knowing and be modified and being therefore the mechanism of " dirty " since upper secondary data is synchronous. Aspect the amount of the coherence data that dirty record may must be transmitted in minimizing, be useful. SDSM system protection memory avoids being written into, and is therefore strengthened to wrong follow-up the writing in the processor (handler) that has wherein recorded accurate dirty information.

But page fault and memory protection are operations consuming time, it may cause the dirty record of lower-performance. Page fault may occupy the relatively large ratio of shared storage expense running time.

Summary of the invention

According to a first aspect of the invention, provide a kind of for record the method for dirty information in software DSM device system, comprise: use the dirty position of page table entry register system, for the dirty record in software DSM device system, it is dirty indicating page table entry with first; For providing the first extra order, virtual shared locked memory pages understands whether data are dirty for this software DSM device system; And for providing the second extra order, virtual shared locked memory pages understands whether data are dirty for operating system.

According to a second aspect of the invention, provide a kind of for record the equipment of dirty information in software DSM device system, comprise: for using the dirty position of page table entry register system, for the dirty record in software DSM device system, indicating page table entry with first is dirty device; Be used to virtual shared locked memory pages to provide the first extra order to understand that whether data are dirty devices for this software DSM device system; And be used to virtual shared locked memory pages to provide the second extra order to understand that whether data are dirty devices for operating system.

According to a third aspect of the invention we, provide a kind of for record the system of dirty information in software DSM device system, having comprised: graphic process unit, and general processor, it is coupled to described graphic process unit, described general processor and graphic process unit are configured to the node in software DSM device system, for using, to be used to indicate page table entry be dirty first to described general processor, for understanding that whether data are the first dirty extra orders for this software DSM device system, and for understanding that whether data are that the second dirty extra order is used the dirty position of page table entry register system for operating system, for the dirty record in described software DSM device system.

Brief description of the drawings

Fig. 1 is the flow chart for one embodiment of the present of invention; And

Fig. 2 is the schematic representation for one embodiment of the present of invention.

Detailed description of the invention

In certain embodiments, on some processors that comprise by the X 86 processor of Intel Company (SantaClara, California) manufacturing, the available dirty position of existing page table entry register system can be used as dirty register system in software DSM device. Because this register system exists, so it can be used to have the dirty record of relatively low expense in certain embodiments. In addition, in certain embodiments, it can improve the performance in the shared storage system with addressable X 86 processor. Therefore, in certain embodiments, programmability advantage can be implemented in the situation that not damaging performance.

In one embodiment, shared storage system can be implemented between X 86 processor and GPU. But, the invention is not restricted to this specific embodiment.

In X 86 processor, in the page table entry of the each page in page storage, there is dirty position bookkeeping (bookkeeping) system. If related pages is modified first, the dirty position in X 86 processor Lookup protocol page table entry. This removing be can't help to processor processing, but the responsibility of systems soft ware alternatively.

According to some embodiments of the present invention, and differently process bit clear in the tradition application of dirty position bookkeeping. The dirty position of page table recording mechanism can be used to be provided for identifying which page to the memory management module of systems soft ware and especially operating system and be more suitable in being gone out (pageout) by page and whether being necessary the ability of write-back (writeback) during at page page-out. Because this system realizes with hardware, so it can be than relying on the more effective dirty recording mechanism of page fault.

In one embodiment, the dirty position of query page list item is to know whether the page is modified. The dirty position of page table entry can be eliminated to indicate this page to be crossed over node in SDSM and synchronous. For example, a node can be X 86 processor, and another node can be graphic process unit. Then, in certain embodiments, page fault does not need to be used to dirty record.

A problem occurred, that is, in running time, more than one agency is will be for difference former thereby handle the dirty position of this page table entry. It is also removed in the dirty position of page table entry that the inquiry of operating system memory module determines about its page replacement, for example, and about the new loading from the second holder to memory.

By for each virtual shared locked memory pages provides two extra orders, can use the dirty position of page table entry this additional function for the dirty record for SDSM system, and do not disturb the operation of the dirty position using in this page table entry.

In one embodiment, for each virtual system locked memory pages provides two extra orders. An extra order is " SDSM is dirty ", and another is " (OS) is dirty for operating system ". Together with the dirty position of page table entry (" PTE is dirty ") of hardware supported, there are three positions in each virtually shared memory page in system.

Certainly,, although current description discussion arranges position, can complete identical thing by mark is set. Therefore, term herein " arranges position " and also comprises mark is set.

Embodiments of the invention provide the programming module of the general and graphics process applicator platform for combining, and comprise integrated or discrete device, multiple graphics processing card and mixed graph treatment system. In addition, embodiments of the invention provide the model of the shared storage between general processor and graphic process unit. Replace and share whole virtual address space, in one embodiment, only can share a part for virtual address space.

Data structure can be shared between processor, and pointer can be delivered to the opposing party from a side, and without any need for marshalling (, front and back serialization). For example, in one embodiment, game engine can comprise physical process (physics), artificial intelligence (AI) and play up. Physical process and AI code can be carried out best on general processor, can in graphic process unit, be carried out best and play up. Data structure may be shared between processor, such as scene graph. But in the embodiment of shared storage model, scene graph can reside in simply in shared storage and by two processors and visit.

In one embodiment, realized the whole programmed environment that comprises that language and running time are supported. The non-figure live load of multiple highly-parallels can be terminated to this environment. This implementation can be operated on OS (, having the different operating system operating on each processor). In addition, between processor, can allow user-level communication.

In one embodiment, memory model is such as the window that the shared virtual address between processor is provided to cut apart global address space (PGAS) language. Conventionally can in this space, be distributed in by programmer any data structure of sharing between processor. This system can be provided in the particular allocation memory function of distribute data in this space. Can annotate static variable with type measure word, so that they are assigned with in shared window. But, unlike PGAS language, in shared window, there is not the concept of similitude. This is because the data in the communal space are moved in the time that it becomes by each processor use between processor high speed buffer storage. And unlike PGAS implementation, the expression of pointer not share with private room between change. Remaining virtual address space is privately owned for processor. Under default situations, data are able to be assigned with in this space 130, and for the opposing party, are not visible. This is cut apart the amount that address space method can be kept relevant memory to needs and cuts down, and makes it possible to realize the more effective implementation for discrete device.

The embodiment of memory model can be extended to multiple graphs and process and mixed configuration. Especially the window of, sharing virtual address can be expanded by leap all devices. Any data structure of distributing in this shared address window can be visible for all agencies, and pointer in this space can be by free exchange. In addition, each agency has its oneself privately owned memory.

Can adopt ownership to expand the embodiment of shared storage model, be concerned with and optimize to make it possible to realize further. In shared virtual address window, arbitrary processor can specify it to have specific address block. If the address realm of sharing in window is had by general processor, general processor knows that graphic process unit can not access those addresses, and therefore does not need to maintain the coherence of those addresses and graphic process unit. The address that graphic process unit has is like this equally.

In one embodiment, shared data can be by copying to private room from the communal space by privatization. The non-pointer that comprises data structure can be simply by copying memory content by privatization. Containing in the pointer of data structure, must be converted into pointer in private data to the pointer of sharing in data at copy package.

Private data can be by copying to the communal space from private room by globalize, and makes it visible for other calculating. The non-pointer that comprises data structure can be simply by copying memory content by globalize. Containing in the pointer of data structure, must be converted into the pointer shared in data (privatization example contrary) to the pointer in private data at copy package.

In one embodiment, compiler generates two binary system-mono-for carrying out on each processor. Because two operating systems may have different executable formats, so generate two different carrying out (executable). Figure binary system comprises the code of carrying out in graphic process unit, and general binary system comprises general processor function. Run time library has the assembly for each processor, and it links with the application binary for each processor, to create carrying out for each processor. In the time that general binary system starts to carry out, it calls and loads figure executable running time of function. Two binary systems create the daemon thread that is used to inter-processor communication.

In one embodiment, each processor can have different page table and different virtual to physical storage conversion. Therefore,, for the content of the virtual address V at (for example, at point of release place) between synchronous processing device, the content of different physical address is by synchronous. But general processor cannot access graphics page table, and graphic process unit cannot be accessed general page table.

During initializing, a part for bus aperture space can be mapped in the user's space of application, and is utilized task queue, message queue and replica buffer and carrys out instantiation. For example, in the time that needs copy the page to graphic process unit from general processor, in the replica buffer of bus aperture, and carry out this buffer of mark by virtual address and manipulation designator by page copy running time. In figure side, daemon thread by with virtual address tag by the content replication of buffer in its address space. Therefore, can copy to two processors all addressable common buffer (bus aperture) from its address space according to the two step process copy-general processor of carrying out this, and graphic process unit obtains this page to its address space from common buffer. Because this aperture is pinned memory, so if processing is cut off context (context), the content in this aperture is not lost. This allows two and carries out processor asynchronously, and this may be crucial, because these two processors may have different operating system, and therefore context switching cannot be by synchronously. In addition, aperture space can be mapped in the user's space of application, therefore makes it possible to realize user class inter-processor communication. This makes application program stack more effective widely than experience OS driver stack.

In the time that graphic process unit is carried out acquisition operations, the corresponding page can be provided in this graphic process unit without access. At read operation place subsequently, if this page has been upgraded and discharged by general processor since graphic process unit collection last time, the page fault handler in graphic process unit copies the page from general processor. Catalogue and privately owned version number can be used to this to determine. Then the page is configured to read-only. At write operation place subsequently, page fault handler creates the backup copies of the page, and page marks, for reading-writing, and is increased to the city edition this shop of this page. At point of release place, carry out difference (diff) with the backup copies of this page, and change is sent to home position (homelocation), increase directory versions number simultaneously. Difference operation calculates the poor of memory location aspect between two pages (, this page and backup thereof), to find out the change of having made. General processor operation completes with symmetric mode. Therefore, gather with point of release between, processor they local storage and cache memory outside operate, and only clear and definite synchronous point place with communicate with one another.

In the time starting, this implementation determines address realm shared between processor, and guarantees that this address realm always keeps mapped (for example, using mmap or Linux). This address realm can dynamic growth, and must not be contiguous, although in 64 bit address space, running time, system can be retained in continuous blocks above.

With reference to Fig. 1, it is the clean page at first that algorithm starts from, and this means that all three positions (SDSMD,, osD and pteD) are zero, indicated in the state 10 in Fig. 1. Here, can have that SDSM inquiry, operating system (OS) inquiry, processor read, SDSM removes and OS removes, it does not change any one in these three positions, and does not represent any state change. When the dirty position of PTE is converted to for the moment from zero, X 86 processor is write 12 the dirty position of PTE (pteD) is automatically set, as shown in state 14.

In the time that operating system is removed the dirty position of this PTE, first it backup to dirty this PTE position in the dirty position of SDSM (SDSMD), and then reality is removed the dirty position of PTE and the dirty position of operating system (osD). This is illustrated at operation 42 places, and it is that operating system is removed. In this case, the vanishing of the dirty position of this PTE, the dirty position of this operating system remains zero, and the dirty position of SDSM becomes one, thereby keeps this dirty state for SDSM system. In the time that operating system is inquired about this dirty position, it reads the logic OR (OR) of the dirty position of PTE and the dirty position of operating system. In the time that SDSM removes this dirty position running time, first it backup to dirty this PTE position in the dirty position of operating system, and the actual dirty position of PTE and the dirty position of SDSM of removing then, as by operate 16 and state 18 indicated.

In the time that SDSM inquires about this dirty position running time, it reads the logic OR (OR) of dirty of the dirty position of PTE and SDSM. The backup of additional dirty position ensures that this dirty information is not lost for other agencies in the time that an agency is just upgrading dirty position. For example, even if SDSM system has been removed the dirty position of PTE and the dirty position of SDSM, in fact operating system make this dirty information be maintained in the dirty position of its operating system. Use logic OR (OR) operation to ensure that any agency can see complete dirty information from its oneself visual angle. For example, even if SDSM system has been removed the dirty position of this PTE and the dirty position of SDSM, operating system still can be seen the thing that it adopts the logic OR (OR) of the dirty position of PTE and the dirty position of operating system to see.

From state 18, processor is write the identical technical change of 20 uses to state 22. SDSM removes 24 and is converted to state 18. From state 22, operating system is removed 26 and is converted to state 28. From state 28, processor is write 34 and is converted to state 36. Operating system is removed 38 and is transformed back into state 28. From state 36, SDSM removes 40 and is converted to state 18. From state 18, operating system is removed 32 and is converted to state 10. From state 28, SDSM removes 30 and also transforms back into state 10.

Except maintaining dirty position, operating system can have system call so that inquiry/removing service is exposed to SDSM running time. Inquiry system calls obtains virtually shared memory address as input, and returns to Boolean to indicate whether this page is dirty. Scavenge system calls to be obtained virtually shared memory address and inputs as it. SDSM calls inquiry system running time and has called to know whether the page has been modified since upper subsynchronous and synchronous only at the dirty page in the situation that. SDSM running time calls scavenge system and calls after synchronous.

In certain embodiments, system call can obtain address realm as inputting to reduce system call expense. In this case, to call and can return to bitmap be dirty to indicate those pages address realm from the viewpoint of SDSM running time to inquiry system.

In one embodiment, extra order can be stored in and the independent form being associated for the kernel data structure of locked memory pages. In the case of comprising the X 86 processor architecture of the untapped position in page table entry, these two extra orders can be two untapped positions. In multiprocessor/multiple nucleus system, dirty bit clear operation also needs suitably to carry out translation look aside buffer (TLB) and lost efficacy and ended by force.

Embodiment as an alternative, because operating system is only removed the operational dirty position of memory paging, is to be can not paging by the page locking of SDSM virtually shared memory so also feasible. This becomes SDSM running time can remove unique agency of dirty position. After data between node in SDSM are synchronous, related pages is changed to clean conditions. If then the page is modified, it can be marked as dirty. Then, for data are synchronous next time, SDSM only inquires about the dirty position of each page running time, and only synchronously has the page of the dirty position of setting. After this, SDSM removes the dirty position of page table entry of the dirty page running time simply.

Computer system 130 shown in Fig. 2 can comprise hard disk drive 134 and removable media 136, and it is coupled to chipset core logic 110 by bus 104. Keyboard and mouse 120 or other traditional components can be coupled to chipset core logic via bus 108. In one embodiment, core logic can be coupled to graphic process unit 112 via bus 105, and main or general processor 100. General processor 100 can be X 86 processor or any processor with the dirty position of page table entry register system. Graphic process unit 112 can also be coupled to frame buffer 114 by bus 106. Frame buffer 114 can be coupled to display screen 118 by bus 107. In one embodiment, graphic process unit 112 can be multithreading, the multi-core parallel concurrent processor that uses single-instruction multiple-data (SIMD) architecture.

Graph processing technique described herein can be realized with various hardware architectures. For example, graphing capability can be integrated in chipset. Alternatively, can use discrete graphic process unit. As another embodiment, graphing capability can be realized by the general processor that comprises polycaryon processor.

Spread all in this description " embodiment " or " embodiment " mentioned to the meaning is to be included at least one implementation comprising in the present invention in conjunction with the described special characteristic of this embodiment, structure or characteristic. Therefore, the appearance of term " embodiment " or " in an embodiment " must not refer to identical embodiment. In addition, special characteristic, structure or characteristic can be set up according to other suitable forms of the particular implementation exception except illustrated, and all these forms can be contained in the application's claim.

Although described the present invention for the embodiment of limited quantity, it should be appreciated by those skilled in the art that resultant many modifications and variations. Anticipate and seek for, appended claim covers all such modifications and variation, within falling into this true spirit of the present invention and scope.

Claims

1. for record a method for dirty information in software DSM device system, comprising:

Use the dirty position of page table entry register system, for the dirty record in software DSM device system, it is dirty indicating page table entry with first;

For providing the first extra order, virtual shared locked memory pages understands whether data are dirty for this software DSM device system; And

For providing the second extra order, virtual shared locked memory pages understands whether data are dirty for operating system.

2. method according to claim 1, comprises hardware based dirty record is provided.

3. method according to claim 1, comprises the software DSM device system that realizes, and one of them node is that graphic process unit and another node are general processors.

4. method according to claim 3, comprises and uses the general processor with the dirty position of page table entry register system.

5. method according to claim 1, comprises in the time that the dirty position of page table entry is eliminated, and is kept for the state of the extra order of this software DSM device system.

6. method according to claim 5, comprises in the time that the dirty position of page table entry is eliminated, and is kept for the state of the extra order of operating system.

7. method according to claim 1, comprises and inquires about dirty position with the logic OR of two positions.

8. for record an equipment for dirty information in software DSM device system, comprising:

Be used for using the dirty position of page table entry register system, for the dirty record in software DSM device system, indicating page table entry with first is dirty device;

Be used to virtual shared locked memory pages to provide the first extra order to understand that whether data are dirty devices for this software DSM device system; And

Be used to virtual shared locked memory pages to provide the second extra order to understand that whether data are dirty devices for operating system.

9. equipment according to claim 8, further comprises the device for realizing software DSM device system, and in this software DSM device system, a node is that graphic process unit and another node are general processors.

10. equipment according to claim 8, further comprises the device for be kept for the state of the extra order of this software DSM device system in the time that the dirty position of page table entry is eliminated.

11. equipment according to claim 10, further comprise the device for be kept for the state of the extra order of operating system in the time that the dirty position of page table entry is eliminated.

12. equipment according to claim 8, further comprise the device for inquire about dirty position with the logic OR of two positions.

13. equipment according to claim 8, further comprise for by the virtually shared memory page locking of software DSM device system be can not paging device.

14. equipment according to claim 13, further comprise the device for page changes being become after synchronous between the node of software DSM device system to clean conditions.

15. 1 kinds for recording the system of dirty information in software DSM device system, comprising:

Graphic process unit; And

General processor, it is coupled to described graphic process unit, described general processor and graphic process unit are configured to the node in software DSM device system, for using, to be used to indicate page table entry be dirty first to described general processor, for understanding that whether data are the first dirty extra orders for this software DSM device system, and for understanding that whether data are that the second dirty extra order is used the dirty position of page table entry register system for operating system, for the dirty record in described software DSM device system.

16. systems according to claim 15, described general processor, in the time that dirty of page table entry is eliminated, is kept for the state of the extra order of this software DSM device system.

17. systems according to claim 16, described general processor, in the time that dirty of page table entry is eliminated, is kept for the state of the extra order of operating system.

18. systems according to claim 16, described general processor is for inquiring about dirty position with the logic OR of two positions.

19. systems according to claim 15, the virtually shared memory page of described general processor storing software Distributed Shared Memory System, the described virtually shared memory page is locked into can not paging.

20. systems according to claim 19, described general processor becomes clean conditions for after synchronous between the node of software DSM device system by page changes.