WO2021248435A1

WO2021248435A1 - Method and apparatus for automatically generating summary document

Info

Publication number: WO2021248435A1
Application number: PCT/CN2020/095762
Authority: WO
Inventors: Lin Chen; Yang Shi
Original assignee: Bayer Aktiengesellschaft; Bayer Healthcare Co., Ltd.
Priority date: 2020-06-12
Filing date: 2020-06-12
Publication date: 2021-12-16
Also published as: CN113468861A; CN113468861B

Abstract

A method and an apparatus for automatically generating summary document (SD). The method comprises: determining a SD template file and multiple individual source files for generating a SD file of a target product, the SD file being used to provide an overview of detailed information described in the multiple individual source files (202); and automatically generating the SD file by using a SD generating engine, wherein the SD generating engine uses a machine learning model, and the machine learning model is input with the SD template file and the multiple individual source files and outputs the SD file (204).

Description

METHOD AND APPARATUS FOR AUTOMATICALLY GENERATING SUMMARY DOCUMENT

TECHNICAL FIELD

The present disclosure relates to a method and an apparatus for automatically generating summary document (SD) .

BACKGROUND

During new drug development process, a set of documents called CTDs (Common Technical Documents) are prepared by the company for regulatory submission purposes. These documents contain technical information of the drug substance and drug products to be reviewed by the authorities and approved for use in clinical trials. A summary document (SD) is normally required in order to provide an overview of the many detailed information described in individual source CTDs (source) . Current approach is to prepare the SD manually via transferring of TLFs (tables, listings and figures) from the source, transform the data into different formats, and edit the languages accordingly. Such an approach is time-consuming and lack of standardization. User-to-user variations exist and create challenges for subsequent activities of review and approval.

It is therefore an object of the present disclosure to provide a TAG (CTD Auto Generator) tool so as to automate this process by performing the required actions automatically, for example, through a web-based software (machine) .

SUMMARY

The present disclosure provides techniques for automatically generating summary document (SD) .

In one example aspect, a method for automatically generating summary document (SD) is disclosed. The method includes determining a SD template file and multiple individual source files for generating a SD file of a target product, the SD file being used to provide an overview of detailed information described in the multiple individual source files; and automatically generating the SD file by using a SD generating engine, wherein the SD generating engine uses a machine learning model, and the machine learning model inputs the SD template file and the multiple individual source files and outputs the SD file.

According to an embodiment, in response to a request for generating the SD file of the target product, determining the SD template file and the multiple individual source files for generating the SD file of the target product, wherein the multiple individual source files includes a set of Common Technical Documents (CTDs) .

According to an embodiment, in response to uploading a set of Common Technical Documents (CTDs) predetermined location, determining the SD template file and the multiple individual source files for generating the SD file of the target product, wherein the Common Technical Documents (CTDs) are used as the multiple individual source files.

According to an embodiment, the machine learning model is based on a user requirement sheet (URS) , wherein the URS comprises multiple items, with each item defining an operation between one of the multiple source file and the SD template file or an operation on the SD template.

According to an embodiment, each item further includes one or more information indicating target section in the SD template file, destination location in the SD template file, identification of the source files, content to be operated in the source files and variables.

According to an embodiment, the SD generating engine automatically performs the operations defined in the multiple items in sequence on item basis based on the information in the item.

According to an embodiment, the SD generating engine performs a first type of operation defined in a first part of the multiple items so as to automatically transfer content in the source file to a designated location of the SD template file in original format of the content based on the information in the item.

According to an embodiment, the content includes at least one of tables, listings and figures, flowchart, text and header in the source file.

According to an embodiment, the SD generating engine performs a second type of operation defined in a second part of the multiple items so as to automatically transform content in the source file to the SD template file by adapting to at least one of destination location, format and styles of the SD template file based on the information in the item.

According to an embodiment, the transform includes transforming contents from at least one of tables, listings and figures, flowchart, text and header in the source file into sentences in the SD template file.

According to an embodiment, the SD generating engine performs a third type of operation defined in a third part of the multiple items so as to automatically edit the content in the SD template file based on the information in the item.

According to an embodiment, the edit includes deleting, replacing and changing color or format of the content.

According to an embodiment, the edit includes editing the content in the SD template file based on a judgment within its logic.

According to an embodiment, the method further comprises output the SD template files as the SD file after performing all of the operations defined in the multiple items.

According to an embodiment, the machine learning model is generated from training data that includes rules defined by pharmaceutical professionals.

According to an embodiment, the method further comprises providing a user interface to receive the request and output the SD file.

According to an embodiment, the SD template file and the multiple individual source files are determined by input from the user interface.

According to an embodiment, the operations are performed in a visualized way through the user interface.

According to an embodiment, the target product is drug product, and the CTDs contain technical information of the drug substance and the drug product to be reviewed by the authorities and approved for use in clinical trials.

In another example aspect, an apparatus for automatically generating summary document (SD) is disclosed. The apparatus comprises one or more processors; and one or more storage devices storing instructions that when executed by the one or more processors cause the one or more processors to perform the operations of the respective method as described herein.

In another example aspect, a computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method as described herein.

With the method and system of the present disclosure, it not only improves the efficiency of SD writing, but also improves the standardization of document quality, as it will get rid of individual writing habits that are typically observed in human writers.

The details of one or more implementations are set forth in the accompanying attachments, the drawings, and the description below. Other features will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows concept of CTD Auto Generator (TAG) in the present disclosure.

FIG. 2 shows an example method for automatically generating summary document (SD) .

FIG. 3 shows an example of preparation-for-writing window of TAG.

FIG. 4 shows an example of a content transfer operation.

FIG. 5 shows an example of a content transformation operation.

FIG. 6 shows an example of a content editing operation.

FIG. 7 shows an example of writing window of TAG.

FIG. 8 shows a block diagram of an apparatus for implementing method for automatically generating summary document (SD) described in the present disclosure.

DETAILED DESCRIPTION

The present disclosure will be described more fully hereinafter with reference to the accompanying figures, in which embodiments of the disclosure are shown. This disclosure may, however, be embodied in many alternate forms and should not be construed as limited to the embodiments set forth herein. Accordingly, while the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure as defined by the claims. Like numbers refer to like elements throughout the description of the figures.

Fig. 1 shows concept of CTD Auto Generator (TAG) in the present disclosure. As shown in the upper part of Fig. 1, it can be seen that in current practice, when a company is preparing a set of CTDs (Technical Registration Documents) for regulatory submission purposes, at a first stage, a writer needs to find multiple sources files and extract lots of tables, listings and figures (TLF) from some source files (for example, P.1.02, P.3.1.01, P.5.1.01 and so on) and provide the TLF to a CTD draft (for example, T.50.20 draft) manually. In addition, the writer needs to conduct content transformation and editing on the TLF from other source files and provided the edited TLF to the CTD draft manually. Then, at a second stage, the writer needs to provide additional human inputs and justifications (for example, stability data) so as to generate the summary document (SD) for regulatory submission purposes.

The first stage has a workload of 70-80%for the whole process, and the second stage has a workload of 20-30%for the whole process. Obviously, such an approach is time-consuming and lack of standardization. In addition, different writer has different writing habits, which creates challenges for subsequent activities of review and approval.

On the other hand, as shown in the lower part of Fig. 1, TAG in the present application can extract the TLFs from multiple source files and provide the TLFs to a CTD template according to a predetermined rule so as to generate a CTD draft (for example, T.50.20 draft) automatically. Then, at a second stage, the writer provides additional human inputs and justifications (for example, stability data) so as to generate the summary document (SD) for regulatory submission purposes. With TAG, the workload at the first stage can be avoided. In addition, with the CTD template and TAG, it can improve the standardization of document quality and get rid of individual writing habits that are typically observed in human writers.

In one example, TAG can be opened at remote server via a web page link. When a user input a particular web page link, the TAG will be activated.

In another example, TAG can be opened at a local client computer via running a software.

FIG. 2 shows an example method for automatically generating summary document (SD) by TAG. The method 200 includes, at step 202, determining a SD template file and multiple individual source files for generating a SD file of a target product, the SD file being used to provide an overview of detailed information described in the multiple individual source files; and at step 204, automatically generating the SD file by using a SD generating engine, wherein the SD generating engine uses a machine learning model, and the machine learning model inputs the SD template file and the multiple individual source files and outputs the SD file.

In the present disclosure, the target product is drug product, and the CTDs contain technical information of the drug substance and the drug product to be reviewed by the authorities and approved for use in clinical trials.

In particular, at the step 202, firstly, TAG determines a SD template file (for example, T.50.20 draft) and multiple individual source files (for example, P.1.02, P.3.1.01, P.5.1.01 and so on) for generating a SD file of a target product.

TAG can be implemented as a web-based software (machine) and run at a server. TAG has a web-based interface that is able to manage administrational rights of users, source CTDs, and write SDs.

In one embodiment, in response to a request for generating the SD file of the target product, TAG can determine the SD template file and the multiple individual source files for generating the SD file of the target product, wherein the multiple individual source files includes a set of Technical Registration Documents (CTDs) . The SD file is used to provide an overview of detailed information described in the multiple individual source files.

For example, as shown in Fig. 3 which shows an example of preparation-for-writing window of TAG, the user can select one template file via the interface. In addition, the interface can provide multiple source files for selection. The source files for one product are stored in one folder. Each folder of “the source file name” corresponds to one product, and has stored a set of Technical Registration Documents (CTDs) related to the product in advance. The user can select one folder for the target product.

For example, when the user selects one template file and the folder “3527964” via the interface, TAG determines the selected template file as the SD template file, and determines the multiple individual source files in the folder “1002670” as the multiple individual source files for generating the SD file of the target product. The multiple individual source files includes a set of Technical Registration Documents (CTDs) .

In another embodiment, in response to uploading a set of Technical Registration Documents (CTDs) to a predetermined location, TAG can determine SD template file and the multiple individual source files for generating the SD file of the target product automatically, wherein the Technical Registration Documents (CTDs) are used as the multiple individual source files.

For example, when the user uploads a set of Technical Registration Documents (CTDs) to a folder, for example, the folder “3527964” , TAG is triggered to determine a SD template file, and determine the uploaded set of Technical Registration Documents (CTDs) as multiple individual source files for generating the SD file of the target product.

After determining the SD template file and the multiple individual source files, at step 204, TAG automatically generates the SD file via a SD generating engine. That is, TAG can be configured to include a SD generating engine, which can uses a machine learning model. The machine learning model inputs the SD template file and the multiple individual source files and outputs the SD file for the target product.

Referring back to Fig. 1, when the multiple source CTDs and the T.50.20 template are input the machine learning model, the machine learning model can output T.50.20 draft as the SD file.

In one embodiment, the machine learning model is based on a URS (User Requirement Sheet) . URS defines the detailed operations to be taught to TAG. URS comprise multiple items, and each item defines an operation between one of the multiple source file and the SD template file or an operation on the SD template. In one embodiment, different product may share the same URS. In another embodiment, different products may have different URS.

In one example, the machine learning model is generated from training data that includes rules defined by pharmaceutical professionals.

An example of URS is shown in the flowing table 1.

Table 1: URS for TAG

As shown in Table 1, the URS comprise 59 items, and each items includes one or more parameters selected from a set of parameters including Reference Template Section, Source CTD document, Step, TLF to copy, Destination in the template, Variable, and Action.

Reference Template Section indicates related section in the SD template.

Source CTD document indicates the source CTD documents to be operated.

Step indicates order of dissected actions of the SD writing process to be performed by TAG. For example, the process can be dissected in 85 steps. TAG performs the actions in an order indicated by the parameter Step.

TLF to copy indicates position and content of TLF in the source files to be copied to the SD template.

Destination in the template indicates destination position in the SD template where the TLF in the source files are copied to.

Variable indicates variable parameters such as date, product information and so on.

Action indicates the particular operations to be performed by TAG.

The operations defined in the URS mainly includes three types of operations. The first type of operation is content transfer. TAG will automatically copy and transfer contents (such as tables, listings and figures, flowchart, text and header in the source file) from the source CTDs to the designated location in the SD template without editing. During this process, the format of the copied content will not be changed as desired.

Fig. 4 shows an example of a content transfer operation. As shown in Fig. 4, when performing content transfer, “COM 654321” in header of the source CTD P.1.01 is copied to header of the SD template without editing. The format of the content “COM 654321” is not changed.

The second type of operation is content transformation. TAG can also transform content from source CTDs to the SD template between different formats, such as transforming contents from table into sentences. TAG can perform content transformation so as to automatically transform content in the source file to the SD template file by adapting to at least one of destination location, format and styles of the SD template file based on the parameters in the item.

The content includes at least one of tables, listings and figures, flowchart, text and header in the source file. The transform operation includes transforming contents of tables, listings, figures, flowchart, text, header in the source file and sentence into contents with different format.

In data transformation, the data transferred will be adapted and changed into different formats, e.g. data to table, tablet to sentence, sentence to table, etc.

Fig. 5 shows an example of a content transformation operation. As shown in Fig. 5, the information in a table of the source CTD is scattered and needs to be transformed into sentences. When performing content transformation, the information in the table of the source CTD is transformed and filled into the corresponding destination positions of the sentence in the SD template. As a result, a new sentence with the information of the table is generated.

The third type of operation is editing. TAG can edit the SD template not only by simple action such as deleting, replacing and changing color or format of the content, but also can made judgment within its logic in order to edit the SD template for each specific drug product.

Fig. 6 shows an example of a content editing operation. For example, as shown in Fig. 6, TAG will search three specific excipients from the source CTD i.e., magnesium stearate, sodium laurilsulfate and lactose monohydrate, indicate the drug formulation and edit the SD content accordingly.

As a result, when TAG is triggered to generate the SD file, TAG will load the URS, and automatically performs the operations defined in the multiple items of the URS in sequence on item basis based on the information in the item.

For example, when TAG is triggered to generate the SD file, TAG will load the URS, and automatically perform the operation defined the first item of the URS. The first item of the URS is as following:

Reference Template Section in the first item indicates that the related section in the SD template is the whole file.

Source CTD document in the first item indicates that the source CTD is P.1.01. xxxxxxxxx_0x. docx.

Step in the first item indicates that this step is the first step to be performed.

TLF to copy indicates that position of TLF to be copied in the source files is the left header, and content of TLF in the source files to be copied is “COM xxxx coated tablets …mg” .

Destination in the template indicates that the destination position in the SD template is “COM 123456 coated tablet 25 mg” . That is, “COM xxxx coated tablets …mg” in P.1.01. xxxxxxxxx_0x. docx is copied to “COM 123456 coated tablet 25 mg”in SD template.

Variable in the first item is “COM 123456 coated tablet 25 mg” .

Action in the first item is that replace with copied contents: "COM xxxx coated tablets …mg" . That is, TAG will replace “COM 123456 coated tablet 25 mg” in SD template with “COM xxxx coated tablets …mg” in P.1.01. xxxxxxxxx_0x. docx.

As a result, the TAG opens the source file P.1.01. xxxxxxxxx_0x. docx, copies the “COM xxxx coated tablets …mg” in the left header of the source file P.1.01. xxxxxxxxx_0x. docx, and replaces “COM 123456 coated tablet 25 mg” in SD template with “COM xxxx coated tablets …mg” in P.1.01. xxxxxxxxx_0x. docx.

After TAG performs the operations defined in the first item, TAG will automatically performs the operations defined in the second item in sequence.

Similarly, TAG will automatically performs the operations defined in the subsequent items in sequence. For example, TAG will delete "No material of animal origin is used for the manufacturing process of the drug product" text in section 4.5, as defined in step 33. If the check-box = Yes, TAG will Keep the sentence. Otherwise, TAG will delete the sentence, as defined in step 55.

In one example, the operations are performed in a visualized way through the user interface.

As shown in Fig. 7, the user interface can display the source CTDs that have been processed, and the source CTD that is being processed and writing information of the sections in the SD template.

After TAG performs all the operations defined in URS, TAG outputs the SD template files as the SD file. For example, the SD file can be output via the user interface.

The apparatus 800 may be embodied in a smartphone, tablet, computer, a server and so on. The apparatus 800 may include one or more processors 802, one or more memories 804. The processor (s) 802 may be configured to implement one or more methods described in the present document. The memory (memories) 804 may be used for storing data and instructions used for implementing the methods and techniques described herein.

During a preliminary validation run, TAG is able to achieve auto-writing of summary documents with only 1%error rate. On average the time saving realized is 20 working hour for the writing of one SD.

From the foregoing, it will be appreciated that specific embodiments of the presently disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the scope of the invention. Accordingly, the presently disclosed technology is not limited except as by the appended claims.

Implementations of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing unit” or “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program (also known as a program, software, software disclosure, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document) , in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code) . A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (disclosure specific integrated circuit) .

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

It is intended that the specification, together with the drawings, be considered exemplary only, where exemplary means an example. As used herein, the singular forms “a” , “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Additionally, the use of “or” is intended to include “and/or” , unless the context clearly indicates otherwise.

While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

A method for automatically generating summary document (SD) , comprising:

determining a SD template file and multiple individual source files for generating a SD file of a target product, the SD file being used to provide an overview of detailed information described in the multiple individual source files; and

generating the SD file automatically by using a SD generating engine, wherein the SD generating engine uses a machine learning model, and the machine learning model is input with the SD template file and the multiple individual source files and outputs the SD file.
The method of claim 1, wherein in response to a request for generating the SD file of the target product, determining the SD template file and the multiple individual source files for generating the SD file of the target product, wherein the multiple individual source files includes a set of Common Technical Documents (CTDs) .
The method of claim 1, wherein in response to uploading a set of Common Technical Documents (CTDs) to predetermined location, determining the SD template file and the multiple individual source files for generating the SD file of the target product, wherein the Common Technical Documents (CTDs) are used as the multiple individual source files.
The method of any of claims 1-3, wherein the machine learning model is based on a user requirement sheet (URS) , wherein the URS comprises multiple items, with each item defining an operation between one of the multiple source file and the SD template file or an operation on the SD template.
The method of claim 4, wherein each item further includes one or more information indicating target section in the SD template file, identification of the source files, step order, destination location in the SD template file, content to be operated in the source files, variable and operation.
The method of claim 5, wherein the SD generating engine automatically performs the operations defined in the multiple items in sequence on item basis based on the information in the item.
The method of claim 6, wherein the SD generating engine performs a first type of operation defined in a first part of the multiple items so as to automatically transfer content in the source file to a designated location of the SD template file in original format of the content based on the information in the item.
The method of claim 7, wherein the content includes at least one of tables, listings and figures, flowchart, text and header in the source file.
The method of claim 6, wherein the SD generating engine performs a second type of operation defined in a second part of the multiple items so as to automatically transform content in the source file to the SD template file by adapting to at least one of destination location, format and styles of the SD template file based on the information in the item.
The method of claim 9, wherein the content includes at least one of tables, listings and figures, flowchart, text and header in the source file.
The method of claim of 10, wherein the transform includes transforming contents of tables, listings, figures, flowchart, text, header in the source file and sentence into contents with different format.
The method of claim 6, wherein the SD generating engine performs a third type of operation defined in a third part of the multiple items so as to automatically edit the content in the SD template file based on the information in the item.
The method of claim 12, wherein the edit includes deleting, replacing and changing color or format of the content.
The method of claim 12, wherein the edit includes editing the content in the SD template file based on a judgment within its logic.
The method of claim 6, further comprising:

output the SD template files as the SD file after performing all of the operations defined in the multiple items.
The method of any of claims 1-3, wherein the machine learning model is generated from training data that includes rules defined by pharmaceutical professionals.
The method of claim 4, further comprising:

providing a user interface to receive the request and output the SD file.
The method of claim 17, wherein the SD template file and the multiple individual source files are determined by input received from the user interface.
The method of claim 17, wherein the operations are performed in a visualized way through the user interface.
The method of claim 2 or 3, wherein the target product is drug product, and the CTDs contain technical information of the drug substance and the drug product to be reviewed by the authorities and approved for use in clinical trials.
An apparatus for automatically generating summary document (SD) , comprising:

one or more processors; and

one or more storage devices storing instructions that when executed by the one or more processors cause the one or more processors to perform the operations of the respective method of any one of claims 1-20.
A computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of claims 1-20.