US20190079968A1

US20190079968A1 - Link-formative auxiliary queries applied at data ingestion to facilitate data operations in a system of networked collaborative datasets

Info

Publication number: US20190079968A1
Application number: US15/943,633
Authority: US
Inventors: David Lee GRIFFITH; Bryon Kristen Jacob; Shad William Reynolds
Original assignee: Data World Inc
Current assignee: Data World Inc
Priority date: 2016-06-19
Filing date: 2018-04-02
Publication date: 2019-03-14
Also published as: US11042537B2

Abstract

Various embodiments relate generally to data science and data analysis and computer software and systems to provide an interface between repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform configured to transmute associations between data arrangements of different formats or different data models to facilitate data operations, such as queries, configured to enhance, for example, an ingested dataset via link-formative queries to form, for example, interrelations among a system of networked collaborative datasets. For example, a method may include analyzing a dataset to detect data values with which to query against in a link-formative query, applying a link-formative query to a dataset, identifying results of the link-formative query, and forming an enhanced dataset to include results a link-formative queries in the dataset.

Description

CROSS-REFERENCE TO APPLICATIONS

This application is a continuation-in-part application of U.S. patent application Ser. No. 15/186,514, filed on Jun. 19, 2016, having Attorney Docket No. DAT-001 and titled “COLLABORATIVE DATASET CONSOLIDATION VIA DISTRIBUTED COMPUTER NETWORKS,” U.S. patent application Ser. No. 15/186,516, filed on Jun. 19, 2016, having Attorney Docket No. DAT-003 and titled “DATASET ANALYSIS AND DATASET ATTRIBUTE INFERENCING TO FORM COLLABORATIVE DATASETS,” U.S. patent application Ser. No. 15/454,923, filed on Mar. 9, 2017, having Attorney Docket No. DAT-009 and titled “COMPUTERIZED TOOLS TO DISCOVER, FORM, AND ANALYZE DATASET INTERRELATIONS AMONG A SYSTEM OF NETWORKED COLLABORATIVE DATASETS,” and U.S. patent application Ser. No. 15/927,004 filed on Mar. 20, 2018, having Attorney Docket No. DAT 019 and titled “LAYERED DATA GENERATION AND DATA REMEDIATION TO FACILITATE FORMATION OF INTERRELATED DATA IN A SYSTEM OF NETWORKED COLLABORATIVE DATASETS,” all of which is herein incorporated by reference in its entirety for all purposes. This application is also related to U.S. patent application Ser. No. 15/______, filed on MM, DD, YYYY, having Attorney Docket No. DAT-017 and titled “TRANSMUTING DATA ASSOCIATIONS AMONG DATA ARRANGEMENTS TO FACILITATE DATA OPERATIONS IN A SYSTEM OF NETWORKED COLLABORATIVE DATASETS.”

FIELD

Various embodiments relate generally to data science and data analysis, computer software and systems, and wired and wireless network communications to interface among repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform configured to transmute associations between data arrangements of different formats or different data models to facilitate data operations, such as queries, configured to enhance, for example, an ingested dataset via link-formative queries and resultant associations as, for example, created interrelations among a system of networked collaborative datasets.

BACKGROUND

Advances in computing hardware and software have fueled exponential growth in the generation of vast amounts of data due to increased computations and analyses in numerous areas, such as in the various scientific and engineering disciplines, as well as in the application of data science techniques to endeavors of good-will (e.g., areas of humanitarian, environmental, medical, social, etc.). Also, advances in conventional data storage technologies provide the ability to store the increasing amounts of generated data. Consequently, traditional data storage and computing technologies have given rise to a phenomenon in which numerous desperate datasets have reached sizes and complexities that tradition data-accessing and analytic techniques are generally not well-suited for assessing conventional datasets.
Conventional technologies for implementing datasets typically rely on different computing platforms and systems, different database technologies, and different data formats, such as CSV, TSV, HTML, JSON, XML, etc. Further, known data-distributing technologies are not well-suited to enable interoperability among datasets. Thus, many typical datasets are warehoused in conventional data stores, which are known as “data silos.” These data silos have inherent barriers that insulate and isolate datasets. Further, conventional data systems and dataset accessing techniques are generally incompatible or inadequate to facilitate data interoperability among the data silos.
Conventional approaches to generate and manage datasets, while functional, suffer a number of other drawbacks. For example, conventional data implementation typically may require manual importation of data from data files having “free-form” data formats. Without manual intervention, such data may be imported into data files with inconsistent or non-standard data structures or relationships. Thus, data practitioners generally are required to intervene to manually standardize the data arrangements. Further, manual intervention by data practitioners is typically required to decide how to group data based on types, attributes, etc. Manual interventions for the above, as well as other known conventional techniques, generally cause sufficient friction to dissuade the use of such data files. Thus, valuable data and its potential to improve the public well-being may be thwarted.
Moreover, traditional dataset generation and management are not well-suited to reducing efforts by data scientists and data practitioners to interact with data, such as via user interface (“UI”) metaphors, over complex relationships that link groups of data in a manner that serves their desired objectives, as well as the application of those groups of data to third party (e.g., external) applications or endpoints processes, such as statistical applications.
Other drawbacks in conventional approaches to traditional data storage and computing technologies include implementations of indexes to join or combine data in different tables using relational database techniques. During data operations, such as relational-based queries applied to tables, an index value representing a value needs to be computed and compared against the other values to search for queried data in one or more tables. Examples of joining two tables related by a column include use indexed associations between primary and foreign key. Computations to employ an index association increases as the number of index associations increases, thereby impeding optimal performance of computing resources, especially in instances in which index associations and corresponding computational comparisons are performed during one or more queries, such as each query.
Thus, what is needed is a solution for facilitating techniques to optimize data operations applied to datasets, without the limitations of conventional techniques.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments or examples (“examples”) of the invention are disclosed in the following detailed description and the accompanying drawings:

FIG. 1 is a diagram depicting an example of a dataset ingestion controller configured to transmute relationships among data in datasets to enhance querying and retrieving results thereof, according to some embodiments;

FIG. 2 is a diagram depicting an example of an atomized data point, according to some embodiments;

FIG. 3 is a diagram depicting an example of formatting a dataset to form a transmuted association, according to some examples;

FIG. 4 is a diagram depicting a dataset query engine configured to implement a query via a transmuted association against a graph data arrangement, according to some examples;

FIG. 5 is a flow diagram depicting an example of transmuting relationships among data during data ingestion to enhance querying and retrieving results thereof, according to some embodiments;

FIG. 6 is a diagram depicting examples of one or more auxiliary query generators for enriching ingested datasets, according to some examples;

FIG. 7 is a flow diagram depicting an example of implementing link-formative queries to enhance datasets, according to some embodiments;

FIGS. 8A to 8D are diagrams depicting computerized tools of a user interface to cause formation of transmuted associations to facilitate link-formative queries, according to some examples; and

FIG. 9 illustrates examples of various computing platforms configured to provide various functionalities to components of a collaborative dataset consolidation system, according to various embodiments.

DETAILED DESCRIPTION

Various embodiments or examples may be implemented in numerous ways, including as a system, a process, an apparatus, a user interface, or a series of program instructions on a computer readable medium such as a computer readable storage medium or a computer network where the program instructions are sent over optical, electronic, or wireless communication links. In general, operations of disclosed processes may be performed in an arbitrary order, unless otherwise provided in the claims.
A detailed description of one or more examples is provided below along with accompanying figures. The detailed description is provided in connection with such examples, but is not limited to any particular example. The scope is limited only by the claims, and numerous alternatives, modifications, and equivalents thereof. Numerous specific details are set forth in the following description in order to provide a thorough understanding. These details are provided for the purpose of example and the described techniques may be practiced according to the claims without some or all of these specific details. For clarity, technical material that is known in the technical fields related to the examples has not been described in detail to avoid unnecessarily obscuring the description.
FIG. 1 is a diagram depicting an example of a dataset ingestion controller configured to transmute relationships among data in datasets to enhance querying and retrieving results thereof, according to some embodiments. Diagram 100 depicts an example of a collaborative dataset consolidation system 110 that may be configured to consolidate one or more datasets to form collaborative datasets including data to enrich datasets by accessing via, for example, transmuted associations to access a community of user datasets external to a dataset 105 a, which may be associated with a user account implemented via computing device 114 b for a user 114 a.
Collaborative dataset consolidation system 110 may be configured to consolidate one or more datasets to form collaborative datasets. A collaborative dataset, according to some non-limiting examples, is a set of data that may be configured to facilitate data interoperability over disparate computing system platforms, architectures, and data storage devices. Further, a collaborative dataset may also be associated with data configured to establish one or more associations (e.g., metadata) among subsets of dataset attribute data for datasets and multiple layers of layered data, whereby attribute data may be used to determine correlations (e.g., data patterns, trends, etc.) among the collaborative datasets.
Further, collaborative dataset consolidation system 110 may be configured to convert a dataset in a first format (e.g., a tabular data structure or an unstructured data arrangement) into a second format (e.g., a graph), and is further configured to interrelate data between a table and a graph, whereby at least one association between multiple tables may be transmuted to form a transmuted association between multiple graphs. Data 101 a may be received in the following examples of data formats: CSV, XML, JSON, XLS, MySQL, binary, free-form, unstructured data formats (e.g., data extract from a PDF file using optical character recognition), etc., among others. Therefore, data operations, such as queries, that are designed for either a tabular or graph data structure may be implemented to access data in both formats or data arrangements. For example, a query applied to a collaborative dataset may be accomplished using either a query designed to access a tabular or relational data arrangement (e.g., a SQL query or variant thereof) or another query designed to access a graph data arrangement (e.g., a SPARQL operation or a variant thereof), which may include data for a collaborative dataset. Further, a query designed to access a tabular data arrangement may be applied differently to, or computed differently, to access a graph data arrangement, at least in one example. Therefore, a collaborative dataset of equivalent data may be configured to be accessible by different queries and programming languages, according to some examples.
Collaborative dataset consolidation system 110 is shown in this example to include a dataset ingestion controller 120 and a dataset attribute manager 161, either of which, or both, may be configured to identify and/or form transmuted associations between dataset 105 a and one or more other datasets 115 a associated with, for example, another format (e.g., a graph data arrangement), which may be stored in repository 140. Collaborative dataset consolidation system 110 may present a correlation via, for example, computing device 114 b to provide dataset-related information to user 114 a. Computing device 114 a may be configured to interoperate with collaborative dataset consolidation system 110 to perform any number of data operations, including queries over interrelated or linked datasets. Thus, a community of users 114 a and 108 a, as well as any other participating user, may discover, share, manipulate, and query dataset-related information of interest in association with collaborative datasets. Collaborative datasets, with or without associated dataset attribute data, may be used to facilitate easier collaborative dataset interoperability (e.g., consolidation) among sources of data that may be differently formatted at origination.
To illustrate formation of a transmuted association, consider an example in which dataset ingestion controller 120 receives data 101 a representing a dataset 105 a, whereby dataset 105 a, while being depicted as a being formatted a table in data 101 a, may be disposed in any data format, arrangement, structure, etc., or may be unstructured. Dataset ingestion controller 120 may arrange data in dataset 105 a into a first data arrangement, or may identify that data in dataset 105 a as being disposed in a data arrangement, such as a first data arrangement. In this example, dataset 105 a may be disposed in a tabular data arrangement that format converter 137 may convert into a second data arrangement, such as a graph data arrangement 142 a. As such, data in a field (e.g., a unit of data in a cell at a row and column) of a table may be disposed in association with a node in a graph (e.g., as a unit of data as linked data). A data operation (e.g., a query) may be applied as either a query against a tabular data arrangement (e.g., based on a relational data model) or graph data arrangement (e.g., based on a graph data model, such using RDF). Since equivalent data disposed in both a field of a table and a node of a graph, either the table or the graph may be used to perform queries and other data operations. Similarly, data datasets disposed in one or more other graph data arrangements 142 b may be disposed or otherwise mapped (e.g., linked) as a dataset into a tabular data arrangement 115 a.
Data analyzer 130 may be configured to identify a referential indicator 113 a for at least a subset of dataset 105 a and another referential indicator 123 a for at least another subset of dataset 115 a. Dataset 115 a may be different than dataset 105 a (e.g., at least a portion of dataset 115 a may be stored or generated external to collaborative dataset consolidation system 110 or repository 140). In some examples, data analyzer 130 may be configured to identify a value 116 a and another value 116 b that may be equivalent, and an association may be formed between values 116 a and 116 b. In at least one case, one of values 116 a and 116 b is a unique value. As shown, value 116 a in a row of dataset 105 a, as a referential indicator, may be used to reference via 106 to value 116 b, which, in turn, also may be used as a referential indicator back to value 116 a. Note, too, that value 116 a in a row of dataset 105 a may be used to employ an identifier of the row as reference 106 (or any portion of dataset 105 a) to another row that includes value 116 b (or any portion of dataset 115 a). In some examples, a subset of dataset 105 a may include one or more columns that include one or more referential indicators 113 a, whereas a subset of dataset 115 a may include one or more columns that include one or more referential indicators 123 a.
According to some examples, dataset analyzer 130 and any of its components, including inference engine 132, may be configured to analyze values 116 a and 116 b to detect or determine equivalency (e.g., during ingestion) and whether one of values 116 a and value 116 b may be used as a reference indicator to the other. For example, inference engine 132 may be configured to analyze data to determine or infer that values 116 a and 116 b are equivalent (e.g., as equivalent numbers, equivalent strings, equivalent classifications, such as data values being zip codes, equivalent data types, etc., or any other equivalent dataset attribute). In the example shown, inference engine 132 (and/or data classifier 124) may determine or infer that data values in column 113 a may include data classified as “zip codes,” whereby data 101 d may be transmitted to a user interface, such as data ingestion interface 102, to confirm whether column data 113 a includes zip codes of a dataset preview 104 for dataset 105 a. Selection device 179 may be used to receive an input via interface 106 as to whether column 113 a includes zip codes (e.g., via selection of user input 171) or not (e.g., via selection of user input 173). A user may confirm formation of association 107 via data 101 d. In alternative implementations, a determination of zip codes associations may be predicted or probabilistically determined by performing various computations, by matching data patterns, etc. For example, equivalency of values may be determined or predicted based on statistical computations, including Bayesian techniques, deep-learning techniques, etc. In view of the foregoing, data ingestion interface 102 may facilitate data equivalency determinations and dataset enrichment for dataset 105 a “in-situ” or “in-line” (e.g., in real time or near real time) to enhance expansion of data in atomized dataset generation during the dataset ingestion and/or graph formation processes with, for example, formation of a transmuted association.
Further, data analyzer 130 may be configured to determine or form an association 107 between referential indicator 123 a and referential indicator 123 b, and, thus, between value 116 a and another value 116 b. In some examples, one of one or more associations 107 between a unique value 116 a may be determined or formed with one or more equivalent values 116 b (or conversely). According to some examples, association 107 may include an indexed-based association, whereby one of values 116 a and 116 b may be stored for a tabular data arrangement as an index that may be used to relate (e.g., join) data from one or more tables using relational database techniques. During data operations, such as queries, performed on tabular data arrangements of datasets 105 a and 115 a, may implement an index value representing one of values 116 a and 116 b for comparing against (e.g., as an equality-determination) the other value to search for queried data. According to some examples, referential indicator 113 a (and/or the data values therein, such as value 116 a) may be referred to, or implemented as, a primary key, whereas referential indicator 123 a (and/or the data values therein, such as value 116 b) may be referred to, or implemented as, a foreign key. Or, conversely, referential indicator 123 a may be a primary key and referential indicator 113 a may be a foreign key.
Data ingestion controller 120 and/or any of its constituent components may be configured to transmute association 107 to form a transmuted association as a link 111 between value 116 a (as one of referential indicator 113 a) and value 116 b (as one of referential indicator 123 a). As shown, transmuted association 107 may form link 111 between, for example, node 199 a and node 199 b, which include data representing value 116 a and value 116 b, respectively. Transmuted association 107, as link 111, then may facilitate integration of dataset 115 a with dataset 105 a, thereby forming a merged dataset as an enriched dataset. When queried or modified subsequently, data enhancement manager 136 may be configured to manage the enrichment (i.e., supplementation of dataset 105 a). According to some examples, a transmuted association 207 may refer to, at least in some cases, a transmutation of an association between or among primary key data and a foreign key data, in a tabular data model, that may be applied or implemented within a graph data model.
In view of the foregoing, the structures and/or functionalities depicted in FIG. 1 illustrate dataset ingestion controller 120 being configured to analyze, detect, and form transmuted associations between dataset 105 a and one or more other datasets 115 a, and the data therein, during ingestion of a set of data 105 a to facilitate expeditious data operations, such as queries, that include the transmuted associations. According to some examples, a query may be applied via linked data (e.g., including link 111) of graph data arrangements 142 a and 142 b, thereby foregoing computing equality operations to detect whether referential indicator 113 a (e.g., value 116 a) matches referential indicator 123 a (e.g., value 116 b). In some examples, an equivalency determination may be performed during ingestion of dataset 105 a, with a transmuted association obviating such determinations in relation to, for example, queries or other data operations. Therefore, queries using one or more links 111 based on transmuted associations may enhance computational performance by, among other things, foregoing computations or calculations relating to the use of indices.
Further, since the structures and/or functionalities of collaborative dataset consolidation system 110 enable a query written against either against a tabular data arrangement or graph data arrangement to extract data from a common set of data, a user (e.g., data scientist) that favors usage of either SQL-equivalent query languages or SPARQL-equivalent query languages, or any other equivalent programming languages, may implement any of the foregoing languages. As such, a data practitioner may more easily query a common data set of data using a familiar query language. To illustrate, consider a query may be directed to a tabular data arrangement to join dataset 105 a to a different dataset 115 a to extract data from both datasets, whereby transmuted association 107 may be used to retrieve results of the query. As shown, a user 108 a may apply a relational query 192 on interface 194 of computing device 108 b to query a graph data arrangement 196.
In one example, a command conforming to relational database operations may be used to query link 111 in a graph database. An example of such a command may include a statement having a syntax associated with relational data operations for accessing a relational data structure. Thus, a SQL-like language or command may be used to access via a transmuted association a graph database to obtain performance enhancements by foregoing indexed-based associations, especially as the number of different links 111 may be integrated with an increasing number of dataset integrations.
Further to diagram 100, format converter 137 may be configured to convert dataset 105 a into another format, such as a graph data arrangement 142 a, which may be transmitted as data 101 c for storage in data repository 140. Graph data arrangement 142 a in diagram 100 may be linkable (e.g., via links 111) to other graph data arrangements to form a collaborative dataset. Also, format converter 137 may be configured to generate ancillary data or descriptor data (e.g., metadata) that describe attributes associated with each unit of data in dataset 105 a. The ancillary or descriptor data can include data elements describing attributes of a unit of data, such as, for example, a label or annotation (e.g., header name) for a column, an index or column number, a data type associated with the data in a column, etc. In some examples, a unit of data may refer to data disposed at a particular row and column of a tabular arrangement (e.g., originating from a cell in dataset 105 a). In some cases, ancillary or descriptor data may be used by data classifier 134 determine whether data may be classified into a certain classification, such as where a column of data includes “zip codes.”
Layer data generator 136 may be configured to form linkage relationships of ancillary data or descriptor data to data in the form of “layers” or “layer data files.” Implementations of layer data files may facilitate the use of supplemental data (e.g., derived or added data, etc.) that can be linked to an original source dataset, whereby original or subsequent data may be preserved. As such, format converter 137 may be configured to form referential data (e.g., IRI data, etc.) to associate a datum (e.g., a unit of data) in a graph data arrangement to a portion of data in a tabular data arrangement. Thus, data operations, such as a query, may be applied against a datum of the tabular data arrangement as the datum in the graph data arrangement. An example of a layer data generator 136, as well as other components of collaborative dataset consolidation system 110, may be described in U.S. patent application Ser. No. 15/927,004 filed on Mar. 20, 2018, having Attorney Docket No. DAT-019 and titled “LAYERED DATA GENERATION AND DATA REMEDIATION TO FACILITATE FORMATION OF INTERRELATED DATA IN A SYSTEM OF NETWORKED COLLABORATIVE DATASETS.”
According to some embodiments, a collaborative data format may be configured to, but need not be required to, format converted dataset 105 a as an atomized dataset. An atomized dataset may include a data arrangement in which data is stored as an atomized data point that, for example, may be an irreducible or simplest data representation (e.g., a triple is a smallest irreducible representation for a binary relationship between two data units) that are linkable to other atomized data points, according to some embodiments. As atomized data points may be linked to each other, data arrangement 142 a may be represented as a graph, whereby converted dataset 105 a (i.e., atomized dataset 105 a) may form a portion of a graph. In some cases, an atomized dataset facilitates merging of data irrespective of whether, for example, schemas or applications differ. Further, an atomized data point may represent a triple or any portion thereof (e.g., any data unit representing one of a subject, a predicate, or an object), according to at least some examples.
As further shown, collaborative dataset consolidation system 110 may include a dataset attribute manager 161, which includes an attribute correlator 163 and a data derivation calculator 165. Dataset ingestion controller 120 and dataset attribute manager 161 may be communicatively coupled to dataset ingestion controller 120 to exchange dataset-related data 107 a and enrichment data 107 b, both of which may exchange data from a number of sources (e.g., external data sources) that may include dataset metadata 103 a (e.g., descriptor data or information specifying dataset attributes), dataset data 103 b (e.g., some or all data stored in system repositories 140, which may store graph data), schema data 103 c (e.g., sources, such as schema.org, that may provide various types and vocabularies), ontology data 103 d from any suitable ontology and any other suitable types of data sources. One or more elements depicted in diagram 100 of FIG. 1 may include structures and/or functions as similarly-named or similarly-numbered elements depicted in other drawings, or as otherwise described herein, in accordance with one or more examples.
In this example, dataset ingestion controller 120 is shown to communicatively coupled to a user interface, such as data ingestion interface 102 via one or both of a user interface (“UI”) element generator 180 and a programmatic interface 190 to exchange data and/or commands (e.g., executable instructions) for facilitating data enrichment of dataset 105 a. UI element generator 180 may be configured to generate data representing UI elements to facilitate the generation of data ingestion interface 102 and graphical elements thereon. For example, UI generator 180 may cause generation UI elements, such as a container window (e.g., icon to invoke storage, such as a file), a browser window, a child window (e.g., a pop-up window), a menu bar (e.g., a pull-down menu), a context menu (e.g., responsive to hovering a cursor over a UI location), graphical control elements (e.g., user input buttons, check boxes, radio buttons, sliders, etc.), and other control-related user input or output UI elements. Programmatic interface 190 may include logic configured to interface collaborative dataset consolidation system 110 and any computing device configured to present data ingestion interface 102 via, for example, any network, such as the Internet. In one example, programmatic interface 190 may be implemented to include an applications programming interface (“API”) (e.g., a REST API, etc.) configured to use, for example, HTTP protocols (or any other protocols) to facilitate electronic communication. According to some examples, user interface (“UI”) element generator 180 and a programmatic interface 190 may be implemented in collaborative dataset consolidation system 110, in a computing device associated with data ingestion interface 102, or a combination thereof. UI element generator 180 and/or programmatic interface 190 may be referred to as computerized tools, or may facilitate employing a user interface as a computerized tool, according to some examples.
In at least one example, additional datasets to enhance dataset 142 a may be determined through collaborative activity, such as identifying that a particular dataset may be relevant to dataset 142 a based on electronic social interactions among datasets and users. For example, data representations of other relevant dataset to which links may be formed may be made available via a dataset activity feed. A dataset activity feed may include data representing a number of queries associated with a dataset, a number of dataset versions, identities of users (or associated user identifiers) who have analyzed a dataset, a number of user comments related to a dataset, the types of comments, etc.). Thus, dataset 142 a may be enhanced via “a network for datasets” (e.g., a “social” network of datasets and dataset interactions). While “a network for datasets” need not be based on electronic social interactions among users, various examples provide for inclusion of users and user interactions (e.g., social network of data practitioners, etc.) to supplement the “network of datasets.” According to various embodiments, one or more structural and/or functional elements described in FIG. 1, as well as below, may be implemented in hardware or software, or both. Examples of one or more structural and/or functional elements described herein may be implemented as set forth in one or more of U.S. patent application Ser. No. 15/186,514, filed on Jun. 19, 2016, having Attorney Docket No. DAT-001 and titled “COLLABORATIVE DATASET CONSOLIDATION VIA DISTRIBUTED COMPUTER NETWORKS,” U.S. patent application Ser. No. 15/186,517, filed on Jun. 19, 2016, having Attorney Docket No. DAT-004 and titled “QUERY GENERATION FOR COLLABORATIVE DATASETS,” and U.S. patent application Ser. No. 15/454,923, filed on Mar. 9, 2017, having Attorney Docket No. DAT-009 and titled “COMPUTERIZED TOOLS TO DISCOVER, FORM, AND ANALYZE DATASET INTERRELATIONS AMONG A SYSTEM OF NETWORKED COLLABORATIVE DATASETS,” each of which is herein incorporated by reference.
FIG. 2 is a diagram depicting an example of an atomized data point, according to some embodiments. Diagram 250 depicts a portion 251 of an atomized dataset that includes an atomized data point 254. In this example, atomized data point 254 and/or its constituent components may facilitate implementation of a transmuted association within a graph data arrangement based on a graph data model. As shown, transmuted associated 270 may be implemented to form a link between a unit of data 243, which may represent a city name, and a unit of data 244, which may represent a magnitude (“MAG”) of a tornado. Unit of data 243 may be associated with a column of data 241, which may serve as a primary key in a tabular data structure 230, whereas unit of data 244 may be associated with a column of data 242, which may serve as a foreign key in a tabular data structure 232.
In some examples, an atomized dataset may be formed by converting a tabular data format into a format associated with the atomized dataset. In some cases, portion 251 of the atomized dataset can describe a portion of a graph that includes one or more subsets of linked data. Further to diagram 250, one example of atomized data point 254 is shown as a data representation 254 a, which may be represented by data representing two data units 252 a and 252 b (e.g., objects) that may be associated via data representing an association 256 with each other. One or more elements of data representation 254 a may be configured to be individually and uniquely identifiable (e.g., addressable), either locally or globally in a namespace of any size. For example, elements of data representation 254 a may be identified by identifier data 290 a, 290 b, and 290 c (e.g., URIs, URLs, IRIs, etc.).
In some embodiments, atomized data point 254 a may be associated with ancillary data 153 to implement one or more ancillary data functions. For example, consider that association 256 spans over a boundary between an internal dataset, which may include data unit 252 a, and an external dataset (e.g., external to a collaboration dataset consolidation), which may include data unit 252 b. Ancillary data 253 may interrelate via relationship 280 with one or more elements of atomized data point 254 a such that when data operations regarding atomized data point 254 a are implemented, ancillary data 253 may be contemporaneously (or substantially contemporaneously) accessed to influence or control a data operation. In one example, a data operation may be a query and ancillary data 253 may include data representing authorization (e.g., credential data) to access atomized data point 254 a at a query-level data operation (e.g., at a query proxy during a query). Thus, atomized data point 254 a can be accessed if credential data related to ancillary data 253 is valid (otherwise, a request to access atomized data point 254 a (e.g., for forming linked datasets, performing analysis, a query, or the like) without authorization data may be rejected or invalidated). According to some embodiments, credential data (e.g., passcode data), which may or may not be encrypted, may be integrated into or otherwise embedded in one or more of identifier data 290 a, 290 b, and 290 c. Ancillary data 253 may be disposed in other data portion of atomized data point 254 a, or may be linked (e.g., via a pointer) to a data vault that may contain data representing access permissions or credentials.
Atomized data point 254 a may be implemented in accordance with (or be compatible with) a Resource Description Framework (“RDF”) data model and specification, according to some embodiments. An example of an RDF data model and specification is maintained by the World Wide Web Consortium (“W3C”), which is an international standards community of Member organizations. In some examples, atomized data point 254 a may be expressed in accordance with Turtle (e.g., Terse RDF Triple Language), RDF/XML, N-Triples, N3, or other like RDF-related formats. As such, data unit 252 a, association 256, and data unit 252 b may be referred to as a “subject,” “predicate,” and “object,” respectively, in a “triple” data point (e.g., as linked data). In some examples, one or more of identifier data 290 a, 290 b, and 290 c may be implemented as, for example, a Uniform Resource Identifier (“URI”), the specification of which is maintained by the Internet Engineering Task Force (“IETF”). According to some examples, credential information (e.g., ancillary data 253) may be embedded in a link or a URI (or in a URL) or an Internationalized Resource Identifier (“IRI”) for purposes of authorizing data access and other data processes. Therefore, an atomized data point 254 may be equivalent to a triple data point of the Resource Description Framework (“RDF”) data model and specification, according to some examples. Note that the term “atomized” may be used to describe a data point or a dataset composed of data points represented by a relatively small unit of data. As such, an “atomized” data point is not intended to be limited to a “triple” or to be compliant with RDF; further, an “atomized” dataset is not intended to be limited to RDF-based datasets or their variants. Also, an “atomized” data store is not intended to be limited to a “triplestore,” but these terms are intended to be broader to encompass other equivalent data representations.
Examples of triplestores suitable to store “triples” and atomized datasets (and portions thereof) include, but are not limited to, any triplestore type architected to function as (or similar to) a BLAZEGRAPH triplestore, which is developed by Systap, LLC of Washington, D.C., U.S.A.), any triplestore type architected to function as (or similar to) a STARDOG triplestore, which is developed by Complexible, Inc. of Washington, D.C., U.S.A.), any triplestore type architected to function as (or similar to) a FUSEKI triplestore, which may be maintained by The Apache Software Foundation of Forest Hill, Md., U.S.A.), and the like.
FIG. 3 is a diagram depicting an example of formatting a dataset to form a transmuted association, according to some examples. Diagram 300 depicts a dataset 310 including subsets of data, including a column of data values representing “zip codes,” disposed in a tabular data arrangement. Format converter 337 may be configured to convert dataset 310 into another format, such as a graph data arrangement. In this case, rows, including row 313 a, of dataset 310 may be associated or otherwise linked to row nodes 321 of a graph (not shown). In some implementations, nodes 321 may also reference data representing entities, records, and the like. Also shown, columns, such as column 315 a, of dataset 310 may be associated with column nodes 302 of a graph (not shown). Other nodes, links, references, etc. of a graph may be implemented (not shown). As shown, a unit of data 311 includes a string or an integer representing a zip code “83631.”
Dataset ingestion controller 320 may be configured to analyze data of dataset 310 against data in a pool of one or more dataset, any of which may be linked to another dataset. An example of one or more datasets is depicted as dataset 370, which may be disposed in a graphical data arrangement. Dataset 370 may be associated with a graph including row nodes 350 and columns nodes 357, as well as other nodes, links, references, etc. (not shown). Here, links from a graph (e.g., via nodes 350 and 357) to units of data may be usable to present dataset 370 in a tabular data arrangement including rows, such as row 313 b, and columns, such as columns 315 b, 315 c, and 315 d.
Dataset ingestion controller 320 may be configured to match data in dataset 310 against data in a pool of data including dataset(s) 370. In the example shown, a value of a unit of data 311 (of dataset 310) may match a value of a unit of data 323 (of dataset 370). Dataset ingestion controller 320 also may be configured to detect data in column 315 a as including an equivalent data classification as column 315 b. In particular, columns 315 a and 315 b include “zip code” data. Hence, data in column 315 a may be identified as a first reference indicator and data in column 315 b may be identified as a second reference indicator. Thus, a unit of data (e.g., data unit 311) may reference 340 to another unit of data (e.g., data unit 323).
In some examples, data in column 315 a may be used to establish a primary key, and data in column 315 b may be used to establish a foreign key (or conversely). Therefore, a user may be presented in a user interface an indication that columns 315 a and 315 b may include zip code data, whereby a user may confirm the columns include equivalent data so that associations, such as association 340, may be used to combine (e.g., join) data of datasets 310 and 370 at columns including reference indicator data. Association 340 may identify that a unit of data (“zip code 83631”) 311 in row 313 a is linked to another unit of data (“zip code 83631”) 323 in row 313 b.
According to some examples, one or more of dataset ingestion controller 320, format converter 337, and layer data generator 338 may be configured to transmute association 340 into a graph data arrangement, whereby a transmuted association 362 may be formed within a graph data arrangement. In the example shown, transmuted association 362 may link a node associated with unit of data 311 and a node associated with a unit of data 323. In the example shown, units of data 311 and 323 may be associated with a layer (“X”) 330, whereby layer data generator 338 identifies links for row node 321 a and column node 302 a for unit of data 311, and identifies links for row node 350 a and column node 357 a for unit of data 323. Layer 330 may also include data representing a link to transmuted association 362.
A graph portion 380 is shown to include one or more links based on a transmuted association derived from a relationship between, for example, primary and secondary keys may be implemented as a portion of a graph. In graph portion 380, a node 390 a associated with a unit of data (e.g., zip code 83631) links to a node 392 a, which is associated with a county name (e.g., county name “Adams County”). Nodes 390 a and 392 a may be linked via link 391 a, which represents that zip code 83631 “is a part of” Adams County. Further, node 392 a may link to a node 392 b, which is associated with a state name (e.g., state name “Idaho”). Nodes 392 a and 392 b may be linked via link 391 c, which represents that county name Adams County “is a part of” state name Idaho. Links 391 a and 391 c may be form one or more portions of a transmuted association 362 in which rows 313 a and 313 b may be combined to associate unit of data (“83631”) of row 313 a to unit of data (“Adams”) 324 and unit of data (“Idaho”) 325, both of which reside in row 313 b.
According to at least on example, an auxiliary query generator described in FIGS. 6 to 7 may be configured to generate an additional link in a graph data arrangement, whereby an additional link may be formed as a “created triple.” Specifically, an auxiliary query may be applied to nodes 390 a, 392 a, and 392 b and links 391 a and 391 c to identify an implicit relationship (i.e., zip code 83631 “is part of” the state of Idaho), thereby forming a triple including node 390 a, link 391 b, and node 392 b, which may be referred to as an explicit relationship.
In view of the foregoing, a relational query, or a variant thereof (e.g., an SQL-equivalent query), may be applied to data in a combination of datasets 310 and 370, which may be presented via a user interface (not shown) as a table of rows and columns. A dataset query engine, as shown in FIG. 4, may be implemented to receive the relational query and apply a query 382 to graph portion 380, thereby foregoing computing comparing data values to detect equalities of indexes used on tabular data arrangements. One or more elements depicted in diagram 300 of FIG. 3 may include structures and/or functions as similarly-named or similarly-numbered elements depicted in other drawings, or as otherwise described herein, in accordance with one or more examples.
FIG. 4 is a diagram depicting a dataset query engine configured to implement a query via a transmuted association against a graph data arrangement, according to some examples. Diagram 400 includes a dataset query engine 439, which may be disposed in a collaborative dataset consolidation system (not shown). Dataset query engine 439 may be configured to receive a query 402 to apply against a combined dataset 420, which is depicted as a combination of tabular data arrangements. In some examples, query 402 may be implemented as either a relational-based query (e.g., in an SQL-equivalent query language) or a graph-based query (e.g., in a SPARQL-equivalent query language). One or more elements depicted in diagram 400 of FIG. 4 may include structures and/or functions as similarly-named or similarly-numbered elements depicted in other drawings, or as otherwise described herein, in accordance with one or more examples.
Combined dataset 420 may be presented in a user interface as a table based on tabular data arrangements in which data in an ingested dataset 105 a and another dataset 115 a may be combined. Dataset 105 a is shown to include a unit of data 116 a associated with a subset of reference indicators (e.g., data of column 113 a), whereas dataset 115 a may include a unit of data 116 b associated with another subset of reference indicators (e.g., data of column 123 a). As shown, value 116 a in a row of dataset 105 a may, as a referential indicator, be used to reference via 106 to value 116 b. Also shown is an association 107 between referential indicator 123 a and referential indicator 123 b, and, thus, between value 116 a and another value 116 b.
Data representing association 107 between value 116 a and another value 116 may be transmuted to form a transmuted association depicted as a transmuted link 111 to combine dataset 105 a disposed in a graph, such as ingested dataset 440 a, with another dataset 115 a disposed in another graph, such as other datasets 440 b. Transmuted link 111 thus facilitates querying a merged dataset 440 as a graph data arrangement via transmuted link 111, which couples node 199 a to 199 b. In one example, node 199 a may be associated with value 116 a and node 199 b may be associated with other value 116 b, whereby transmuted link 111 may include data characterizing a relationship or property associating values 116 a and 116 b. In this example, transmuted link 111 includes data characterizing values associated with nodes 199 a and 199 b as being equivalent (e.g., equal or sufficiently similar to each other). According to various examples, data may vary for transmuted link 111 and nodes 199 a and 199 b to form any number of triples. In view of the foregoing, a query 402 configured to query a relational data model may be received into dataset query engine 439, which, in turn, transmits a query 406 for application against graph data arrangements as a merged dataset 440. Query 406 omits or otherwise need not invoke application of, or computations for, an index-based association to query linked data of a graph. A query being applied to node 199 a may be extended to include node 199 c.
FIG. 5 is a flow diagram depicting an example of transmuting relationships among data during data ingestion to enhance querying and retrieving results thereof, according to some embodiments. At 502 of flow 550, data representing a dataset may be received into a dataset ingestion controller. At 504, data representing the dataset may be identified as being disposed in, or may be arranged within, a first data arrangement having a first format, such as a tabular format. At 506, a first referential indicator for a first set of the dataset in the first data arrangement may be identified. The first referential indicator may refer to one or more data values disposed, for example, in a column, or may refer to one or more columns of data. At 508, an association may be determined, whereby the association may exist between a value representative of the first referential indicator and an equivalent value representative of a second referential indicator associated with a second set of a different dataset. At 510, an ingested dataset may be formatted into a second data arrangement having a second format, such as a graph format. Note that 510 may be disposed anywhere in flow 500, such as subsequent to 506. At 512, an association may be transmuted to form a transmuted association, as a link between a value and an equivalent value. At 514, a transmuted association may be integrated into at least a portion of the first data arrangement. Further, transmuted association may be integrated into a combined dataset (e.g., a merged dataset), and may persist for subsequent dataset links and data enhancements.
FIG. 6 is a diagram depicting examples of one or more auxiliary query generators for enriching ingested datasets, according to some examples. Diagram 600 includes a dataset ingestion controller 620, which includes a dataset analyzer 630.
Further, dataset ingestion controller 620 includes a data enhancement manager 636 including one or more auxiliary query generators, such as auxiliary query generators 638 a, 638 b, 638 c, and 638 n. Data enhancement manager 636 may be communicatively coupled to a data repository 640 storing any number of datasets in a pool of datasets, including dataset 642, within a graph data arrangement. Also, data enhancement manager 636 may be coupled to a computing device (not shown) to present a data enhancement interface 603, which may accept user input to initiate generation of auxiliary queries as “link-formative” queries. A link-formative query may be configured to generate results for integrating or merging back into an ingested dataset 605 a, thereby enhancing ingested data set 605 a. A link-formative query may be a query that, for example, invokes or otherwise is configured to form links, at least in some implementations. Thus, results of a link-formative query may be a graph including a created or new dataset of linked data. One or more elements depicted in diagram 600 of FIG. 6 may include structures and/or functions as similarly-named or similarly-numbered elements depicted in other drawings, or as otherwise described herein, in accordance with one or more examples.
In the example shown, data representing a dataset 605 a may be received into a dataset ingestion controller (not shown). Dataset ingestion controller 620 and/or its constituent components may identify dataset 605 a is, or may otherwise arranged, in a first data arrangement (e.g., a tabular data arrangement) in a first format (e.g., a table). Dataset ingestion controller 620 may transform a tabular data arrangement in which dataset 605 a is disposed into dataset 644, which is a second data arrangement (e.g., a graph data arrangement) in a second format (e.g., a graph) in which data in dataset 605 a is disposed. Dataset analyzer 630 may be configured to analyze data representing dataset 605 a to detect subsets of data values for which to perform a query (e.g., as a link-formative query). An example of a subset of data values includes data values in, for example, a column 613 a for analyzing and detecting whether to perform a link-formative query.
An auxiliary query generator, such as one of auxiliary query generators 638 a, 638 b, 638 c, and 638 n, may be configured to identify a subset of data, such as one or more data values in column 613 a that may be compared against a pool of datasets to identify equivalent data values or dataset attributes with which to form links (e.g., as at least a portion of a link-formative query) among data in the subset of data in column 613 a and the pool of datasets. A pool of datasets 642 may include any number of linked data-based graph data arrangements, at least some of which may be stored in repository 640. In some examples, an auxiliary query generator may identify equivalent data values (or dataset attributes) in dataset 605 a and pool of datasets 642 upon which to perform a link-formative query. A link-formative query may be configured to perform an auxiliary or subsidiary query on ingested datasets 605 a for identifying linkable datasets in pool of datasets 642, generating another subset of data to form a created subset of linkable data (e.g., data that can form linked data), and integrating or merging the created subset of data back into, for example, dataset 605 a as an enrichment to dataset 644, according to some examples. In some cases, a created subset includes new data values that are absent in ingested dataset 605 a and may be introduced into a new or added column of a tabular data arrangement for ingested dataset 605 a. Further, data values of column 613 a, such as “zip code data values,” may be linked to a created subset of data, as linked data, which can be presented graphically as “linked data” in a user interface, examples of which are depicted in FIGS. 8A and 8B.
A link-formative query may be initiated, for example, by an auxiliary query generator, to search for specific subsets of data in pool of data 642 that may be associated with specific subsets of data in columns of 605 a. A search may be based on a specific subset or column of data includes data classified to include similar data types, data classifications (e.g., zip code data), etc. According to some examples, each of auxiliary query generators 638 a, 638 b, 638 c, and 638 n may use specific subsets of data to search (e.g., query) a pool of datasets 642. Consider the following example in which auxiliary query generator 638 a may be configured to identify or use “zip code data” disposed in column 613 a to search for other data associated with zip code data in pool of datasets 642 from which to from a new, created dataset. Auxiliary query generator 638 a may be configured to identify or use “infectious disease data” (e.g., flu outbreak data, such as data values representing different flu types, such as A, A2, B, C, H5, H5N1, etc.) disposed in column 613 a to search for other data associated with health-related data in pool of datasets 642 from which to form another new, created dataset. Other auxiliary query generators may implement any subset of data values in columns 613 a to perform link-formative queries “in-situ” or “in-line” (e.g., in real time or near real time) to enhance expansion of data in atomized dataset generation during the dataset ingestion and/or graph formation processes, which may be prior to subsequent data operations, such as queries. According to some examples, a link-formative query may be based on a transmuted association. Other auxiliary query generators may implement any subset of data values in columns 613 a to perform link-formative queries “in-situ” or “in-line” (e.g., in real time or near real time) to enhance expansion of data in atomized dataset generation during the dataset ingestion and/or graph formation processes. According to some examples, a link-formative query may be based on a transmuted association.
In alternative examples, at least one of auxiliary query generators 638 a, 638 b, 638 c, and 638 n may identify or use a subset of data values disposed in column 613 a to compute or modify the subset of data values to form a created subset of data. For example, auxiliary query generator 638 c may be configured to initiate a query to identify whether to use “data-related” data (e.g., day, month, year, time, etc.) disposed in column 613 a for modification to, for example, modify an annotation or form of date-related information (e.g., removing day and month to present year only dates). Thus, modified date-related data may be disposed in a new, created column that may be implemented, such as column 613 a, within dataset 605 a. As another example, auxiliary query generator 638 n may be configured to identify or use numeric data values disposed in column 613 a for use in machine learning computations. As such, auxiliary query generator 638 n may initiate a query to identify whether to initiate a computation or modification to “normalize” the numeric data values into, for example, a range from zero (“0”) to one (“1”). In some cases, a response to the query may originate from dataset enrichment interface 603. Other auxiliary query generators may implement any other computations or modifications to any subset of data values in columns 613 a to perform link-formative queries or modifications “in-situ” or “in-line.”
In some examples, operation of data enhancement manager 636 and/or its constituent components, such as auxiliary query generators 638 a, 638 b, 638 c, and 638 n, may be guided or supplemented by performance of executable instructions based on commands received responsive to inputs via dataset enrichment interface 603. In one implementation, dataset analyzer 630 may detect that column 613 a includes “zip code” data, and, in response, dataset enrichment interface 603 may present via interface portion 606 selections with which to generate commands based on whether column 613 a includes zip code data (e.g., via selection of input 671), or do not include zip code data (e.g., via selection of input 673). Further, dataset enrichment interface 603 may be configured to present an interface portion 607 to provide selections from which a user input data may be generated to perform one or more auxiliary queries. So, if selection 671 is activated, interface portion 607 may provide a selection 608 a to include population data (e.g., related to zip code data), a selection 608 b to include congressional data (e.g., related to zip code data), a selection 608 c to include crime data (e.g., related to zip code data), and a selection 608 d to include data directed to a modify a date format (e.g., year only date information). Each of selections 608 a, 608 b, 608 c, and 608 d may initiate a link-formative query (e.g., an auxiliary query), the results of which may be formatted for inclusion as columnar data, such as in column 613 a of dataset 605 a. The results of each of the link-formative queries may be integrated back into in either dataset 605 a in a tabular data arrangement or in dataset 644 graph data arrangement, which includes data of dataset 605 a disposed in a graph. Thus, either the link-formative queries or results therefrom, or both, may be stored in repository 640 for subsequent use.
Operation of data enhancement manager 636 and/or its constituent components, such as auxiliary query generators 638 a, 638 b, 638 c, and 638 n, may be automatic (e.g., without user input) in some examples. Further, merged datasets and results of link-formative queries may persist so that an integrated dataset, such as merge datasets 644 a, may be modified or supplemented (e.g., via data ingestion) subsequent to initial formation. Thus, operation of auxiliary query generators 638 a, 638 b, 638 c, and 638 n may be automatically activated repeatedly until, for example, a user removes or deletes a subset of data from merged dataset 644 a. As shown, merged dataset (e.g., an enhanced dataset) includes a graph 640 a of ingested data associated with datasets 605 a and a graph 640 b of a pool of datasets. A transmuted link 611 may link graph 640 a to graph 640 b, whereby graphs 640 a and 640 b may include atomized datasets.
Moreover, results of an auxiliary query (e.g., a link-formative query) may be implemented as link 391 b, responsive to a link-formative query that identifies “state name” data 392 b based on a column of “zip code” data 390 a in accordance with FIG. 3. Therefore, a link-formative query may be configured to form an explicit or direct link 391 b based on implicit or indirect links 391 a and 391 c via node 392 a, which may be associated with “county name” data. In at least one example, formation of link 391 b (e.g., based on a link-formative query) provides for a created dataset that includes at least one additional triple.
FIG. 7 is a flow diagram depicting an example of implementing link-formative queries to enhance datasets, according to some embodiments. At 702, data representing a dataset may be received, for example, into dataset ingestion controller. At 704, a first data arrangement in a first format may be identified in which the data representing the dataset is arranged. At 706, a first data arrangement in the first format may be transformed into a second data arrangement in a second format, which may be graph-related. In some examples, a first data arrangement may be transformed into an atomized dataset that includes triples.
At 708, data representing the dataset may be analyzed to detect subsets of data values for which to query against in a link formative query. For example, an association may be determined, whereby the association may be between a value representative of a first referential indicator and an equivalent value representative of a second referential indicator, which may be associated with a different dataset. The different dataset may be a table or graph, or may be externally disposed. In some examples, one of the first referential indicator and the second referential indicator may be a primary key. The other of the first referential indicator and the second referential indicator may be a foreign key. In some examples, an association between referential indicators may be transmuted to form a transmuted association between the value and the equivalent value. In some examples, a transmuted association includes an association between referential indicators that is converted, formatted, or mapped into a graph data arrangement, according to at least one example to facilitate queries that, for example, need not implement indices to compute equivalent data. In some examples, a transmuted association facilitates link-formative queries to create datasets with, for example, explicit and direct links.
At 710, one or more link-formative queries may be applied to dataset in a second data arrangement. As such, link-formative queries may be applied to graph data arrangements, which may include a pattern of triples. At 712, results of the one or more link-formative queries may be identified. In some examples, results may be determined as a subset of resultant triples associated with a pattern of triples. A result of at least one link-formative query may be referred to as an auxiliary graph data arrangement, according to some examples. As a graph, auxiliary graph data arrangement may be integrated to form a merged graph. In at least one example, a link-formative query may apply a graph-based statement or command to identify patterns of linked data, such as triples, matching data defining a desired result, whereby the desired result “constructs” a created graph-based dataset. A graph-based statement or command may include a CONSTRUCT clause based on, for example, a graph querying language (e.g., SPARQL, or the like), the CONSTRUCT clause being configured to form created graphs matching a query pattern, which may be set forth, for example, in a WHERE clause. Other graph-based statements or commands that create graphs (e.g., new triples) may be used, and are not limited to SPARQL-based statement or commands. At 712, an enhanced dataset may be formed, whereby the enhanced dataset may include results of one or more link-formative queries in the dataset.
FIGS. 8A to 8D are diagrams depicting computerized tools of a user interface to cause formation of transmuted associations to facilitate link-formative queries, according to some examples. Diagram 800 of FIG. 8A depicts as file (e.g., .CSV) including data representing a dataset, which may be identified or arranged into a tabular data arrangement 804. According to some examples, interface portion 806 includes selections 871 and 873 to receive user input as to whether a column, such as column 810, includes a zip code data. In this case, selection 871 is selected and, in response, executable instructions are activated to link numeric zip codes “78703” and “78731” of column 810 to other data in, for example, a graph data arrangement. In at least one example, linking of zip codes “78703” and “78731” to “other datasets” may be facilitated by way of implementing a transmuted association. As shown, graphical identifiers 812 encircling each zip code number indicates that corresponding zip codes “78703” and “78731” may link to form an enriched dataset (e.g., as a merged dataset). For example, column 810 may link to data 821 representing, for example, “population density” per zip code, as depicted in a choropleth of zip codes in interface portion 820. As another example, a selection device, such as cursor 814, may cause presentation of interface portion 822, which includes enhanced data not within file 802, and made available by implementing a transmuted association (not shown) in a graph. In the example shown, zip code number 78731 and column 810 may link to enhanced datasets 824 and 826. Dataset 824 includes hypertext links to congressional data, climate data, crime data, demographic data, and hotel data, all of which are associated with zip code 78731.
FIG. 8B is a diagram 830 depicting interface processes of computerized tools to form enhanced dataset, according to some examples. While dataset 832 may be presented in diagram 830 in a tabular data arrangement, units of data therein may be linked to an underlying merged graph as graph data arrangement. To enhance dataset 832, a command may be activated to present interface portion 834, which presents options to enhance dataset 832. Selection 836, if selected, may cause presentations of user inputs in a user interface to enhance dataset 832 a by adding either city data in column 837 or state data in column 839. Responsive to one or more selections 831 a and 831 b, dataset 832 a may be integrated with data in columns 837 and/or 839, thereby forming a merged dataset, at least in the example shown. In some examples, a data arrangement 832 may be transformed into an atomized dataset including subsets of linked data points (e.g., in a graph). The data representing dataset 832 may be analyzed to detect zip code data values 78703 and 7873a, with which to query against in a link-formative query. A link-formative query applied to a pool of datasets, based on the zip code data values, may identify linkable data points in a pool of datasets that include, for example, “city name” data and “state name” data.
FIG. 8C is a diagram 850 depicting another example of interface processes of computerized tools to form an enhanced dataset, at least in some examples. To enhance dataset 832, a command may be activated to present interface portion 854, which presents options to aggregate data via computations or data modification applied to dataset 852. Selection 856, if selected, may cause presentations to enhance dataset 852 a by adding data in a column representing “mean age” data 860 linked to zip codes 78703 and 78731, responsive to selections related to interface portions 857, 858, and 859. Thus, mean age data 860 may be derived via calculations applied to linked datasets that includes age-related data related to zip codes. Selection 864, if activated, may cause formation of an enhanced dataset based on 852 that includes mean age data 860.
In some examples, data representing 852 dataset may be analyzed to detect zip code data values with which to query against a pool of datasets based on a link-formative query. A link-formative query may be applied to a pool of datasets (e.g., other atomized datasets) based on zip code data values to detect other data points associated with zip codes. A data value to form a computed data value of “mean age” data 860 associated with another data point may be calculated. The computed result includes a column of “mean age” data 860 that may include additional linkable data points in a pool of atomized datasets to further enhance formation of a merged dataset. The linkable data points may be linked to dataset 852 a responsive to activation of input 864.
FIG. 8D is a diagram 870 depicting yet another example of interface processes of computerized tools to form an enhanced dataset, at least in some examples. To enhance dataset 872, a command may be activated to present interface portion 874, which presents options to aggregate data via computations or data modification applied to dataset 872. Selection 876, if selected, may cause presentations of user input selections to enhance dataset 872, which may include columns 873 and 880, by adding data in a column representing “ACS mean age” data 882 linked to zip codes 78703 and 78731, responsive to selections related to interface portions 877, 878, and 879. “ACS mean age” data may refer to American Community Survey (“ACS”) data provided in U.S. Census data, as one example. Thus, ACS mean age data 882 may be derived via modifications applied to linked datasets that includes age-related data related to zip codes. Selection 884, if activated, may cause formation of an enhanced dataset based on 872 that includes ACS mean age data 884.
FIG. 9 illustrates examples of various computing platforms configured to provide various functionalities to any of one or more components of a collaborative dataset consolidation system, according to various embodiments. In some examples, computing platform 900 may be used to implement computer programs, applications, methods, processes, algorithms, or other software, as well as any hardware implementation thereof, to perform the above-described techniques.
In some cases, computing platform 900 or any portion (e.g., any structural or functional portion) can be disposed in any device, such as a computing device 990 a, mobile computing device 990 b, and/or a processing circuit in association with initiating the formation of collaborative datasets, as well as analyzing forming enhance datasets using transmuted associations, via user interfaces and user interface elements, according to various examples described herein.
Computing platform 900 includes a bus 902 or other communication mechanism for communicating information, which interconnects subsystems and devices, such as processor 904, system memory 906 (e.g., RAM, etc.), storage device 908 (e.g., ROM, etc.), an in-memory cache (which may be implemented in RAM 906 or other portions of computing platform 900), a communication interface 913 (e.g., an Ethernet or wireless controller, a Bluetooth controller, NFC logic, etc.) to facilitate communications via a port on communication link 921 to communicate, for example, with a computing device, including mobile computing and/or communication devices with processors, including database devices (e.g., storage devices configured to store atomized datasets, including, but not limited to triplestores, etc.). Processor 904 can be implemented as one or more graphics processing units (“GPUs”), as one or more central processing units (“CPUs”), such as those manufactured by Intel® Corporation, or as one or more virtual processors, as well as any combination of CPUs and virtual processors. Computing platform 900 exchanges data representing inputs and outputs via input-and-output devices 901, including, but not limited to, keyboards, mice, audio inputs (e.g., speech-to-text driven devices), user interfaces, displays, monitors, cursors, touch-sensitive displays, LCD or LED displays, and other I/O-related devices.
Note that in some examples, input-and-output devices 901 may be implemented as, or otherwise substituted with, a user interface in a computing device associated with a user account identifier in accordance with the various examples described herein.
According to some examples, computing platform 900 performs specific operations by processor 904 executing one or more sequences of one or more instructions stored in system memory 906, and computing platform 900 can be implemented in a client-server arrangement, peer-to-peer arrangement, or as any mobile computing device, including smart phones and the like. Such instructions or data may be read into system memory 906 from another computer readable medium, such as storage device 908. In some examples, hard-wired circuitry may be used in place of or in combination with software instructions for implementation. Instructions may be embedded in software or firmware. The term “computer readable medium” refers to any tangible medium that participates in providing instructions to processor 904 for execution. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks and the like. Volatile media includes dynamic memory, such as system memory 906.
Known forms of computer readable media includes, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can access data. Instructions may further be transmitted or received using a transmission medium. The term “transmission medium” may include any tangible or intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine, and includes digital or analog communications signals or other intangible medium to facilitate communication of such instructions. Transmission media includes coaxial cables, copper wire, and fiber optics, including wires that comprise bus 902 for transmitting a computer data signal.
In some examples, execution of the sequences of instructions may be performed by computing platform 900. According to some examples, computing platform 900 can be coupled by communication link 921 (e.g., a wired network, such as LAN, PSTN, or any wireless network, including WiFi of various standards and protocols, Bluetooth®, NFC, Zig-Bee, etc.) to any other processor to perform the sequence of instructions in coordination with (or asynchronous to) one another. Computing platform 900 may transmit and receive messages, data, and instructions, including program code (e.g., application code) through communication link 921 and communication interface 913. Received program code may be executed by processor 904 as it is received, and/or stored in memory 906 or other non-volatile storage for later execution.
In the example shown, system memory 906 can include various modules that include executable instructions to implement functionalities described herein. System memory 906 may include an operating system (“O/S”) 932, as well as an application 936 and/or logic module(s) 959. In the example shown in FIG. 9, system memory 906 may include any number of modules 959, any of which, or one or more portions of which, can be configured to facilitate any one or more components of a computing system (e.g., a client computing system, a server computing system, etc.) by implementing one or more functions described herein.
The structures and/or functions of any of the above-described features can be implemented in software, hardware, firmware, circuitry, or a combination thereof. Note that the structures and constituent elements above, as well as their functionality, may be aggregated with one or more other structures or elements. Alternatively, the elements and their functionality may be subdivided into constituent sub-elements, if any. As software, the above-described techniques may be implemented using various types of programming or formatting languages, frameworks, syntax, applications, protocols, objects, or techniques. As hardware and/or firmware, the above-described techniques may be implemented using various types of programming or integrated circuit design languages, including hardware description languages, such as any register transfer language (“RTL”) configured to design field-programmable gate arrays (“FPGAs”), application-specific integrated circuits (“ASICs”), or any other type of integrated circuit. According to some embodiments, the term “module” can refer, for example, to an algorithm or a portion thereof, and/or logic implemented in either hardware circuitry or software, or a combination thereof. These can be varied and are not limited to the examples or descriptions provided.
In some embodiments, modules 959 of FIG. 9, or one or more of their components, or any process or device described herein, can be in communication (e.g., wired or wirelessly) with a mobile device, such as a mobile phone or computing device, or can be disposed therein.
In some cases, a mobile device, or any networked computing device (not shown) in communication with one or more modules 959 or one or more of its/their components (or any process or device described herein), can provide at least some of the structures and/or functions of any of the features described herein. As depicted in the above-described figures, the structures and/or functions of any of the above-described features can be implemented in software, hardware, firmware, circuitry, or any combination thereof. Note that the structures and constituent elements above, as well as their functionality, may be aggregated or combined with one or more other structures or elements. Alternatively, the elements and their functionality may be subdivided into constituent sub-elements, if any. As software, at least some of the above-described techniques may be implemented using various types of programming or formatting languages, frameworks, syntax, applications, protocols, objects, or techniques. For example, at least one of the elements depicted in any of the figures can represent one or more algorithms. Or, at least one of the elements can represent a portion of logic including a portion of hardware configured to provide constituent structures and/or functionalities.
For example, modules 959 or one or more of its/their components, or any process or device described herein, can be implemented in one or more computing devices (i.e., any mobile computing device, such as a wearable device, such as a hat or headband, or mobile phone, whether worn or carried) that include one or more processors configured to execute one or more algorithms in memory. Thus, at least some of the elements in the above-described figures can represent one or more algorithms. Or, at least one of the elements can represent a portion of logic including a portion of hardware configured to provide constituent structures and/or functionalities. These can be varied and are not limited to the examples or descriptions provided.
As hardware and/or firmware, the above-described structures and techniques can be implemented using various types of programming or integrated circuit design languages, including hardware description languages, such as any register transfer language (“RTL”) configured to design field-programmable gate arrays (“FPGAs”), application-specific integrated circuits (“ASICs”), multi-chip modules, or any other type of integrated circuit.
For example, modules 959 or one or more of its/their components, or any process or device described herein, can be implemented in one or more computing devices that include one or more circuits. Thus, at least one of the elements in the above-described figures can represent one or more components of hardware. Or, at least one of the elements can represent a portion of logic including a portion of a circuit configured to provide constituent structures and/or functionalities.
According to some embodiments, the term “circuit” can refer, for example, to any system including a number of components through which current flows to perform one or more functions, the components including discrete and complex components. Examples of discrete components include transistors, resistors, capacitors, inductors, diodes, and the like, and examples of complex components include memory, processors, analog circuits, digital circuits, and the like, including field-programmable gate arrays (“FPGAs”), application-specific integrated circuits (“ASICs”). Therefore, a circuit can include a system of electronic components and logic components (e.g., logic configured to execute instructions, such that a group of executable instructions of an algorithm, for example, and, thus, is a component of a circuit). According to some embodiments, the term “module” can refer, for example, to an algorithm or a portion thereof, and/or logic implemented in either hardware circuitry or software, or a combination thereof (i.e., a module can be implemented as a circuit). In some embodiments, algorithms and/or the memory in which the algorithms are stored are “components” of a circuit. Thus, the term “circuit” can also refer, for example, to a system of components, including algorithms. These can be varied and are not limited to the examples or descriptions provided. Further, none of the above-described implementations are abstract, but rather contribute significantly to improvements to functionalities and the art of computing devices.
Although the foregoing examples have been described in some detail for purposes of clarity of understanding, the above-described inventive techniques are not limited to the details provided. There are many alternative ways of implementing the above-described invention techniques. The disclosed examples are illustrative and not restrictive.

Claims

1. A method comprising:

receiving data representing a dataset into dataset ingestion controller;

identifying a first data arrangement in a first format in which the data representing the dataset is arranged;

transforming the first data arrangement in the first format into a second data arrangement in a second format;

analyzing the data representing the dataset to detect subsets of data values for which to query against in a link-formative query;

applying one or more link-formative queries to the dataset in the second data arrangement;

identifying results of the one or more link-formative queries; and

forming an enhanced dataset to include the results of the one or more link-formative queries in the dataset.

2. The method of claim 1 wherein transforming the first data arrangement into the second data arrangement comprises:

transforming the first data arrangement into an atomized dataset.

3. The method of claim 2 wherein the atomized dataset includes subsets of linked data points.

4. The method of claim 2 wherein identifying results of the one or more link-formative queries further comprises:

forming an association between at least one data point in the atomized dataset to another data point in another atomized dataset stored in a repository.

5. The method of claim 4 wherein forming the association comprises:

forming a direct link between the at least one data point in the atomized dataset to the another data point to the another atomized dataset.

6. The method of claim 1 wherein forming the association comprises:

storing the results of the one or more link-formative queries;

detecting at least a portion of the dataset introduced into the dataset ingestion controller; and

implementing the enhanced dataset.

7. The method of claim 6 further comprising:

receiving a query; and

implementing the enhanced dataset automatically based on detecting the at least portion of the dataset.

8. The method of claim 1 wherein analyzing the data representing the dataset to detect subsets of the data values comprises:

determining an association with a value representative of a first referential indicator to an equivalent value representative of a second referential indicator associated with a different dataset;

transmuting the association to form a transmuted association between the value and the equivalent value; and

identifying a linkable data point in an atomized dataset based on the transmuted association referencing another data point in another atomized dataset.

9. The method of claim 8 further comprising:

wherein one of the first referential indicator and the second referential indicator is a primary key, and the other of the first referential indicator and the second referential indicator is a foreign key.

10. The method of claim 8 further comprises:

generating data to present one or more user inputs in a user interface to initiate execution of instructions to associate the key referential indicator to the second referential indicator; and

receiving a response to at least one of the one or more user inputs to implement the linkable data point in the atomized dataset to the another dataset.

11. The method of claim 1 wherein transforming the first data arrangement in the first format into the second data arrangement having the second format comprises:

formatting the dataset in a tabular data arrangement as the first data arrangement having a table format as the first format into a graph data arrangement as the second data arrangement having a graph format as the second format,

wherein the enhanced dataset is a merged graph.

12. The method of claim 1 further comprising:

storing the results of the one or more link-formative queries in a layer file, the results including at least a subset of referential indicators.

13. The method of claim 1 wherein applying the one or more link-formative queries to the second data arrangement comprises:

analyzing the data representing the dataset automatically; and

applying the one or more link-formative queries automatically.

14. The method of claim 1 wherein transforming the first data arrangement into the second data arrangement comprises:

transforming the first data arrangement into an atomized dataset including triples,

wherein applying the one or more link-formative queries to the dataset in the second data arrangement comprises:

applying at least one link-formative query to a pattern of triples.

15. The method of claim 14 wherein identifying the results of the one or more link-formative queries comprises:

determining a subset of resultant triples associated with the pattern of triples,

wherein a result of the at least one link-formative query is an auxiliary graph data arrangement.

16. The method of claim 1 further comprising:

transforming the first data arrangement into an atomized dataset including subsets of data points;

analyzing the data representing the dataset to detect zip code data values with which to query against in the link-formative query;

applying the link-formative query to the dataset based on the zip code data values; and

identifying a linkable data point in the atomized dataset that referencing another data point in another atomized dataset based on the zip code data values.

17. The method of claim 1 further comprising:

transforming the first data arrangement into an atomized dataset including subsets of linked data points;

analyzing the data representing the dataset to detect data values with which to query against in the link-formative query;

applying the link-formative query to the dataset based on the data values;

identifying another data point in another atomized dataset associated with the data values;

computing a data value to form a computed data value associated with the another data point; and

identifying a linkable data point in the atomized dataset that references the computed data value of the another data point.

18. An apparatus comprising:

a memory including executable instructions; and

a processor, responsive to executing the instructions, is configured to:

receive data representing a dataset into dataset ingestion controller;

identify a first data arrangement in a first format in which the data representing the dataset is arranged;

transform the first data arrangement in the first format into a second data arrangement in a second format;

analyze the data representing the dataset to detect subsets of data values for which to query against in a link-formative query;

apply one or more link-formative queries to the dataset in the second data arrangement;

identify results of the one or more link-formative queries; and

form an enhanced dataset to include the results of the one or more link-formative queries in the dataset.

19. The apparatus of claim 16 wherein a subset of the instructions to cause the processor to transform the first data arrangement in the first format into the second data arrangement in the second format further causes the processor to:

transform the first data arrangement into an atomized dataset,

wherein the atomized dataset includes subsets of linked data points.

20. The apparatus of claim 16 wherein the processor is further configured to:

store the results of the one or more link-formative queries;

detect at least a portion of the dataset introduced into the dataset ingestion controller; and

receive a query; and

implement the enhanced dataset automatically based on detecting the at least portion of the dataset.