[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

MX2015009825A - Division de flujo de datos para acceso a datos de baja latencia. - Google Patents

Division de flujo de datos para acceso a datos de baja latencia.

Info

Publication number
MX2015009825A
MX2015009825A MX2015009825A MX2015009825A MX2015009825A MX 2015009825 A MX2015009825 A MX 2015009825A MX 2015009825 A MX2015009825 A MX 2015009825A MX 2015009825 A MX2015009825 A MX 2015009825A MX 2015009825 A MX2015009825 A MX 2015009825A
Authority
MX
Mexico
Prior art keywords
log data
data streams
data
split
low
Prior art date
Application number
MX2015009825A
Other languages
English (en)
Other versions
MX356571B (es
Inventor
Samuel Rash
Dhrubajyoti Borthakur
Zheng Shao
Eric Hwang
Original Assignee
Facebook Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Facebook Inc filed Critical Facebook Inc
Publication of MX2015009825A publication Critical patent/MX2015009825A/es
Publication of MX356571B publication Critical patent/MX356571B/es

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)
  • Storage Device Security (AREA)

Abstract

Se describen aquí técnicas para facilitar y acelerar el procesamiento de datos de registro al dividir flujos de datos. Los grupos de extremo frontal generan grandes cantidades de datos de registro en tiempo real y transfieren los datos de registro a un grupo de agregación. El grupo de agregación está diseñado para agregar flujos de datos de registro de entrada de diferentes servidores y grupos de extremo frontal. El grupo de agregación además divide a los datos de registro en una pluralidad de flujos de datos de manera que los flujos de datos son enviados a una aplicación de recepción en paralelo. En una modalidad, los datos de registro son aleatoriamente divididos para asegurar que los datos de registro sean uniformemente distribuidos en los flujos de datos divididos. En otra modalidad, la aplicación que recibe los flujos de datos divididos determina cómo se dividen los datos de registro.
MX2015009825A 2013-01-31 2014-01-17 Division de flujo de datos para acceso a datos de baja latencia. MX356571B (es)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13/756,340 US10223431B2 (en) 2013-01-31 2013-01-31 Data stream splitting for low-latency data access
PCT/US2014/012120 WO2014120487A1 (en) 2013-01-31 2014-01-17 Data stream splitting for low-latency data access

Publications (2)

Publication Number Publication Date
MX2015009825A true MX2015009825A (es) 2016-04-04
MX356571B MX356571B (es) 2018-06-04

Family

ID=51224089

Family Applications (1)

Application Number Title Priority Date Filing Date
MX2015009825A MX356571B (es) 2013-01-31 2014-01-17 Division de flujo de datos para acceso a datos de baja latencia.

Country Status (10)

Country Link
US (1) US10223431B2 (es)
JP (1) JP6258975B2 (es)
KR (1) KR101885688B1 (es)
CN (1) CN104969213B (es)
AU (1) AU2014212780B2 (es)
BR (1) BR112015017391A2 (es)
CA (1) CA2897338C (es)
IL (1) IL239811B (es)
MX (1) MX356571B (es)
WO (1) WO2014120487A1 (es)

Families Citing this family (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9609050B2 (en) * 2013-01-31 2017-03-28 Facebook, Inc. Multi-level data staging for low latency data access
US9600312B2 (en) 2014-09-30 2017-03-21 Amazon Technologies, Inc. Threading as a service
US9678773B1 (en) 2014-09-30 2017-06-13 Amazon Technologies, Inc. Low latency computational capacity provisioning
US9146764B1 (en) 2014-09-30 2015-09-29 Amazon Technologies, Inc. Processing event messages for user requests to execute program code
US9537788B2 (en) 2014-12-05 2017-01-03 Amazon Technologies, Inc. Automatic determination of resource sizing
US9588790B1 (en) 2015-02-04 2017-03-07 Amazon Technologies, Inc. Stateful virtual compute system
US9733967B2 (en) 2015-02-04 2017-08-15 Amazon Technologies, Inc. Security protocols for low latency execution of program code
US10706970B1 (en) 2015-04-06 2020-07-07 EMC IP Holding Company LLC Distributed data analytics
US10511659B1 (en) * 2015-04-06 2019-12-17 EMC IP Holding Company LLC Global benchmarking and statistical analysis at scale
US10541936B1 (en) * 2015-04-06 2020-01-21 EMC IP Holding Company LLC Method and system for distributed analysis
US10791063B1 (en) 2015-04-06 2020-09-29 EMC IP Holding Company LLC Scalable edge computing using devices with limited resources
US10541938B1 (en) 2015-04-06 2020-01-21 EMC IP Holding Company LLC Integration of distributed data processing platform with one or more distinct supporting platforms
US10015106B1 (en) 2015-04-06 2018-07-03 EMC IP Holding Company LLC Multi-cluster distributed data processing platform
US10515097B2 (en) * 2015-04-06 2019-12-24 EMC IP Holding Company LLC Analytics platform for scalable distributed computations
US10776404B2 (en) 2015-04-06 2020-09-15 EMC IP Holding Company LLC Scalable distributed computations utilizing multiple distinct computational frameworks
US10860622B1 (en) 2015-04-06 2020-12-08 EMC IP Holding Company LLC Scalable recursive computation for pattern identification across distributed data processing nodes
US10425350B1 (en) 2015-04-06 2019-09-24 EMC IP Holding Company LLC Distributed catalog service for data processing platform
CN106815274B (zh) * 2015-12-02 2022-02-18 中兴通讯股份有限公司 基于Hadoop的日志数据挖掘方法及系统
US9910713B2 (en) 2015-12-21 2018-03-06 Amazon Technologies, Inc. Code execution request routing
US10656861B1 (en) 2015-12-29 2020-05-19 EMC IP Holding Company LLC Scalable distributed in-memory computation
US11132213B1 (en) 2016-03-30 2021-09-28 Amazon Technologies, Inc. Dependency-based process of pre-existing data sets at an on demand code execution environment
CN105912609B (zh) * 2016-04-06 2019-04-02 中国农业银行股份有限公司 一种数据文件处理方法和装置
US10102040B2 (en) 2016-06-29 2018-10-16 Amazon Technologies, Inc Adjusting variable limit on concurrent code executions
WO2018182060A1 (ko) * 2017-03-28 2018-10-04 ㈜리얼타임테크 관계형 데이터베이스 기반의 텍스트 로그데이터 저장 및 검색 방법
US10853115B2 (en) 2018-06-25 2020-12-01 Amazon Technologies, Inc. Execution of auxiliary functions in an on-demand network code execution system
US11146569B1 (en) 2018-06-28 2021-10-12 Amazon Technologies, Inc. Escalation-resistant secure network services using request-scoped authentication information
US11099870B1 (en) 2018-07-25 2021-08-24 Amazon Technologies, Inc. Reducing execution times in an on-demand network code execution system using saved machine states
US11099917B2 (en) 2018-09-27 2021-08-24 Amazon Technologies, Inc. Efficient state maintenance for execution environments in an on-demand code execution system
US11243953B2 (en) * 2018-09-27 2022-02-08 Amazon Technologies, Inc. Mapreduce implementation in an on-demand network code execution system and stream data processing system
US11943093B1 (en) 2018-11-20 2024-03-26 Amazon Technologies, Inc. Network connection recovery after virtual machine transition in an on-demand network code execution system
US11010188B1 (en) 2019-02-05 2021-05-18 Amazon Technologies, Inc. Simulated data object storage using on-demand computation of data objects
US11861386B1 (en) 2019-03-22 2024-01-02 Amazon Technologies, Inc. Application gateways in an on-demand network code execution system
US11119809B1 (en) 2019-06-20 2021-09-14 Amazon Technologies, Inc. Virtualization-based transaction handling in an on-demand network code execution system
US11159528B2 (en) 2019-06-28 2021-10-26 Amazon Technologies, Inc. Authentication to network-services using hosted authentication information
US11115404B2 (en) 2019-06-28 2021-09-07 Amazon Technologies, Inc. Facilitating service connections in serverless code executions
US11190609B2 (en) 2019-06-28 2021-11-30 Amazon Technologies, Inc. Connection pooling for scalable network services
US11119826B2 (en) 2019-11-27 2021-09-14 Amazon Technologies, Inc. Serverless call distribution to implement spillover while avoiding cold starts
US11714682B1 (en) 2020-03-03 2023-08-01 Amazon Technologies, Inc. Reclaiming computing resources in an on-demand code execution system
US11188391B1 (en) 2020-03-11 2021-11-30 Amazon Technologies, Inc. Allocating resources to on-demand code executions under scarcity conditions
CN112416699A (zh) * 2020-06-05 2021-02-26 上海哔哩哔哩科技有限公司 指标数据收集方法及系统
CN111680009B (zh) * 2020-06-10 2023-10-03 苏州跃盟信息科技有限公司 日志处理方法、装置、存储介质和处理器
US11593270B1 (en) 2020-11-25 2023-02-28 Amazon Technologies, Inc. Fast distributed caching using erasure coded object parts
US11550713B1 (en) 2020-11-25 2023-01-10 Amazon Technologies, Inc. Garbage collection in distributed systems using life cycled storage roots
US11797522B2 (en) * 2021-01-29 2023-10-24 International Business Machines Corporation Database log writing based on log pipeline contention
US11388210B1 (en) 2021-06-30 2022-07-12 Amazon Technologies, Inc. Streaming analytics using a serverless compute system
US11968280B1 (en) 2021-11-24 2024-04-23 Amazon Technologies, Inc. Controlling ingestion of streaming data to serverless function executions
US12015603B2 (en) 2021-12-10 2024-06-18 Amazon Technologies, Inc. Multi-tenant mode for serverless code execution
US11995085B2 (en) 2022-02-25 2024-05-28 Visa International Service Association System, method, and computer program product for efficiently storing multi-threaded log data

Family Cites Families (82)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US1024646A (en) 1911-06-24 1912-04-30 Jeremiah Muckey Mold for concrete curbs for graves.
US6212515B1 (en) * 1998-11-03 2001-04-03 Platinum Technology, Inc. Method and apparatus for populating sparse matrix entries from corresponding data
US6408292B1 (en) 1999-08-04 2002-06-18 Hyperroll, Israel, Ltd. Method of and system for managing multi-dimensional databases using modular-arithmetic based address data mapping processes on integer-encoded business dimensions
WO2001039012A2 (en) 1999-11-22 2001-05-31 Avenue, A, Inc. Efficient web server log processing
US20110128972A1 (en) 2000-04-17 2011-06-02 Randy Thornton Peer to peer dynamic network link acceleration
US7120662B2 (en) 2000-04-17 2006-10-10 Circadence Corporation Conductor gateway prioritization parameters
US7139791B1 (en) * 2000-07-13 2006-11-21 International Business Machines Corporation Apparatus and method for providing access to a data stream by a plurality of users at a same time
US7127492B1 (en) * 2000-10-31 2006-10-24 International Business Machines Corporation Method and apparatus for distributed application acceleration
US20030046394A1 (en) * 2000-11-03 2003-03-06 Steve Goddard System and method for an application space server cluster
US6978458B1 (en) 2000-11-17 2005-12-20 Oracle International Corporation Distributing data items to corresponding buckets for use in parallel operations
US20020184327A1 (en) * 2001-05-11 2002-12-05 Major Robert Drew System and method for partitioning address space in a proxy cache server cluster
US7895350B1 (en) * 2001-07-05 2011-02-22 Motive, Inc. N-way data stream splitter
US8156216B1 (en) 2002-01-30 2012-04-10 Adobe Systems Incorporated Distributed data collection and aggregation
US7337214B2 (en) 2002-09-26 2008-02-26 Yhc Corporation Caching, clustering and aggregating server
US6698815B1 (en) 2002-10-02 2004-03-02 Grupo Antolin Ingenieria, S.A. Sun visor with mirror assembly
US8676843B2 (en) * 2002-11-14 2014-03-18 LexiNexis Risk Data Management Inc. Failure recovery in a parallel-processing database system
US7506355B2 (en) * 2002-11-22 2009-03-17 Microsoft Corporation Tracking end-user content viewing and navigation
US7469253B2 (en) * 2003-04-01 2008-12-23 Microsoft Corporation Associative hash partitioning using pseudo-random number generator
US20040230744A1 (en) * 2003-05-17 2004-11-18 Teh Jin Teik Cache content protocol for a content data delivery system
US8234517B2 (en) * 2003-08-01 2012-07-31 Oracle International Corporation Parallel recovery by non-failed nodes
US7523145B2 (en) * 2004-04-22 2009-04-21 Opentv, Inc. System for managing data in a distributed computing system
US7644050B2 (en) * 2004-12-02 2010-01-05 International Business Machines Corporation Method and apparatus for annotation-based behavior extensions
WO2005124632A2 (en) * 2004-06-08 2005-12-29 Rosenthal Collins, Group, Llc Method and system for providing electronic information for multi-market electronic trading
US20080201299A1 (en) * 2004-06-30 2008-08-21 Nokia Corporation Method and System for Managing Metadata
US7380173B2 (en) * 2004-08-03 2008-05-27 International Business Machines Corporation Identifying temporal ambiguity in a aggregated log stream
US8161318B2 (en) 2005-02-07 2012-04-17 Mimosa Systems, Inc. Enterprise service availability through identity preservation
US20060206923A1 (en) 2005-03-09 2006-09-14 Macrovision Corporation Method and system for self-encrypting key identification
US20060236074A1 (en) * 2005-04-14 2006-10-19 Arm Limited Indicating storage locations within caches
JP2006318312A (ja) 2005-05-13 2006-11-24 Victor Co Of Japan Ltd 情報配信管理サーバ
US7587388B2 (en) * 2005-07-28 2009-09-08 Microsoft Corporation Separating uploads into aggregate and raw data storage
US20070027974A1 (en) * 2005-08-01 2007-02-01 Microsoft Corporation Online service monitoring
US8775273B2 (en) 2005-11-23 2014-07-08 Ebay Inc. System and method for transaction automation
US20070124437A1 (en) * 2005-11-30 2007-05-31 Cisco Technology, Inc. Method and system for real-time collection of log data from distributed network components
US7636743B2 (en) * 2005-12-19 2009-12-22 Commvault Systems, Inc. Pathname translation in a data replication system
US7822808B2 (en) 2006-04-21 2010-10-26 Motenet, Inc. Collection and distribution of sense-on-demand data
KR100950661B1 (ko) 2006-05-20 2010-04-02 삼성전자주식회사 통신 시스템에서 신호 수신 장치 및 방법
US20070299631A1 (en) 2006-06-27 2007-12-27 Microsoft Corporation Logging user actions within activity context
US8037052B2 (en) 2006-11-22 2011-10-11 General Electric Company Systems and methods for free text searching of electronic medical record data
JP5396001B2 (ja) 2006-12-13 2014-01-22 楽天Ｅｄｙ株式会社 情報処理装置、情報処理装置の制御方法、及び情報処理装置の制御プログラム
US20080288500A1 (en) * 2007-02-13 2008-11-20 Demetrios Sapounas Physiological data processing architecture for situation awareness
JP5320678B2 (ja) 2007-02-20 2013-10-23 日本電気株式会社 データ分散格納システム及びデータ分散方法、それに用いる装置並びにそのプログラム
US20080256079A1 (en) 2007-04-11 2008-10-16 Yahoo! Inc. Load-based technique to balance data sources to data consumers
JP4422742B2 (ja) 2007-06-11 2010-02-24 達也 進藤 全文検索システム
US8321579B2 (en) * 2007-07-26 2012-11-27 International Business Machines Corporation System and method for analyzing streams and counting stream items on multi-core processors
US7698422B2 (en) 2007-09-10 2010-04-13 Specific Media, Inc. System and method of determining user demographic profiles of anonymous users
US20090157766A1 (en) * 2007-12-18 2009-06-18 Jinmei Shen Method, System, and Computer Program Product for Ensuring Data Consistency of Asynchronously Replicated Data Following a Master Transaction Server Failover Event
US8019863B2 (en) 2008-03-28 2011-09-13 Ianywhere Solutions, Inc. Synchronizing events between mobile devices and servers
US8893131B2 (en) * 2008-04-11 2014-11-18 Yahoo! Inc. System and/or method for bulk loading of records into an ordered distributed database
JP5257311B2 (ja) * 2008-12-05 2013-08-07 ソニー株式会社 情報処理装置、および情報処理方法
KR101207510B1 (ko) 2008-12-18 2012-12-03 한국전자통신연구원 클러스터 데이터 관리시스템 및 클러스터 데이터 관리 시스템에서 공유 재수행 로그를 이용한 데이터 재구축 방법
US8392530B1 (en) * 2008-12-18 2013-03-05 Adobe Systems Incorporated Media streaming in a multi-tier client-server architecture
US8078825B2 (en) * 2009-03-11 2011-12-13 Oracle America, Inc. Composite hash and list partitioning of database tables
US8224924B2 (en) 2009-03-27 2012-07-17 Bank Of America Corporation Active email collector
JP4911198B2 (ja) * 2009-06-03 2012-04-04 富士通株式会社 ストレージ制御装置、ストレージシステムおよびストレージ制御方法
JP5395565B2 (ja) 2009-08-12 2014-01-22 株式会社日立製作所 ストリームデータ処理方法及び装置
US8230046B2 (en) * 2009-12-15 2012-07-24 Facebook, Inc. Setting cookies in conjunction with phased delivery of structured documents
WO2011076463A1 (en) 2009-12-23 2011-06-30 International Business Machines Corporation Deduplication of encrypted data
US8348840B2 (en) 2010-02-04 2013-01-08 Robert Bosch Gmbh Device and method to monitor, assess and improve quality of sleep
US8943053B2 (en) 2010-02-26 2015-01-27 Telefonaktiebolaget L M Ericsson (Publ) Social data ranking and processing
US9081888B2 (en) 2010-03-31 2015-07-14 Cloudera, Inc. Collecting and aggregating log data with fault tolerance
US9082127B2 (en) * 2010-03-31 2015-07-14 Cloudera, Inc. Collecting and aggregating datasets for analysis
US9317572B2 (en) 2010-03-31 2016-04-19 Cloudera, Inc. Configuring a system to collect and aggregate datasets
US8788543B2 (en) * 2010-05-13 2014-07-22 International Business Machines Corporation Scalable, concurrent resizing of hash tables
US8819236B2 (en) 2010-12-16 2014-08-26 Microsoft Corporation Resource optimization for online services
CN103392314B (zh) * 2010-12-29 2016-06-15 思杰系统有限公司 用于可扩展的n核统计信息聚合的系统和方法
CA2827572C (en) 2011-02-23 2019-06-18 Level 3 Communications, Llc Analytics management
US20120246303A1 (en) 2011-03-23 2012-09-27 LogRhythm Inc. Log collection, structuring and processing
US8775556B1 (en) * 2011-07-20 2014-07-08 Google Inc. Automated segmentation and processing of web site traffic data over a rolling window of time
US8775941B1 (en) * 2011-07-20 2014-07-08 Google Inc. System for monitoring and reporting deviations of real-time analytics data from expected analytics data
US20130030868A1 (en) * 2011-07-25 2013-01-31 Cbs Interactive, Inc. Scheduled Split Testing
US11127309B2 (en) 2011-08-09 2021-09-21 The Mitre Corporation Flight management system operator
US9747592B2 (en) 2011-08-16 2017-08-29 Verizon Digital Media Services Inc. End-to-end content delivery network incorporating independently operated transparent caches and proxy caches
US8429162B1 (en) * 2011-09-27 2013-04-23 Amazon Technologies, Inc. Facilitating data redistribution in database sharding
US20150356104A9 (en) 2011-10-04 2015-12-10 Electro Industries/Gauge Tech Systems and methods for collecting, analyzing, billing, and reporting data from intelligent electronic devices
US20130238391A1 (en) 2012-03-09 2013-09-12 Oracle International Corporation Product oriented web site analytics
WO2013177660A1 (en) 2012-05-31 2013-12-05 Netsweeper Inc. Policy service logging using graph structures
US8850263B1 (en) 2012-09-14 2014-09-30 Amazon Technologies, Inc. Streaming and sampling in real-time log analysis
US9104745B1 (en) 2012-09-28 2015-08-11 Palo Alto Networks, Inc. Distributed log collector and report generation
US8447854B1 (en) * 2012-12-04 2013-05-21 Limelight Networks, Inc. Edge analytics query for distributed content network
US9152502B2 (en) 2012-12-21 2015-10-06 Microsoft Technology Licensing, Llc Data error detection and correction using hash values
US9633022B2 (en) 2012-12-28 2017-04-25 Commvault Systems, Inc. Backup and restoration for a deduplicated file system
US9609050B2 (en) 2013-01-31 2017-03-28 Facebook, Inc. Multi-level data staging for low latency data access

Also Published As

Publication number Publication date
US20140214752A1 (en) 2014-07-31
JP2016515228A (ja) 2016-05-26
CA2897338C (en) 2018-08-14
KR20150114965A (ko) 2015-10-13
US10223431B2 (en) 2019-03-05
CN104969213B (zh) 2018-01-12
IL239811A0 (en) 2015-08-31
MX356571B (es) 2018-06-04
IL239811B (en) 2019-01-31
KR101885688B1 (ko) 2018-08-06
BR112015017391A2 (pt) 2017-07-11
JP6258975B2 (ja) 2018-01-10
CN104969213A (zh) 2015-10-07
AU2014212780B2 (en) 2018-09-13
WO2014120487A1 (en) 2014-08-07
CA2897338A1 (en) 2014-08-07
AU2014212780A1 (en) 2015-07-23

Similar Documents

Publication Publication Date Title
MX2015009825A (es) Division de flujo de datos para acceso a datos de baja latencia.
PH12015502654A1 (en) Sharing a virtual hard disk across multiple virtual machines
MY166917A (en) Enhanced block-request streaming system for handling low-latency streaming
NZ754204A (en) Object tracking system optimization and tools
TN2017000306A1 (en) Low latency video streaming
WO2013016679A3 (en) Systems and methods for generating and using a digital pass
MX363982B (es) Sistema de gestion de trafico distribuido y tecnicas.
MX337624B (es) Procesamiento sincronizado de datos por recursos de computo conectados en red.
EP3117406A4 (en) Device, system and method for aggregating networks and serving data from those networks to computers
TR201820876T4 (tr) Bir çokyollu çevrede uyarlanabilir akışlı iletim için sistem ve yöntem.
BR112013026818A2 (pt) métodos e sistemas para transmissão de dados
WO2013033259A3 (en) Systems and methods for a video sharing social network
WO2015000919A3 (en) Method and processor for efficient video processing in a streaming environment
RU2015138523A (ru) Устройства обработки данных и способы обработки данных
GB2556533A8 (en) Mixing content into a content stream
AU2014410705A1 (en) Data processing method and apparatus
IL241640B (en) Method for executing queries on streaming data using graphic processing units
MY163287A (en) Power management for image processing apparatus
EP2765517A3 (en) Data stream splitting for low-latency data access
WO2012140673A3 (en) Audio controller
TW201613357A (en) Audio and video sharing method and system
PH12019500795A1 (en) Method and system for the transmission of bioinformatics data
PH12016000206B1 (en) Method and system for self-registration and self-assembly of electrical devices
IN2014CN02291A (es)
JP2016006958A5 (es)

Legal Events

Date Code Title Description
FG Grant or registration