“If you can cache everything in a very efficient way, you can often change the game” We software engineers often face problems that require the dissemination of a dataset which doesn’t fit the label “big data”. Examples of this type of problem include: Product metadata on an ecommerce siteDocument metadata in a search engineMetadata about movies and TV shows on an Internet television networkWhen f
Previously we wrote about our traffic intuition tool, Flux. We have some announcements and updates to share about this project. First, we have renamed the project to Vizceral. More importantly, Vizceral is now open source! Open SourceVizceral transformed the way we understand and digest information about the state of traffic flowing into the Netflix control plane. We wanted to be able to intuit de
At Netflix, our goal is to predict what you want to watch before you watch it. To do this, we run a large number of machine learning (ML) workflows every day. In order to support the creation of these workflows and make efficient use of resources, we created Meson. Meson is a general purpose workflow orchestration and scheduling framework that we built to manage ML pipelines that execute workloads
In summary, the Keystone pipeline is a unified event publishing, collection, and routing infrastructure for both batch and stream processing. We have two sets of Kafka clusters in Keystone pipeline: Fronting Kafka and Consumer Kafka. Fronting Kafka clusters are responsible for getting the messages from the producers which are virtually every application instance in Netflix. Their roles are data co
The Netflix Way to deal with Big Data Problems 1. The way to deal with Big data problems Monal Daxini March 2016 2. Monal Daxini Real Time Data Infrastructure Senior Software Engineer, Netflix https://www.linkedin.com/in/monaldaxini @monaldax #Netflix #Keystone 3. We help Produce, Store, Process, Move Events @ scale 4. Tell me more... ● Big Data Ecosystem @ Netflix ● How we built a scalable event
Back in January of 2014 we wrote about the need for better visibility into our complex operational environments: The core of the message in that post was about the need for fine-grained, contextual and scalable insights into the experiences of our customers and behaviors of our services. While our execution has evolved somewhat differently from our original vision, the underlying principles behind
Our new Keystone data pipeline went live in December of 2015. In this article, we talk about the evolution of Netflix’s data pipeline over the years. This is the first of a series of articles about the new Keystone data pipeline. Netflix is a data-driven company. Many business and product decisions are based on insights derived from data analysis. The charter of the data pipeline is to collect, ag
Slides from the 2016/01/28 Advanced Amazon Web Services (AWS) Meetup. Netflix overviewed the usage of containers at Netflix. We covered technologies we are working on in the runtime (Titus) and developer experience (Newt). We talked about how the Titus container management system was different from others and our journey with Docker, Mesos, Netflix Fenzo and eventually Amazon Elastic Container Ser
This document discusses using Spring Cloud Netflix to connect microservices. It covers: - An overview of Netflix and Spring Cloud - How to use Spring Cloud Netflix libraries like Eureka, Ribbon and Hystrix to implement service discovery, load balancing and fault tolerance between microservices - Demos of creating Eureka service registry and registering producer/consumer servicesRead less
Ribbon comprises of multiple components some of which are used in production internally and some of which were replaced by non-OSS solutions over time. This is because Netflix started moving into a more componentized architecture for RPC with a focus on single-responsibility modules. So each Ribbon component gets a different level of attention at this moment. More specifically, here are the compon
Springが、Netflixが公開している彼らのOSS群を説明している記事を読みました。ちょっと、記憶に止めておきたい内容を探っているのでそのメモがてら。 記事: http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html これは、Netflixのmicroservicesを構成するツール群を説明していました。 Service Discovery Eureka Clients Circuit Breaker Hystrix Intelligent Routing Zuul Client Side Load Balancing Ribbon Service Discovery: Eureka Clients Service Discoveryはmicsoservicesベースのアーキテクチャとして大事な要素 C
検索しているとなにかとNetflixのgithubリポジトリがヒットするので、全部（2015/07/18現在分）調査してみた。 github APIで https://github.com/Netflix のリストを全部取得して、名前・概要・URL・最終更新日時 (なんの更新だ?) を抽出。 AWS用のプロダクトが多かったのでまずそれらと、その他という分類にした。その他はほとんどがJavaライブラリ・システムだが、一部WebアプリケーションやPythonライブラリがある。 日本語での説明はReadmeやWikiを見て書いているが、理解が正しくないかもしれない。 AWS用 aws-autoscaling Tools and Documentation about using Auto Scaling URL: https://github.com/Netflix/aws-autoscalin
Testing Strategies In A Microservice Architectureを読んだを読んでいる途中に出てきた、Circuit Breakerと呼ばれる機構を調べてみました。Martin Fowler氏がこの記事で言及しているものでした。 このCircuit Breaker patternは、Release It! 本番用ソフトウェア製品の設計とデプロイのためにで描かれているような、本番環境化において発生する、複数システムが関係するからこそ発生する障害を抑えることも目的としたデザインパターンのようです。「複数システムが関係するからこそ発生する障害」とは、一部システムの負荷が高まりタイムアウトするといったことを含みます。 内容自体は、 障害検出のための共有のオブジェクト(Circuit Breacker)を用意して、監視・検出できるようにする ということらしいです。Ne
マルチクラウドのインフラアプリデリバリの自動化と管理ツールがNETFLIXから11/17にオープンソースで公開されたのでさわって見た。 NETFLIXがGoogle,Microsoft,Pivotalと共同開発したそうです。 名前は「Spinnaker」(スピネーカー)、意味は"三角形の帆"ですが由来は不明です。 NETFLIXブログ Netflix、マルチクラウド対応の継続的デリバリを実現する「Spinnaker」をオープンソースで公開 － Publickey Pivotalブログ Cloud Native Continuous Delivery With Netflix Spinnaker | Pivotal P.O.V. 本家 The Netflix Tech Blog: Global Continuous Delivery with Spinnaker Spinnaker: Glo
Fenzo is a scheduler Java library for Apache Mesos frameworks that supports plugins for scheduling optimizations and facilitates cluster autoscaling. Apache Mesos frameworks match and assign resources to pending tasks. Fenzo presents a plugin-based, Java library that facilitates scheduling resources to tasks by using a variety of possible scheduling objectives, such as bin packing, balancing acros
by Leena Janardanan, Bruce Wobbe, Vilas Veeraraghavan IntroductionMerchandising Application Platform (MAP) was conceived as a middle-tier service that would handle real time requests for content discovery. MAP does this by aggregating data from disparate data sources and implementing common business logic into one distinct layer. This centralized layer helps provide common experiences across devic
Bringing Netflix to our millions of subscribers is no easy task. The product comprises dozens of services in our distributed environment, each of which is operating a critical component to the experience while constantly evolving with new functionality. Optimizing the launch of these services is essential for both the stability of the customer experience as well as overall performance and costs. T
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く