Nikhilesh Nukala — Consultant (Data Engineering), Yuhao Zhu — Advanced Analytics Consultant, Guilherme Braccialli — Principal Data Engineer, Tom Goldenberg- Jr Principal (Data Engineering), QuantumBlack This blog will demonstrate a performance benchmark in Apache Spark between Scala UDF, PySpark UDF and PySpark Pandas UDF.At QuantumBlack, we often deal with multiple terabytes of data to drive adva
This series of blog posts and corresponding talks aims to provide you with a pragmatic view on containers from a historic perspective. Together we will discover modern cloud architectures layer by layer, which means we will start at the Linux Kernel level and end up at writing our own secure cloud native applications. Simple examples paired with the historic background will guide you from the begi
Apache Iceberg Branch & Tags With Apache Hive 4.xIntroduction:For sophisticated snapshot lifecycle management, Iceberg supports branches and tags which are named references to snapshots with their own independent lifecycles. This lifecycle is controlled by branch and tag level retention policies. Branches are independent lineages of snapshots and point to the head of the lineage. Prerequisites:Wor
The purpose of this blog is to provide a list of Shuffle Service Implementations in Apache Spark and motivation for their design choices. This blog will also cover the high level understanding of Spark Shuffle service. Credit Note: All Credit goes to actual blogs and you tube videos published already for each Shuffle service implementation and the relevant links have been added in each section of
事業の成果や開発組織のリソースを最終的なアウトプット先である管理会計や財務会計に、私達が日頃から作っているソフトウェアどういった風に計上されているか。前提としては、自社開発でのソフトウェア資産計上とする。 まずは、なぜ資産計上をするべきかなのか、ひいてはプロダクト開発にどう関わってくるかを考えていく。 なんでソフトウェアを資産計上しないといけないのか私達が作ったソフトウェアというを資産計上しなければいけないかというと、端的に言えば、正しい財務諸表（B/S : 貸借対照表）のためです。あと普通に資産計上しないと脱税になるはず。（実際には、色々な資産巡りでのあれこれでの対策防止があるらしい） B/Sの中で、固定資産という枠があり、ソフトウェアはその中の「無形固定資産」にあたります。細かいですが、ソフトウェアで計上できるものは確実に将来の収益や費用削減が見込まれるものになります。それ以外は、費用
The new generation data lake table formats (Apache Hudi, Apache Iceberg, and Delta Lake) are getting more traction every day with their superior capabilities compared to Apache Hive. They enable cost-effective cloud solutions for big data analysis with ACID transactions, schema evolution, time travel, and more. Table FormatsTable format technology is essential for big data environments to properly
https://speakerdeck.com/ktock/dockerkaracontainerdhefalseyi-xing背景: Kubernetes 1.24は組み込み機能としてのDocker対応を打ち切る2014年に公開された初期のKubernetesはDockerにのみ対応していましたが、2016年のKubernetes 1.5では Container Runtime Interface (CRI) と呼ばれる共通インターフェースが導入され、 CRIに対応した任意のランタイムが利用可能になりました。以来、様々なランタイムが開発されてきましたが、2022年現在では containerd と CRI-O の 2つが主流です。 CRIが導入されてからも、Kubernetesに組み込まれているDocker対応機能(dockershim)が広く使われていましたが、2022年4月リリース予
Have you seen your application get stuck or fail to respond to health check requests, and you can’t find any explanation? It might be because of the CPU quota limit. We will explain more here. TL;DR: We would highly recommend removing CPU Limits in Kubernetes (or Disable CFS quota in Kublet) if you are using a kernel version with CFS quota bug unpatched. There is a serious, known CFS bug in the ke
This is a guide which will help you get started with etcd and help you understand how it is used in a kubernetes setup. To put in one sentence, etcd is a distributed, reliable key-value store for the most critical data of a distributed system. Every necessary detail of every resource in a cluster is stored in the form of key-value pairs divided into different directories as per the type of resourc
For a young company, the value of a VP of Product or CPO can be difficult to ascertain. Many argue that a senior PM with 3–8 years of experience would satisfy the business’ strategic needs. And that’s indeed true for the first PM hire as I outlined in my 0–1 article. On the other hand, a CPO brings a deeper and more strategic product perspective that can help companies build a vision for the futur
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く