feat: implement basic zanzibar API contract #266

zepatrik · 2020-10-08T09:58:19Z

This PR is an umbrella PR that keeps track of everything that goes into implementing a zanzibar-like next-gen Keto. It shows the current state of the zanzibar branch where the development happens. If you have general feedback/usecases/.... please let us know either in this PR or by opening an issue.

robinbraemer · 2020-11-09T21:53:23Z

I was just about to build exactly this authorization system based off of Google’s Zanzibar paper, but then found the very good looking “zanzibar/expand-api” branches in the Keto project. Great work so far! 👍

I would love to help and contribute to Keto’s ACL as I’m also familiar with the Zanzibar paper and began some thoughts about such system myself for my organization.

@zepatrik I assume the Zanzibar based ACL system will be a public API as well as provide the under layer for the planned RBAC, ABAC etc. systems as its that flexible enough?

Let me know if there is something I could help with in terms of implementing or as another code reviewer giving ideas and giving feedback!

Things needed to be discussed, I think (or at a later stage):

database interface for zanzibar (Spanner-like global scalability -> CockroachDB with geo-partitioning +follower reads in combination with Zookie concept from the paper)
distributed caching of computed/recent results (for fast responses to requests specifying zookie)
- use google/groupcache to improve throughput (and check tuple once, spread result to nodes - for frequently read queries/check requests)
implementation of the offline secondary indexing system for deeply nested relationships (Leopard, or however we will call it)
rename the “Zookie”? ^^

zepatrik · 2020-11-10T09:35:00Z

Awesome, do you want to join our slack for discussions? https://slack.ory.sh/
I would also be up to jump on a call at some point. Just PM me (Patrik (ory)) on slack.

For transparency, here are some answers to your questions:

There will be a public API that allows requests such as "is user allowed to do relation on object" (currently the check package).
There will be a public API to expand user sets (the expand package).
The ACL language itself is flexible enough to allow implementation of RBAC, ABAC, ... We want to evaluate it today to figure out the limitations now
For the database interface we will probably go with https://github.com/gobuffalo/pop as we had good experience with that. It allows us to support multiple databases at once, including CRDB or Yugabyte through the postgres driver. This way you can use them to get a cloudspanner like database.
The other optimizations you mentioned are planed for a later stage when we have the functionality working.
All the distributed and fan-out part of zanzibar will be added later as well. We try to use algorithms now that allow us to do the recursion over multiple nodes later on.
The whole idea of a zookie is not yet implemented as it heavily relies on the database. Naming in general is something we have to rework anyway.

api/keto/acl/v1alpha1/read_service.proto

internal/persistence/sql/relationtuples.go

jon-whit · 2021-02-03T01:55:06Z

@zepatrik I'd love to pitch in and work on this with you. I too am working on a similar design after having read the Zanzibar paper. I'm also trying to get this type of architecture built and scaled up to work for an internal use case at my company. So I'd have plenty of time to work on it.

I'd love to join the slack and collaborate with you weekly or bi-weekly.

aeneasr · 2021-03-05T17:04:22Z

I would like to get this merged - what's the status on this?

zepatrik · 2021-03-05T17:26:13Z

I updated the milestone: https://github.com/ory/keto/milestone/4
All of the issues are not too much work, just the last details for the release. I think we are at a point where we could rename the current master to e.g. legacy and make this the new master branch, what do you think?

aeneasr · 2021-03-05T18:03:17Z

Yeah, I think that makes sense - setting up a legacy branch based on current master and then force-pushing the zanzibar branch.

aeneasr

Update the introduction page https://github.com/ory/keto/blob/master/docs/docs/index.md

aeneasr · 2021-03-11T17:32:20Z

docs/docs/secure.md

 access control on their own. Any request made to any Keto API is considered
 authenticated, authorized, and is thus being executed. However, these endpoints
 are very sensitive as they define who is allowed to do what in your system.

 Please protect these endpoints using
-[ORY Oathkeeper](https://github.com/ory/oathkeeper) or a comparable API Gateway.
+[Ory Oathkeeper](https://github.com/ory/oathkeeper) or a comparable API Gateway.
 How you protect them, is up to you.

 If you require support for this, consider [asking us](mailto:hi@ory.sh).


Would probably link to GitHub discussions here instead

aeneasr · 2021-03-11T17:35:26Z

docs/sidebar.json

-    "engines/rbac",
-    "engines/acp-ory",
-    "engines/acp-aws"
+  "Concepts": [


Move concepts above guides

aeneasr

docs/docs/concepts/objects.mdx

docs/docs/concepts/subjects.mdx

aeneasr · 2021-03-12T06:26:45Z

docs/docs/configure-deploy.md

@@ -1,166 +0,0 @@
---
-id: configure-deploy
-title: Configure and Deploy


We have this document in all the other projects and it helps people understand how to wire things together. I think it makes sense to re-add and update this, even if it is similar to the Try Along. However, the try-along does not explain how to set up or connect the database, does not say anything about migrations, and so on.

docs/docs/examples/cat-videos.mdx

contrib/cat-videos-example/up.sh

aeneasr · 2021-03-12T07:03:27Z

Makefile

+		&& \
+		npm i \
+		&& \
+		npm test


Please add a "make docker" target which builds the Dockerfile including SQLite!

aeneasr · 2021-03-12T07:05:28Z

Dockerfile

-#
-# $ packr; GO111MODULE=on GOOS=linux GOARCH=amd64 go build; docker build -t oryd/keto:latest .; rm keto; packr clean
-FROM alpine:3.9
+FROM golang:1.16-alpine AS builder


Please follow the structure of https://github.com/ory/kratos/tree/master/.docker

The main Dockerfile is an empty container where goreleaser injects its binary into.

aeneasr · 2021-03-12T07:16:53Z

docs/docs/examples/cat-videos.mdx

+# clone the repository if you don't have it yet
+git clone git@github.com:ory/keto.git && cd keto
+
+docker-compose -f contrib/cat-videos-example/docker-compose.yml up


This fails with several error messages, including the fallback URL.

docs/sidebar.json

aeneasr

Please replace all /TODO links and/or write the content for those links.

docs/docs/concepts/relation-tuples.mdx

aeneasr · 2021-03-12T07:29:25Z

docs/docs/concepts/namespaces.mdx

+[namespace migration CLI reference](/TODO) and
+[running in production guide](/TODO) to learn more about that process.


Update links, write migration guide.

aeneasr · 2021-03-12T07:29:28Z

docs/docs/concepts/namespaces.mdx

+
+Ory Keto knows
F438
 the concept of namespaces to organize
+[relation tuples](./relation-tuples). Namespaces have a configuration that
+defines the relations, and some other important values ([see reference](/TODO)).


Update links

aeneasr · 2021-03-12T07:29:47Z

docs/docs/concepts/namespaces.mdx

+The namespaces' purpose is to split up the data into coherent partitions, each
+with its corresponding configuration. Internally each namespace has its own
+table in the database to allow setting individual
+[storage specific options](/TODO).


Update links - basically just search for /TODO

aeneasr

Can you please re-generate swagger and rebuild the docs so that I can verify the REST API? :)

docs/docs/examples/olymp-file-sharing.mdx

internal/check/engine.go

aeneasr · 2021-03-12T07:37:43Z

internal/check/handler.go

+
+// swagger:route GET /check read getCheck
+//
+// Check a relation tuple


NYT case for all swagger titles please :)

aeneasr · 2021-04-01T10:34:09Z

Is there a reason why /relationtuple is not following REST best practice of plural? Also, is relation tuple one word or two? If it is two, should we use /relation-tuples?

aeneasr

Except for

Is there a reason why /relationtuple is not following REST best practice of plural? Also, is relation tuple one word or two? If it is two, should we use /relation-tuples?

and the "todo" pages this is good to go from my side!

aeneasr

~~What can I say? Good job!~~

~~NickUfer reacted with thumbs up emoji NickUfer reacted with hooray emoji NickUfer reacted with rocket emoji~~

zepatrik marked this pull request as draft October 8, 2020 09:58

zepatrik added the corp/m3 Up for M3 at Ory Corp. label Oct 8, 2020

zepatrik self-assigned this Oct 8, 2020

aeneasr mentioned this pull request Oct 11, 2020

ORY Keto will become the first open source Google Zanzibar implementation! #267

Closed

zepatrik mentioned this pull request Nov 2, 2020

Limit applied before filtering #280

Closed

aeneasr added corp/m4 Up for M4 at Ory Corp. and removed corp/m3 Up for M3 at Ory Corp. labels Nov 9, 2020

zepatrik mentioned this pull request Dec 22, 2020

Keto integration with Kratos and Oathkeeper #381

Closed

robinbraemer approved these changes Jan 16, 2021

View reviewed changes

api/keto/acl/v1alpha1/read_service.proto Outdated Show resolved Hide resolved

internal/persistence/sql/relationtuples.go Outdated Show resolved Hide resolved

zepatrik mentioned this pull request Jan 16, 2021

SQL persister performance improvements #396

Closed

1 task

aeneasr requested changes Mar 11, 2021

View reviewed changes

aeneasr requested changes Mar 12, 2021

View reviewed changes

aeneasr reviewed Mar 12, 2021

View reviewed changes

aeneasr requested changes Mar 12, 2021

View reviewed changes

zepatrik mentioned this pull request Mar 12, 2021

docs: suggestions from review #486

Merged

aeneasr marked this pull request as ready for review April 1, 2021 10:01

aeneasr reviewed Apr 1, 2021

View reviewed changes

aeneasr force-pushed the zanzibar branch from 62b7ea8 to 6f04bbb Compare April 1, 2021 10:58

aeneasr approved these changes Apr 1, 2021

View reviewed changes

aeneasr merged commit d32767f into master Apr 1, 2021

aeneasr force-pushed the master branch from 10f4b2e to d32767f Compare April 1, 2021 13:08

zepatrik deleted the zanzibar branch April 8, 2021 10:20

		[namespace migration CLI reference](/TODO) and
		[running in production guide](/TODO) to learn more about that process.

feat: implement basic zanzibar API contract #266

feat: implement basic zanzibar API contract #266

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!