API importer for Notion #8710

hmacr · 2025-03-17T13:43:55Z

This PR enables importing Notion documents using an API integration.

Pending tasks

Note: bigger PR, but not as big as it seems to be.. ~9k lines are related to tests.

* tom/misc-fixes (#8650) * Revert "chore: Upgrade `path-to-regexp` (#8636)" (#8652) This reverts commit 58c4a48. * chore(deps): bump axios from 1.7.9 to 1.8.2 (#8653) Bumps [axios](https://github.com/axios/axios) from 1.7.9 to 1.8.2. - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](axios/axios@v1.7.9...v1.8.2) --- updated-dependencies: - dependency-name: axios dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Bump dd-trace (#8654) * chore: Bump @koa/bull-board (#8655) * create import_tasks table * rate-limited notion api calls * completion flow * ImportsProcessor create event handling * update imports table migration file * works till final step * page title and emoji * works almost - tiny bugs in persistence * remove local oauth redirect * stash * Passing test suite * fix: Nested lists * Add video node * fix: Missing paragraphs * Add more mention support * works with attachments and mentions * better types * page type from ui * wip: database support * list consolidated imports * websocket events * name param in import dialog * import policy * handle failures * abstraction and cleanup * even more abstraction and typings * attachments logic to base api-import task * delete import and policies * indexes and scopes * notion database support * order * fix: Filter empty rich_text * Add callout support * Add column, column_list support (flatted into document until we have support) * Add link_preview * Add synced_block (flatten into page) * refactor persistence * generate import name * fix issues; don't use documentCreator as it is lossy based on text * remove local oauth redirect * remove commented code --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Tom Moor <tom@getoutline.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Tom Moor <tom.moor@gmail.com>

Copilot

Pull Request Overview

This PR introduces an API importer for Notion, adding both backend API routes and frontend integration components to enable importing Notion pages into the application. Key changes include:

New API endpoints and OAuth handling for Notion (plugins/notion/server/api/notion.ts and schema).
Frontend integration for initiating and managing Notion imports via a new dialog and import menu.
Additional store, WebSocket event listeners, and plugin hook support for managing import-related functionality.

Reviewed Changes

Copilot reviewed 66 out of 66 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
plugins/notion/server/api/notion.ts	Implements Notion callback and search endpoints.
plugins/notion/client/Imports.tsx	Provides the Notion import button and OAuth flow initiation.
plugins/notion/server/env.ts	Configures the Notion client credentials.
app/models/Import.ts	Introduces a new Import model to track import state.
plugins/notion/client/index.tsx	Registers the Notion import UI component via plugin hooks.
app/menus/ImportMenu.tsx	Adds a context menu for import items with delete actions.
app/scenes/Settings/components/ImportListItem.tsx	Displays import details including status and page counts.
app/stores/ImportsStore.ts	Provides a dedicated store for managing Import models.
plugins/notion/server/api/schema.ts	Defines schemas for Notion callback and search requests.
plugins/notion/client/components/ImportDialog.tsx	Implements a dialog to select pages and set permissions for import.
app/scenes/Settings/Import.ts	Updates the import scene to integrate the new Notion import flow.
app/components/WebsocketProvider.ts	Adds event listeners for real-time import updates.
app/utils/PluginManager.ts	Introduces a new plugin hook for imports.
app/components/Dialogs.tsx	Improves dialog onClose handling.
app/hooks/usePolicy.ts	Adds an option to force reload policy data.
app/stores/DialogsStore.ts	Updates the dialogs store to support onClose callbacks.
app/stores/RootStore.ts	Registers the new ImportsStore.

Comments suppressed due to low confidence (1)

plugins/notion/client/components/ImportDialog.tsx:89

The dependency array for the handleStartImport callback includes 'name', which is not defined in this scope. Remove 'name' from the dependency array to avoid potential runtime errors.

}], [name, pagesWithPermission, onSubmit]);

tommoor · 2025-03-18T02:29:25Z

app/hooks/usePolicy.ts

-export default function usePolicy(entity?: string | Model | null) {
+export default function usePolicy(
+  entity?: string | Model | null,
+  { force }: { force: boolean } = { force: false }


This seems dangerous, would rather not have it 😬

Hmm we need a way to reload policy when a property in entity changes 🤔
In almost all cases, usePolicy is used in a fresh page / render - so entity remains fresh for the useEffect dep reference checks. But for ImportListItem, the ref remains the same as the realtime progress count keeps getting updated.
Perhaps, force-loading the import model in ImportListItem is better as an one-off case?

tommoor · 2025-03-18T02:59:52Z

plugins/notion/server/oauth.ts

+import { NotionUtils } from "../shared/NotionUtils";
+import env from "./env";
+
+export class NotionOAuth {


Not sure why this one deserves it's own class… could be static methods on NotionClient perhaps

yea, can be included in NotionClient.

tommoor · 2025-03-18T03:02:54Z

server/constants.ts

@@ -0,0 +1 @@
+export const PagePerImportTask = 3;


Not worth a new file here

tommoor · 2025-03-18T03:08:30Z

server/models/Import.ts

+  @IsNumeric
+  @Default(0)
+  @Column(DataType.INTEGER)
+  pageCount: number;


Is there a more generic term here thinking this will be used for more than just Notion.

progress?
completedCount?

I'm thinking pageCount is generic enough? In the end, a document or a page from an external source will be imported? 🤔

documentCount maybe?

documentCount is better - I would rather use our language than notion's

tommoor · 2025-03-18T03:09:26Z

server/policies/import.ts

+import { allow } from "./cancan";
+import { and, isTeamAdmin, isTeamMutable } from "./utils";
+
+allow(User, ["createImport", "readImport"], Team, (actor, team) =>


Suggested change

allow(User, ["createImport", "readImport"], Team, (actor, team) =>

allow(User, ["createImport", "listImports"], Team, (actor, team) =>

This would be more typical

tommoor · 2025-03-18T03:16:12Z

server/queues/processors/ImportsProcessor.ts

+  public async perform(event: ImportEvent) {
+    const importModel = await Import.findByPk<Import<T>>(event.modelId, {
+      rejectOnEmpty: true,
+      paranoid: false,


Should probably query with an update lock

tommoor · 2025-03-18T03:19:29Z

server/routes/api/imports/imports.ts

+    authorize(user, "readImport", user.team);
+
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    const where: WhereOptions<Import<any>> = service


Need to include teamId in query here

tommoor · 2025-03-18T03:19:58Z

server/routes/api/imports/imports.ts

+    const importModel = await Import.findByPk(id, {
+      rejectOnEmpty: true,
+      transaction,
+      lock: transaction.LOCK.UPDATE,


No need for update lock or transaction here as we're only querying

tommoor · 2025-03-18T03:21:00Z

server/routes/api/imports/imports.ts

+
+    authorize(user, "createImport", user.team);
+
+    const pendingImport = await Import.findOne({


nbd but could just use Import.count here

tommoor · 2025-03-18T03:22:06Z

server/queues/tasks/UploadAttachmentsForImportTask.ts

+
+        const attachment = await Attachment.findByPk(item.attachmentId, {
+          rejectOnEmpty: true,
+          include: [{ association: "user" }],


Not clear why user is needed here?

my bad, not needed here.

hmacr · 2025-03-18T08:09:14Z

server/queues/processors/ImportsProcessor.ts

+            const isRootDocument =
+              !parentExternalId || !!importInput[parentExternalId];
+
+            await Document.createWithCtx(


On par with what's happening in existing file-based import flow, i.e. publish documents.create event for each document.

I noticed that this leads to flood of requests from the app when the import is complete. I really think we should suppress this event, and retain just the collections.create events.

One thing you'll see is that a lot of processors suppress handling when the event is from an import, e.g.

outline/server/queues/processors/NotificationsProcessor.ts

Line 60 in 2b44a61

event.data.source === "import"

Ah yes, I was aware of it - current impl sets the data source.
But I didn't debug enough to find what was causing these additional requests, looks like WebsocketsProcessor needs some updating (missing apiImportId check) - will change it to check for event.data.source === "import"

outline/server/queues/processors/WebsocketsProcessor.ts

Lines 52 to 54 in 021a286

if (event.name === "documents.create" && document.importId) {

return;

}

hmacr · 2025-03-18T15:47:40Z

All comments are addressed, pushed a couple of improvements - lmk if something needs changing.

Meanwhile, I'll work on adding a "cancel" option and some tests for APIs & tasks.

Add policies to fileOperations.list Move policy reaction to model layer

hmacr and others added 18 commits March 7, 2025 02:15

config driven imports page

b7e6d36

existing notion importer as plugin

0914dbf

oauth redirect

c471046

exchange code for access_token and persist

d1eb948

show oauth errors

a36b7a6

list root pages in a dialog

c294f41

basic table UI

90a9b14

create import

0b35e06

remove local oauth redirect

0ba22c2

ui

7974d58

Add link_to_page support

e8a2229

lint

eac7c37

i18n

2eda9da

collection content overwritten in collections.list API

0011bfd

load latest policy when import completes

9ddaf41

doc source metadata

50639e7

Merge branch 'main' into notion-importer

33f278b

hmacr requested a review from tommoor March 17, 2025 13:43

tommoor requested a review from Copilot March 17, 2025 13:46

Copilot AI reviewed Mar 17, 2025

View reviewed changes

hmacr added 5 commits March 17, 2025 19:24

cleanup name

8f06f35

tsc

d71ad7a

CleanupOldImportTasks task

b95c3a7

ErrorTimedOutImports task

0e21aad

restrict to 1 active import per team

03d4132

tommoor reviewed Mar 18, 2025

View reviewed changes

hmacr added 3 commits March 18, 2025 10:09

remove constants file

1dc9a7a

move oauth to NotionClient

3f7280c

rename policy

98718dd

hmacr added 5 commits March 18, 2025 11:21

fetch import with lock

e9de4ac

buffer issue

c1bb509

better docStructure update

68a45fd

teamId

432714d

fix table indexes

3f98207

hmacr commented Mar 18, 2025

View reviewed changes

hmacr added 7 commits March 18, 2025 20:03

jsdoc

d793b66

suppress websocket events for imports

0ec97c9

fix NotionConverter test

659a383

move NotionConverter inside plugin

11544b3

rename pageCount to documentCount

a1b3c9d

re-throw upload attachments error for retry

015a9bb

handle child_page, child_database in NotionConverter

bb3f041

hmacr and others added 14 commits March 19, 2025 00:54

cancel import from UI

b730c5c

modify cleanup tasks to account for canceled state

6d92053

remove obsolete snapshot

2ed98e7

tests

0693b6a

handle bot owner NA case

8327d4b

timeout error

04db4f5

use createdAt, updatedAt from Notion

25edb28

load root pages in processor

d9f8cda

handle empty doc

eb345bc

tests

e8f9e9e

Merge branch 'main' into notion-importer

ba89f9f

cleanup

1027e93

Improve usePolicy hook

cec1aef

Add policies to fileOperations.list Move policy reaction to model layer

fix: Mark NOTION_ env as optional

f99507d

tommoor merged commit 6e98568 into main Mar 23, 2025
12 checks passed

hmacr deleted the notion-importer branch March 27, 2025 10:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

API importer for Notion #8710

API importer for Notion #8710

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

	allow(User, ["createImport", "readImport"], Team, (actor, team) =>
	allow(User, ["createImport", "listImports"], Team, (actor, team) =>


		authorize(user, "createImport", user.team);

		const pendingImport = await Import.findOne({

	if (event.name === "documents.create" && document.importId) {
	return;
	}

Uh oh!

API importer for Notion #8710

API importer for Notion #8710

Uh oh!

Conversation

Uh oh!

Pending tasks

Uh oh!

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!