RELEASE ENG-268 Util script #3925
Conversation
Backmerge master into develop
Backmerge master into develop
Ref eng-268 Signed-off-by: Rafael Leite <2132564+leite08@users.noreply.github.com>
ENG-268 Add script to ingest pts from CSV
WalkthroughA log statement was removed from the patient ingestion route, leaving the logic unchanged. Additionally, a new TypeScript script was added to ingest patient data from a CSV file into OpenSearch, with support for chunked processing, error handling, and concurrency control. The Lambda function timeout setting was increased from 3 to 5 minutes. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Script
participant API
User->>Script: Run ingest-patients.ts with CSV file
Script->>Script: Parse CSV, normalize IDs, group by customer
loop For each chunk of 100 patients (5 concurrent)
Script->>API: POST /internal/medical/patient-consolidated (with patient IDs)
API-->>Script: Responds with ingestion result
alt Failure
Script->>Script: Log error, record failed chunk
end
end
Script->>User: Output summary and error report
Possibly related PRs
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
npm error code ERR_SSL_WRONG_VERSION_NUMBER 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
🧰 Additional context used📓 Path-based instructions (1)`**/*.ts`: - Use the Onion Pattern to organize a package's code in layers - Try to use immutable code and avoid sharing state across different functions, objects, and systems - Try...
🔇 Additional comments (1)
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
packages/utils/src/open-search/ingest-patients.ts (2)
56-70: Consider improving type safety for CSV data processing.The use of
anytype is documented with eslint-disable, but consider creating a more specific interface for the CSV data structure to improve type safety.Define a more specific interface:
+interface CsvRow { + [key: string]: string | undefined; +} - // eslint-disable-next-line @typescript-eslint/no-explicit-any - .on("data", (data: any) => { + .on("data", (data: CsvRow) => {
127-145: Consider improving function return type and parameter structure.While the function works correctly, consider improving the design to better follow functional programming principles and error handling guidelines.
Consider using a Result-like pattern for better error handling:
+interface MappingResult { + success: boolean; + data?: PatientRecord; + errors?: Array<{ field: string; error: string }>; +} -export function mapCsvToIds(csvPatient: { - cxId: string | undefined; - cx_id: string | undefined; - cxid: string | undefined; - id: string | undefined; - patientId: string | undefined; - patientid: string | undefined; -}): PatientRecord | Array<{ field: string; error: string }> { +export function mapCsvToIds(csvPatient: Record<string, string | undefined>): MappingResult { const cxIdRaw = csvPatient.cxId ?? csvPatient.cx_id ?? csvPatient.cxid ?? csvPatient.id; const cxId = cxIdRaw ? normalizeExternalIdUtils(cxIdRaw) : undefined; const patientIdRaw = csvPatient.patientId ?? csvPatient.patientid ?? csvPatient.id; const patientId = patientIdRaw ? normalizeExternalIdUtils(patientIdRaw) : undefined; if (!cxId || !patientId) { - return [{ field: "general", error: "Missing required fields" }]; + return { + success: false, + errors: [{ field: "general", error: "Missing required fields" }] + }; } - return { cxId, patientId }; + return { success: true, data: { cxId, patientId } }; }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
packages/api/src/routes/internal/medical/patient-consolidated.ts(0 hunks)packages/utils/src/open-search/ingest-patients.ts(1 hunks)
💤 Files with no reviewable changes (1)
- packages/api/src/routes/internal/medical/patient-consolidated.ts
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.ts`: - Use the Onion Pattern to organize a package's code in layers - Try to use immutable code and avoid sharing state across different functions, objects, and systems - Try...
**/*.ts: - Use the Onion Pattern to organize a package's code in layers
- Try to use immutable code and avoid sharing state across different functions, objects, and systems
- Try to build code that's idempotent whenever possible
- Prefer functional programming style functions: small, deterministic, 1 input, 1 output
- Minimize coupling / dependencies
- Avoid modifying objects received as parameter
- Only add comments to code to explain why something was done, not how it works
- Naming
- classes, enums:
PascalCase- constants, variables, functions:
camelCase- file names:
kebab-case- table and column names:
snake_case- Use meaningful names, so whoever is reading the code understands what it means
- Don’t use negative names, like
notEnabled, preferisDisabled- For numeric values, if the type doesn’t convey the unit, add the unit to the name
- Typescript
- Use types
- Prefer
constinstead oflet- Avoid
anyand casting fromanyto other types- Type predicates: only applicable to narrow down the type, not to force a complete type conversion
- Prefer deconstructing parameters for functions instead of multiple parameters that might be of
the same type- Don’t use
nullinside the app, only on code interacting with external interfaces/services,
like DB and HTTP; convert toundefinedbefore sending inwards into the code- Use
async/awaitinstead of.then()- Use the strict equality operator
===, don’t use abstract equality operator==- When calling a Promise-returning function asynchronously (i.e., not awaiting), use
.catch()to
handle errors (seeprocessAsyncErrorandemptyFunctiondepending on the case)- Date and Time
- Always use
buildDayjs()to createdayjsinstances- Prefer
dayjs.duration(...)to create duration consts and keep them asduration- Prefer Nullish Coalesce (??) than the OR operator (||) to provide a default value
- Avoid creating arrow functions
- Use truthy syntax instead of
in- i.e.,if (data.link)notif ('link' in data)- Error handling
- Pass the original error as the new one’s
causeso the stack trace is persisted- Error messages should have a static message - add dynamic data to MetriportError's
additionalInfoprop- Avoid sending multiple events to Sentry for a single error
- Global constants and variables
- Move literals to constants declared after imports when possible (avoid magic numbers)
- Avoid shared, global objects
- Avoid using
console.logandconsole.errorin packages other than utils, infra and shared,
and try to useout().loginstead- Avoid multi-line logs
- don't send objects as a second parameter to
console.log()orout().log()- don't create multi-line strings when using
JSON.stringify()- Use
eslintto enforce code style- Use
prettierto format code- max column length is 100 chars
- multi-line comments use
/** */- scripts: top-level comments go after the import
packages/utils/src/open-search/ingest-patients.ts
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: check-pr / lint-build-test
- GitHub Check: check-pr / lint-build-test
- GitHub Check: Analyze (javascript)
- GitHub Check: api / deploy
🔇 Additional comments (5)
packages/utils/src/open-search/ingest-patients.ts (5)
1-12: LGTM: Proper import organization and configuration.The imports are well-organized with dotenv configuration correctly placed at the top, and all necessary dependencies are imported following the established patterns.
23-25: Good constant declarations following guidelines.Constants are properly declared after imports using meaningful names and camelCase convention as specified in the coding guidelines.
89-113: LGTM: Excellent implementation of concurrent processing with proper error handling.The implementation correctly uses
executeAsynchronouslyfor controlled concurrency, implements network retries, and properly captures failed ingestions for later analysis. The error handling preserves error details while continuing processing.
147-149: LGTM: Proper module execution pattern.The script correctly uses the standard Node.js pattern to execute the main function only when the file is run directly.
36-37: 🛠️ Refactor suggestionReplace console.log with out().log for consistency.
The coding guidelines specify to avoid
console.login packages and useout().loginstead. Since this is a script in the utils package, consider using the standard logging utility.Import the logging utility at the top:
+import { out } from "@metriport/core/util/log";Then replace console.log statements throughout the file:
- console.log(`############ Starting... ############`); + out().log(`############ Starting... ############`);Apply this pattern to all other console.log statements in the file.
Also applies to: 73-74, 76-76, 79-79, 87-87, 93-93, 109-109, 116-118, 122-124
⛔ Skipped due to learnings
Learnt from: thomasyopes PR: metriport/metriport#3466 File: packages/api/src/routes/ehr/shared.ts:122-147 Timestamp: 2025-03-17T17:01:17.227Z Learning: Avoid using console.log and console.error in packages other than utils, infra and shared, and try to use out().log instead according to Metriport's coding guidelines.Learnt from: leite08 PR: metriport/metriport#3489 File: packages/api/src/routes/ehr/elation/appointment-webhook.ts:32-36 Timestamp: 2025-03-21T00:21:26.928Z Learning: Use `out().log` instead of `console.log` for logging in packages other than utils, infra and shared.Learnt from: leite08 PR: metriport/metriport#3857 File: packages/utils/src/consolidated/filter-consolidated.ts:39-68 Timestamp: 2025-05-27T23:51:45.100Z Learning: Utility scripts in the packages/utils directory don't require comprehensive error handling or logging changes (like switching from console.log to out().log), as they are meant for local/developer environments, not production code.
Ref eng-268 Signed-off-by: Rafael Leite <2132564+leite08@users.noreply.github.com>
Ref eng-268 Signed-off-by: Rafael Leite <2132564+leite08@users.noreply.github.com>
ENG-268 Increase ingestion timeout
Issues:
Dependencies
none
Description
Testing
Check each PR.
Release Plan
masterSummary by CodeRabbit
Summary by CodeRabbit