Tags: pietern/mlops-stacks
Tags
Rename Stack to Stacks and Fix Bugs (databricks#107) * Rename Stack to Stacks and Fix Bugs * Update version to 0.208.1 * Changed private preview notice to public preview * Add success_message * Fix bug * Fix doc * Fix deployment bug * Fix doc bug
Update MLOps Stacks to Use Databricks Asset Bundles (databricks#77) * Remove terraform, bootstrap script, terraform related github actions Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Remove project_name_alphanumeric. Add project_name_alphanumeric_underscore and update the project folder name to project_name_alphanumeric_underscore Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Move the requriements file into the project folder and update mlflow version to resolve a recipe issue Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Update the model name and experiment under different environments. The experiment folder will be removed until bundles supports managing it. Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Add project name as prefix of github and ADO workflows Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Update notebooks to enter notebook folder and current folder. Update format. Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Directly do model deployment in the workflow run and avoid the callback to github. Add model name parameter to BatchInference notebook to avoid reading it from terraform output file Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * add bundle file and bundle resources. Some small fix Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Prepare the batch inference with FS input table in CUJ prod workspace, update bundle to use the new table and fix fs batch inference Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Fix tests Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Support shared cluster between tasks in one job. Fix a path. Remove explicit token. Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Add CI CD support for mlops using bundle and update notebook path according latest bundle update Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Fix CI PR comment Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Fix integration tests Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Add git source info to the notebook parameters Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Remove Bricks CLI repo token Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Support authentication for an Azure service principal in mlops-stack bundle Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Fix Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Remove experiment parent dir (databricks#9) * Remove experiment parent dir Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Update experiment to be in the current user's directory Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Fix test Signed-off-by: Mingyu Li <mingyu.li@databricks.com> --------- Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Add new bricks bundle ML resource configuration to docs and remove terraform (databricks#5) * update docs Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Add bundle and remove terraform from gitignore Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Remove terraform from MLOps-stacks except ADO CICD. Fix tests and update more docs Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Add file names to gitignore Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * More doc changes. Move the CICD set up section from ML resource README to mlops-setup doc Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Address comments Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Do not use hard coded aws doc for bricks CLI bundles since azure doc is also available Signed-off-by: Mingyu Li <mingyu.li@databricks.com> --------- Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Use variables for bundles * Add example and fix tests * Fix link * Address bug bash issues and polish stacks for private preview * Add bundle run example to docs * Fix nits * Resolve merge conflicts and add TODOs * Fix broken link * Rename bricks CLI to databricks * Add doc for generating example batch inference input (databricks#14) Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Fix Bug Bash Issues * Fixes * Azure DevOps integration using Databricks Asset Bundles (databricks#12) * vscode gitignore * vscode gitignore * typo in test configs * update reqs path * update tests-ci to use bundles * add script to install bricks cli * update main -> {{cookiecutter.default_branch}} * add bundle cicd * update docs * typo * update docs to have ado * update pipelines to use cookiecutter URLs * doc updates * remove print statement * update setup script bricks -> databricks rename * update to use databricks cli and lowercase staging * lowercase standardize, databricks CLI * update conditions * update name * parameterize databricks cli version * bricks -> databricks * lowercase displayName * update pipelines * minor doc updates * revert condition * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/.azure/devops-pipelines/{{cookiecutter.project_name}}-tests-ci.yml Co-authored-by: Vladimir Kolovski <vladimir.kolovski@databricks.com> * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/{{cookiecutter.project_name_alphanumeric_underscore}}/databricks-resource/README.md Co-authored-by: Vladimir Kolovski <vladimir.kolovski@databricks.com> * Update tests/example-project-configs/aws/aws-github.yaml Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com> * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/.azure/devops-pipelines/{{cookiecutter.project_name}}-bundle-cicd.yml Co-authored-by: Vladimir Kolovski <vladimir.kolovski@databricks.com> * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/.azure/devops-pipelines/{{cookiecutter.project_name}}-bundle-cicd.yml Co-authored-by: Vladimir Kolovski <vladimir.kolovski@databricks.com> * update url * update docs to use SPs over PATs * use SP credentials * use SP credentials * update path * standardize on Azure Pipelines * minor doc updates * typos * Fix links (databricks#16) Signed-off-by: Mingyu Li <mingyu.li@databricks.com> --------- Signed-off-by: Mingyu Li <mingyu.li@databricks.com> Co-authored-by: Vladimir Kolovski <vladimir.kolovski@databricks.com> Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com> Co-authored-by: Mingyu Li <mingyu.li@databricks.com> * some docs fixes. * more changes * fix broken links. * fix links * add cleanup in hook * remove mlflow recipes from another place. * make changes * small fix * doc changes --------- Signed-off-by: Mingyu Li <mingyu.li@databricks.com> Co-authored-by: Mingyu Li <mingyu.li@databricks.com> Co-authored-by: Niall Turbitt <52167569+niall-turbitt@users.noreply.github.com> Co-authored-by: Vladimir Kolovski <vladimir.kolovski@databricks.com> Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com> Co-authored-by: Jerry Liang <jerry.liang@databricks.com> Co-authored-by: Jerry Liang <66143562+jerrylian-db@users.noreply.github.com>
Unified project run modes (databricks#72) * Unified project run modes Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Remove model's depedency on batch_inference_job Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Revert change of adding model name to BatchInference parameters Signed-off-by: Mingyu Li <mingyu.li@databricks.com> * Update model name and experiment name to use 'dev' prefix when users manually run notebooks Signed-off-b 62F2 y: Mingyu Li <mingyu.li@databricks.com> --------- Signed-off-by: Mingyu Li <mingyu.li@databricks.com>