8000 add last_collected checks to retry_errored_repos to correctly follow … by sgoggins · Pull Request #2760 · chaoss/augur · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

add last_collected checks to retry_errored_repos to correctly follow … #2760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 25, 2024
Merged

add last_collected checks to retry_errored_repos to correctly follow … #2760

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions augur/tasks/start_tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from __future__ import annotations

Check warning on line 1 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0114: Missing module docstring (missing-module-docstring)

Raw Output:
augur/tasks/start_tasks.py:1:0: C0114: Missing module docstring (missing-module-docstring)

Check warning on line 1 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0114: Missing module docstring (missing-module-docstring)

Raw Output:
augur/tasks/start_tasks.py:1:0: C0114: Missing module docstring (missing-module-docstring)
import logging
import os
#from celery.result import AsyncResult
Expand All @@ -6,15 +6,15 @@
from sqlalchemy import and_,update


from augur.tasks.github import *

Check warning on line 9 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
W0401: Wildcard import augur.tasks.github (wildcard-import)

Raw Output:
augur/tasks/start_tasks.py:9:0: W0401: Wildcard import augur.tasks.github (wildcard-import)

Check warning on line 9 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
W0401: Wildcard import augur.tasks.github (wildcard-import)

Raw Output:
augur/tasks/start_tasks.py:9:0: W0401: Wildcard import augur.tasks.github (wildcard-import)
if os.environ.get('AUGUR_DOCKER_DEPLOY') != "1":
from augur.tasks.data_analysis import *

Check warning on line 11 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
W0401: Wildcard import augur.tasks.data_analysis (wildcard-import)

Raw Output:
augur/tasks/start_tasks.py:11:4: W0401: Wildcard import augur.tasks.data_analysis (wildcard-import)

Check warning on line 11 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
W0401: Wildcard import augur.tasks.data_analysis (wildcard-import)

Raw Output:
augur/tasks/start_tasks.py:11:4: W0401: Wildcard import augur.tasks.data_analysis (wildcard-import)
from augur.tasks.github.detect_move.tasks import detect_github_repo_move_core, detect_github_repo_move_secondary

Check warning on line 12 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.detect_move.tasks import detect_github_repo_move_core, detect_github_repo_move_secondary" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:12:0: C0413: Import "from augur.tasks.github.detect_move.tasks import detect_github_repo_move_core, detect_github_repo_move_secondary" should be placed at the top of the module (wrong-import-position)

Check warning on line 12 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.detect_move.tasks import detect_github_repo_move_core, detect_github_repo_move_secondary" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:12:0: C0413: Import "from augur.tasks.github.detect_move.tasks import detect_github_repo_move_core, detect_github_repo_move_secondary" should be placed at the top of the module (wrong-import-position)
from augur.tasks.github.releases.tasks import collect_releases

Check warning on line 13 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.releases.tasks import collect_releases" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:13:0: C0413: Import "from augur.tasks.github.releases.tasks import collect_releases" should be placed at the top of the module (wrong-import-position)

Check warning on line 13 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.releases.tasks import collect_releases" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:13:0: C0413: Import "from augur.tasks.github.releases.tasks import collect_releases" should be placed at the top of the module (wrong-import-position)
from augur.tasks.github.repo_info.tasks import collect_repo_info, collect_linux_badge_info

Check warning on line 14 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.repo_info.tasks import collect_repo_info, collect_linux_badge_info" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:14:0: C0413: Import "from augur.tasks.github.repo_info.tasks import collect_repo_info, collect_linux_badge_info" should be placed at the top of the module (wrong-import-position)

Check warning on line 14 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.repo_info.tasks import collect_repo_info, collect_linux_badge_info" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:14:0: C0413: Import "from augur.tasks.github.repo_info.tasks import collect_repo_info, collect_linux_badge_info" should be placed at the top of the module (wrong-import-position)
from augur.tasks.github.pull_requests.files_model.tasks import process_pull_request_files

Check warning on line 15 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.pull_requests.files_model.tasks import process_pull_request_files" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:15:0: C0413: Import "from augur.tasks.github.pull_requests.files_model.tasks import process_pull_request_files" should be placed at the top of the module (wrong-import-position)

Check warning on line 15 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.pull_requests.files_model.tasks import process_pull_request_files" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:15:0: C0413: Import "from augur.tasks.github.pull_requests.files_model.tasks import process_pull_request_files" should be placed at the top of the module (wrong-import-position)
from augur.tasks.github.pull_requests.commits_model.tasks import process_pull_request_commits

Check warning on line 16 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.pull_requests.commits_model.tasks import process_pull_request_commits" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:16:0: C0413: Import "from augur.tasks.github.pull_requests.commits_model.tasks import process_pull_request_commits" should be placed at the top of the module (wrong-import-position)

Check warning on line 16 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.github.pull_requests.commits_model.tasks import process_pull_request_commits" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:16:0: C0413: Import "from augur.tasks.github.pull_requests.commits_model.tasks import process_pull_request_commits" should be placed at the top of the module (wrong-import-position)
from augur.tasks.git.dependency_tasks.tasks import process_ossf_dependency_metrics

Check warning on line 17 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.git.dependency_tasks.tasks import process_ossf_dependency_metrics" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:17:0: C0413: Import "from augur.tasks.git.dependency_tasks.tasks import process_ossf_dependency_metrics" should be placed at the top of the module (wrong-import-position)

Check warning on line 17 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
C0413: Import "from augur.tasks.git.dependency_tasks.tasks import process_ossf_dependency_metrics" should be placed at the top of the module (wrong-import-position)

Raw Output:
augur/tasks/start_tasks.py:17:0: C0413: Import "from augur.tasks.git.dependency_tasks.tasks import process_ossf_dependency_metrics" should be placed at the top of the module (wrong-import-position)
from augur.tasks.github.traffic.tasks import collect_github_repo_clones_data
from augur.tasks.gitlab.merge_request_task import collect_gitlab_merge_requests, collect_merge_request_metadata, collect_merge_request_commits, collect_merge_request_files, collect_merge_request_comments
from augur.tasks.gitlab.issues_task import collect_gitlab_issues, collect_gitlab_issue_comments
Expand Down Expand Up @@ -322,17 +322,26 @@
engine = self.app.engine
logger = logging.getLogger(create_collection_status_records.__name__)

#TODO: Isaac needs to normalize the status's to be abstract in the

Check warning on line 325 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
W0511: TODO: Isaac needs to normalize the status's to be abstract in the  (fixme)

Raw Output:
augur/tasks/start_tasks.py:325:5: W0511: TODO: Isaac needs to normalize the status's to be abstract in the  (fixme)

Check warning on line 325 in augur/tasks/start_tasks.py

View workflow job for this annotation

GitHub Actions / runner / pylint 

[pylint] reported by reviewdog 🐶
W0511: TODO: Isaac needs to normalize the status's to be abstract in the  (fixme)

Raw Output:
augur/tasks/start_tasks.py:325:5: W0511: TODO: Isaac needs to normalize the status's to be abstract in the  (fixme)
#collection_status table once augur dev is less unstable.
with DatabaseSession(logger,engine) as session:
query = s.sql.text(f"""UPDATE collection_status SET secondary_status = '{CollectionState.PENDING.value}'"""
f""" WHERE secondary_status = '{CollectionState.ERROR.value}' ;"""
f""" WHERE secondary_status = '{CollectionState.ERROR.value}' and secondary_data_last_collected is NULL;"""
f"""UPDATE collection_status SET core_status = '{CollectionState.PENDING.value}'"""
f""" WHERE core_status = '{CollectionState.ERROR.value}' ;"""
f""" WHERE core_status = '{CollectionState.ERROR.value}' and core_data_last_collected is NULL;"""
f"""UPDATE collection_status SET facade_status = '{CollectionState.PENDING.value}'"""
f""" WHERE facade_status = '{CollectionState.ERROR.value}' ;"""
f""" WHERE facade_status = '{CollectionState.ERROR.value}' and facade_data_last_collected is NULL;"""
f"""UPDATE collection_status SET ml_status = '{CollectionState.PENDING.value}'"""
f""" WHERE ml_status = '{CollectionState.ERROR.value}' ;"""
f""" WHERE ml_status = '{CollectionState.ERROR.value}' and ml_data_last_collected is NULL;"""

f"""UPDATE collection_status SET secondary_status = '{CollectionState.SUCCESS.value}'"""
f""" WHERE secondary_status = '{CollectionState.ERROR.value}' and secondary_data_last_collected is not NULL;"""
f"""UPDATE collection_status SET core_status = '{CollectionState.SUCCESS.value}'"""
f""" WHERE core_status = '{CollectionState.ERROR.value}' and core_data_last_collected is not NULL;;"""
f"""UPDATE collection_status SET facade_status = '{CollectionState.SUCCESS.value}'"""
f""" WHERE facade_status = '{CollectionState.ERROR.value}' and facade_data_last_collected is not NULL;;"""
f"""UPDATE collection_status SET ml_status = '{CollectionState.SUCCESS.value}'"""
f""" WHERE ml_status = '{CollectionState.ERROR.value}' and ml_data_last_collected is not NULL;;"""
)

session.execute_sql(query)
Expand Down
Loading
0