Set up sync for dedicated autoload transfer job #71
Conversation
There was a problem hiding this comment.
Reviewable status: 0 of 1 approvals obtained (waiting on @cristinaleonr)
daily-archive-transfers.yaml line 49 at r1 (raw file):
'stctl', '-gcs.source=pusher-$PROJECT_ID', '-gcs.target=archive-$PROJECT_ID', '-time=00:00:00',
I see the configuration in GCP console says "Hourly", but I expected these flags to only apply once per day at UTC midnight, 00:00:00.
What am I missing?
There was a problem hiding this comment.
Reviewable status: 0 of 1 approvals obtained (waiting on @cristinaleonr)
daily-archive-transfers.yaml line 49 at r1 (raw file):
Previously, stephen-soltesz (Stephen Soltesz) wrote…
I see the configuration in GCP console says "Hourly", but I expected these flags to only apply once per day at UTC midnight, 00:00:00.
What am I missing?
It sounds like stctl does not currently support "hourly" frequency options. But, this setting is configurable after a transfer job is created in the cloud console web UI.
So, I proposed that we keep this declared configuration and manually update the frequency setting.
Please add a TODO to add this functionality natively, and a note warning that this declaration is not identical to what is in the corresponding GCP config.
There was a problem hiding this comment.
Reviewable status: 0 of 1 approvals obtained (waiting on @stephen-soltesz)
daily-archive-transfers.yaml line 49 at r1 (raw file):
Previously, stephen-soltesz (Stephen Soltesz) wrote…
It sounds like stctl does not currently support "hourly" frequency options. But, this setting is configurable after a transfer job is created in the cloud console web UI.
So, I proposed that we keep this declared configuration and manually update the frequency setting.
Please add a TODO to add this functionality natively, and a note warning that this declaration is not identical to what is in the corresponding GCP config.
It created this job, which I edited to run hourly.
I deleted the old job. I don't know why it still shows up in the UI when you click the link. If you try to make any updates, it says it has been deleted.
I added the comment with the NOTE and the TODO.
There was a problem hiding this comment.
Reviewed 1 of 1 files at r2.
Reviewable status:complete! 1 of 1 approvals obtained
daily-archive-transfers.yaml line 49 at r1 (raw file):
Previously, cristinaleonr (Cristina Leon) wrote…
It created this job, which I edited to run hourly.
I deleted the old job. I don't know why it still shows up in the UI when you click the link. If you try to make any updates, it says it has been deleted.
I added the comment with the NOTE and the TODO.
Thank you.
This PR sets up a sync for a dedicated transfer job of autoloaded data.
The job runs hourly. It only transfers data under the "autoload" prefix from the pusher bucket to the archive bucket.
https://pantheon.corp.google.com/transfer/jobs/transferJobs%2F10435470182208531695/configuration?project=mlab-sandbox
This change is