Use new APIs for exports #95
Conversation
warehouse/bigquery.go
Outdated
|
|
||
| if field.FullStoryFieldName == "" { | ||
| // We need to pull the type from the existing schema | ||
| bqType = existing[i].Type |
There was a problem hiding this comment.
nit: Should we move this block to the top so that a bqType is always available? And then the rest of the code flow can be the same and doesn't need the special field.FullStoryFieldName != "" handling above?
Co-authored-by: Clay Goddard <clay@fullstory.com>
|
Ok @claygoddard I think I addressed all of the first pass. One question: should we update the Readme with this PR, or do it after this is merged? In one sense, I think it's fine to wait since this is more or less backwards compatible. In another sense, the readme isn't accurate anymore 😬. I think my preference is that we merge this, then follow up with the README changes and release a 1.0.0 binary. WDYT? |
I don't feel strongly so totally up to you. I'd be fine with you including it here though for consistency with the code and config. |
I think there are a few minor ones that weren't addressed that you didn't respond to or said you would address but didn't do anything. I think they were mostly comment improvements / nits so otherwise I think this LG! |
Whoops, this was wrong on my end. I was clicking through some of the comments which was landing me on old historical views of the files 🤦. Things look good all in all! |
This is significant update which migrates the existing export client from the existing APIs to the new (as of yet, undocumented) "Segment Export" APIs.
The aim is to be as backwards compatible as possible, but there are few notable changes from the existing workflow:
Exports are not "bundled"
With the existing APIs, the exports were "bundled" into a time period that was defined in the "Data Export" settings for in the FullStory application. The new APIs do not have this restriction. Instead, the new APIs allows a specifying a time period for which to export data. However, the change here means that we no longer support "listing" exports since they are now user defined. Instead, a
StartTimemust be specified in the config if starting fresh. However, if a "sync" exists for the provided configuration,StartTimewill simply be ignored.The second implication of this is that the export duration must be specified in the config as well. The default value (if unspecified) is now 1 hour.
New fields
With the new APIs a few new fields have been added:
Special cased fields
PageAgent. If a database already has this column,PageUserAgentwill be translated to match. Otherwise, the column will be namedPageUserAgent.