PyTorch serialization formats #31877
Comments
|
I agree we should have these docs |
|
I believe the zip format should have a magic number that is used for versioning but I am not sure we have publicly documented it |
|
The higher order question is if there is a consistent and "official" way to refer to ALL formats, including the legacy formats. If the scheme in the description above makes sense, then Zip format version in the |
|
Going through the current versions down to 0.3, it looks like we haven't been that great about versioning. The eager format has been very stable as the multi-pickle file (no changes since 0.3, the The TorchScript format has gone through more structural changes with only 1 really fundamental change (when There is also a new serialization format for eager ("Eager v2" below) coming in
These commands show the contents of the zipfiles for each version, the binaries are all from here. Script v1, v2, v3, v4 $ unzip script_example_1.4.0.pt
Archive: script_example_1.4.0.pt
extracting: script_example_1.5.0a0/version
extracting: script_example_1.5.0a0/data/0
extracting: script_example_1.5.0a0/data/1
extracting: script_example_1.5.0a0/data.pkl
inflating: script_example_1.5.0a0/code/__torch__.py
inflating: script_example_1.5.0a0/code/__torch__.py.debug_pkl
extracting: script_example_1.5.0a0/constants.pkl
$ unzip script_example_1.3.1.pt
Archive: script_example_1.3.1.pt
extracting: script_example_1.3.1/version
inflating: script_example_1.3.1/code/__torch__.py
inflating: script_example_1.3.1/code/__torch__.py.debug_pkl
extracting: script_example_1.3.1/constants.pkl
extracting: script_example_1.3.1/data/0
extracting: script_example_1.3.1/data/1
extracting: script_example_1.3.1/data.pkl
$ unzip script_example_1.2.0.pt
Archive: script_example_1.2.0.pt
extracting: script_example_1.2.0/version
extracting: script_example_1.2.0/code/script_example_1.2.0.py
extracting: script_example_1.2.0/debug/script_example_1.2.0.pkl
extracting: script_example_1.2.0/attributes.pkl
extracting: script_example_1.2.0/tensors/0
extracting: script_example_1.2.0/tensors/1
extracting: script_example_1.2.0/model.json
$ unzip script_example_1.1.0.pt
Archive: script_example_1.1.0.pt
extracting: script_example_1.1.0/version
extracting: script_example_1.1.0/code/script_example_1.1.0.py
extracting: script_example_1.1.0/attributes.pkl
extracting: script_example_1.1.0/tensors/0
extracting: script_example_1.1.0/tensors/1
extracting: script_example_1.1.0/model.json
$ unzip script_example_1.0.0.pt
Archive: script_example_1.0.0.pt
extracting: script_example_1.0.0/version
extracting: script_example_1.0.0/code/script_example_1.0.0.py
extracting: script_example_1.0.0/tensors/0
extracting: script_example_1.0.0/tensors/1
extracting: script_example_1.0.0/model.jsonEager v2 $ unzip unzip eager_example_new_1.5.0a0.pt
Archive: eager_example_new_1.5.0a0.pt
extracting: eager_example_new_1.5.0a0/version
extracting: eager_example_new_1.5.0a0/data.pkl
extracting: eager_example_new_1.5.0a0/data/94148253811520 |
|
@driazati thank you for sharing the files. 686e8d3 introduced I'm trying to get more specific on how different formats can be uniquely named. For example, instead of "this file is in .tar format", would it be correct to say "this file is in PyTorch v0.1.1 format"? Not sure For example, assume a tool needs to tell a user which format a def get_display_format(file):
if is_tar_file(file):
return "XXXXX" # PyTorch v0.1.1?
if is_multi_pickle_file(file):
return "XXXXX" # PyTorch v0.1.10?
if is_zip_file(file):
if has_version_file(1) or has_version_file(None):
if has_model_json(file):
return "XXXXX" # PyTorch v1.0?
if has_data_pkl(file) and has_code_folder(file):
return "XXXXX" # PyTorch v1.3 (TorchScript)?
if has_data_pickle(file) and not has_code_folder(file):
return "XXXXX" # PyTorch v1.4?
else:
if has_code_folder(file):
return "XXXXX v" + generate_display_version_from_version_file(file) + " (TorchScript)"
else:
return "XXXXX v" + generate_display_version_from_version_file(file)Question 1: What should those specific Question 2: What can be done to simplify this so Question 3: Some files include tensor state only while others include code or model structure as well. Since this seems to cause confusion among users is there a recommended way to include this in the format |
|
Can you give some more context on the background for this issue? An easy way to get everything into 1 format would be to load and re-save in the latest version of PyTorch. Since we're fully backwards compatible down to 0.1, we can load any objects (in the eager case) or TorchScript models and save them with the latest format.
|
|
Context is which format description Netron should show to users for diagnosing issues or discussing the changes PyTorch is going through. The other goal is to have some principles for version changes that will be followed going forward. Would this be a correct representation of what we discussed so far? def get_display_format(file):
if is_tar_file(file):
return "PyTorch Eager v0.1.1"
if is_multi_pickle_file(file):
return "PyTorch Eager v0.1.10"
if is_zip_file(file):
if has_version(file, 2) or has_version(file, 1) or has_version(file, None):
if has_model_json(file):
return "PyTorch Script v1.0"
if has_data_pkl(file):
if has_code_folder(file):
if has_version_file(file, 2):
return "PyTorch Script v1.4"
return "PyTorch Script v1.3"
else:
return "PyTorch Eager v1.4"
else:
if has_code_folder(file):
return "PyTorch Script v" + generate_display_version_from_version_file(file)
else:
return "PyTorch Eager v" + generate_display_version_from_version_file(file)It is still unclear how |
|
Looks mostly good (pinging @ezyang for any thoughts), a couple notes:
So in the end it'd look something like def get_display_format(file):
if is_tar_file(file):
return "PyTorch v0.1.1"
if is_multi_pickle_file(file):
return "PyTorch v0.1.10"
if is_zip_file(file):
if has_model_json(file):
if has_attribute_pkl():
return "TorchScript v1.1"
else:
return "TorchScript v1.0"
if has_data_pkl(file):
if has_constants_pkl(file):
if has_version_file(file, 2):
return "TorchScript v1.4"
return "TorchScript v1.3"
else:
return "PyTorch v1.4"We discussed the |
|
Is
Is there a way for tools to derive the PyTorch version needed for a given TorchScript file? If |
|
@driazati I have little to say about the exact details of how we are version testing, or what the variants of the versions should be named (aligning them with PyTorch releases sounds reasonable). What I don't see in this discussion is whether or not the team is going to commit to accurately reporting versions on the file format going forward, and if so, what mechanisms we can put in place to make sure that we update it when we make changes to the format (since it seems the lightweight mechanism of code review isn't working). A simple stopgap is to have it report the version of PyTorch which exported the model... |
Agree. Given the scheme we discussed above this would also make the most sense for tooling. The current |
Uh oh!
There was an error while loading. Please reload this page.
@soumith, @ezyang, as PyTorch serialization formats keep changing and evolving, is there a scheme to name and version the different formats to avoid confusion?
Something along these lines:
sys_info,pickle,storages,tensors8a0a6cfc9c...signatureconstants.pklandmodel.jsonconstants.pklanddata.pkldata.pklbut no codecc @ezyang @gchanan @zou3519 @suo
The text was updated successfully, but these errors were encountered: