8000 Rust - Fix Serialization when tokens are part of original vocab by n1t0 · Pull Request #315 · huggingface/tokenizers · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Rust - Fix Serialization when tokens are part of original vocab #315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 22, 2020
Merged

Rust - Fix Serialization when tokens are part of original vocab #315

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions tokenizers/src/tokenizer/added_vocabulary.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -459,8 +459,6 @@ impl Serialize for AddedVocabulary {
where
S: Serializer,
{
let mut vocabulary = serializer.serialize_seq(Some(self.added_tokens_map.len()))?;

let mut added_tokens = self
.added_tokens_map_r
.iter()
Expand All @@ -473,6 +471,7 @@ impl Serialize for AddedVocabulary {
// We need to have these added tokens ordered by ascending ID
added_tokens.sort_unstable_by_key(|o| o.id);

let mut vocabulary = serializer.serialize_seq(Some(added_tokens.len()))?;
for token in added_tokens {
vocabulary.serialize_element(&token)?;
}
Expand Down
0