Support view() on batch dimensions for non-contiguous tensors? #3653

vadimkantorov · 2017-11-12T07:16:31Z

a = torch.Tensor(16, 8, 32, 64)
a.view(-1, 32, 64)                   # works
a.transpose(-1, -2).view(-1, 64, 32) # doesn't
a.view(-1, 32, 64).transpose(-1, -2) # works but doesn't fit some interfaces, i.e. sometimes we want to do view as the last operation

Such view's are needed to implement 4D+ bmm that can treat all dimensions except last two as batch dimensions (similarly to Linear module's behavior). Unless I move transpose inside the bmm func (which would not match the existing interface but well), an extra contiguous call is needed.

Does it make sense to support such view call? On one hand it breaks invariant that view always returns a contiguous tensor, on the other side the situation for several batch dimensions may be common.

Earlier discussed in #764

The text was updated successfully, but these errors were encountered:

apaszke · 2017-11-12T11:18:08Z

There's a good reason for the view invariant - most of such reshapes are impossible to pull off using stride tricks if then tensor isn't contiguous. On the other hand, making it contiguous inside view would mean that sometimes the returned tensor shares storage with input, and sometimes doesn't. This is important for cases like these: x.view(-1)[::x.size(1) + 1] += c (add c to diagonal of matrix x). If you know/suspect that tensors might sometimes be non-contiguous just add .contiguous() before .view() it's a no-op if the tensor already is contiguous

fmassa · 2017-11-12T11:26:51Z

I think Vadim's point is that if the subspace of the non-contiguous tensor that we are modifying is contiguous, then the stride tricks should work, and we could reduce some of the restrictions in view.
This is similar in spirit with the unsqueeze function, that required dedicated implementation because of the restrictions in view.
Not sure if there are edge cases there though, and might just be safer/better to have everything uniform.

apaszke · 2017-11-12T11:43:31Z

Ohhh I haven't noticed this, my bad. What "works but doesn't fit some interfaces" means? I think it should give results equivalent to the previous line (if it worked). I think relaxing the constraint on view to be able to work with certain classes of non-contiguous tensors would be reasonable

vadimkantorov · 2017-11-12T11:53:06Z

Yep, you got me right :)

By "works but doesn't fit some interfaces" I meant that bmm interface accepts BxPxQ and BxQxR, so if I'm implementing a generalized version that flattens some first dimensions, then the inputs must already be properly transposed by caller. And then there's this problem that we can't flatten first dimensions of non-contiguous tensors. In other words, extending a function like bmm requires first a call to transpose and then view, so the last line can't be used without having to change the bmm's contract.

apaszke · 2017-11-13T12:15:49Z

I'm not sure if I understand, can't you just call contiguous() inside your function (this doesn't change the interface), and only then flatten the leading dimensions? This will be a no-op if user gave you a properly transposed tensor, and will fix the problem otherwise

vadimkantorov · 2017-11-13T14:28:55Z

That's what I currently do.

Though B will always be non-contiguous and an unnecessary copy of B will happen, since transpose must be applied first (instead of blas call with right transa, transb arguments):

import torch
def bmm(A, B):
    print(A.is_contiguous(), B.is_contiguous())
    return torch.bmm(A.contiguous().view(-1, A.size(-2), A.size(-1)), B.contiguous().view(-1, B.size(-2), B.size(-1))).view(A.size()[:-1] + (-1, ))

a = torch.Tensor(16, 8, 32, 64)
bmm(a, a.transpose(-1, -2)) # prints (True, False)

If view is augmented to support the way @fmassa described better than I did, then maybe .is_contiguous(dim = [0,1,2]) support also makes sense, so that the caller can find out if the tensor is contiguous in the given subspace of dimensions and .contiguous(dim = [0,1,2]) to force it so.

ssnl · 2017-12-06T21:05:13Z

working on this

soumith · 2017-12-18T07:08:32Z

closed via #4062

vadimkantorov mentioned this issue Nov 28, 2017

expanded scalar is not contiguous #3919

Closed

soumith assigned ssnl Dec 1, 2017

soumith added enhancement labels Dec 1, 2017

ssnl mentioned this issue Dec 6, 2017

Allow .view on noncontig tensors when certain conditions are met #4062

Merged

soumith closed this as completed Dec 18, 2017

CatalinVoss mentioned this issue Mar 24, 2019

Fix Language Modeling Loss huggingface/transformers#404

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support view() on batch dimensions for non-contiguous tensors? #3653

Support view() on batch dimensions for non-contiguous tensors? #3653

Support view() on batch dimensions for non-contiguous tensors? #3653

Support view() on batch dimensions for non-contiguous tensors? #3653

Comments